Innovation in Evaluation
IDCAP aimed to create a comprehensive chain of evidence that demonstrates the value of training healthcare workers. Using a comprehensive study design and new evaluation tools, IDCAP evaluated the impact of training on three levels: individual clinician capacity, facility performance, and patient health outcomes. Then, the impact of the training components was combined with their costs to evaluate the cost-effectiveness of the interventions.
Study Design
As no rigorous evidence of the impact of on-site training in Africa existed at the onset of the project, OSS was randomized to half of 36 mid-sized health facilities distributed throughout Uganda. A cluster-randomized, parallel-arm trial for evaluation of OSS was combined with a pre-post (for individual clinicians) and interrupted time series (for facility performance and health outcomes) design to evaluate IMID. To our knowledge, IDCAP is the first program to use a randomized controlled trial to illustrate the impact and incremental effectiveness of on-site training for the health workforce in sub-Saharan Africa.
Data Surveillance System
IDCAP developed a comprehensive, practical and sustainable Data Surveillance System (DSS) to enable the evaluation of facility-level performance at 36 sites across Uganda. Indicators measured quality of care related to infectious diseases such as HIV/AIDS, malaria, tuberculosis, and supporting functions such as emergency triage. The DSS built on existing Ministry of Health registers and enhanced data collection systems. Key elements of this system included a trained data entry assistant based on-site, a revised outpatient record form, adaptive technologies for reliable data transmission, and regular data support visits. This data surveillance network is one of IDCAP’s lasting contributions to the field of impact evaluation, and is already being used to evaluate the effectiveness of new programs.
Evaluation Tools
IDCAP developed new evaluation tools and methods, such as the Case Scenarios and Clinical Assessment Tools, modified Medical Form 5, and two-sample mortality survey to measure the impact of the training package.
The Medical Form 5
A main source of data in support of IDCAP indicators was the modified Medical Form 5 (MF5), enhanced from a blank sheet to a coded form for improved and standardized outpatient data collection and reporting. The modified MF5 was very well received by local authorities and the central government, which has recommended that it be adopted by all national health facilities. The IDCAP sites also reported using the MF5 system to complete their HMIS forms in record time and with improved accuracy, when compared to systems used prior to its implementation. This helps free up clinicians’ valuable time for clinic work, rather than paper work.
Case Scenarios
Case Scenarios, also referred to as clinical vignettes, were designed to measure MLP’s clinical competence. IDCAP was the first to use case scenarios to longitudinally measure the effects of a training program. Case scenarios have been demonstrated to be more accurate than medical records, are standardized across trainees, and require no adjustment for case complexity in their scoring. Furthermore, the Case Scenarios have the ability to evaluate each trainee’s competency in areas not possible with clinical observation. For example, they allow the assessment of the trainee’s management of less prevalent, more complex cases, and make it possible to observe the trainee’s management of patients over time.
Clinical Assessment Tools
To measure clinical practice, Clinical Assessment Tools were designed to score observed patient encounters. Improving on existing instruments, IDCAP’s Clinical Assessment Tools contain fields for specific information about patient history and symptoms, require detailed diagnosis and treatment data, and offered more objective scoring than their predecessors.
The clinical observation tool included two important innovations: 1) Observers recorded information on patients as well as participant practice so that accurate patient information was available, and 2) Observers conducted a history and physical examination after the participant to complete missing or incorrect information, and then continued the assessment. This additional information then allowed analysts to control for case complexity during data analysis.
Mortality Survey
IDCAP completed a population-based mortality survey to measure the impact of the IDCAP intervention on mortality among children less than five years who lived within a 5km radius of each IDCAP health facility. In collaboration with the Uganda Bureau of Statistics (UBOS), IDCAP developed an innovative two-sample method of measuring mortality. In contrast to the Demographic and Health Surveys (DHS) that seek to estimate national mortality rates based on birth histories reported by birth mothers, the IDCAP survey sought to estimate mortality rates on all children under-five in the selected households. Household and Birth History Questionnaires were designed and translated into13 local languages. The innovative design and implementation of this survey, and the rich demographic data it produced, inform demographic research already underway, and contribute significantly to future initiatives.
Cost-effectiveness Analysis
IDCAP collaborated with researchers from WHO’s Department of Health Systems Financing, the South African Center for Epidemiological Modeling and Analysis (SACEMA), and the Futures Institute to develop an epidemiological model for evaluating the cost-effectiveness of IDCAP’s integrated interventions. The investigators agreed on a method to combine acute illnesses like malaria and pneumonia and chronic or longer term diseases, such as tuberculosis and HIV, and used both original IDCAP data and disease parameters from the literature and other models. Preliminary analyses based on the acute illnesses suggested that the IMID saved 15 lives per year per site relative to no intervention and the combination of IMID and OSS saved 22.9, at a cost of $72 per Disability Adjusted Life Year (DALY) for IMID and $122 for the combination of IMID and OSS. The incremental cost effectiveness of OSS when added to IMID was $111 per DALY. While each country and organization sets its own standard for an affordable cost per DALY, the WHO considers a health intervention to be cost-effective and highly cost-effective if it costs less than three times the annual per capita GDP and less than the per capita GDP, respectively. Uganda’s GDP in 2011 was roughly US$ 487 per capita, making IMID and OSS highly cost-effective interventions both individually and in combination by this broad WHO standard.
