Data for Impact hosted a one-hour webinar sharing guidance for using routine data in evaluations. More: https://www.data4impactproject.org/resources/webinars/routine-data-use-in-evaluation-practical-guidance/

1. Routine data use in evaluation: practical
guidance
Eva Silvestre, Ph.D., Tulane University, Data for Impact
Webinar, September 24, 2020

2. Outline of
webinar
• Introduction
• Overview of activity
• Key points from guidance
document
• Examples:
Ukraine – Zola Allen
Kenya- Lyubov Teplitskaya
• Questions

3. Routine data sources
• Routine data sources are
collected at regular
intervals at public, private,
and community-level health
facilities and institutions.
• There has been a lack of
confidence in routine
health information systems
(RHIS).
• USAID and other donors
have made significant
investments to improve
these systems through a
wide range of
interventions.

4. • Historically, routine data have been passed over by
evaluators in favor of other options, such as stand-
alone surveys tailored to meet evaluation objectives.
• But primary data collection can be expensive and
time-intensive.
• Availability of routine data, the perception of its cost-
efficiency, and the complex nature of health
interventions being implemented has led to its
increased use in evaluations.
Routine data in evaluation

5. Approach • Reviewed evaluations conducted
by MEASURE Evaluation that used
routine data.
• Literature review to identify
additional examples of studies that
used routine data.
• 18 evaluations selected for further
investigation, based on the type of
evaluation design, the source of
data, and the health area being
evaluated.
• Focused on 13 briefs whose
original authors could provide
further information.

6. Understand
the RHIS of
the country
or countries
of interest
• Become familiar with RHIS in
general and for the country where
the evaluation will take place.
• It is helpful to know how data is
collected, transmitted, aggregated,
and used there.
• Know the process for getting
permission to access the data.
• Look for foundational documents
for countries, such as:
• HIS strategic plan
• Annual health statistics reports
• Core health indicators reference
sheets
• PRISM assessment results and DQA
and RDQA results

7. RHIS
context
• HIS across the world have
evolved in unique ways.
• The history and context is
important, especially if the
evaluation covers a long time
period.
• Contextual factors can affect
data collection:
• Disease outbreaks like Ebola or
COVID-19
• Natural disasters
• Labor disputes where data is withheld
to extract concessions
• Violent conflict

8. • During the evaluation period, have there been changes to the way the
indicators of interest have been defined?
• In the time period, have there been changes in the way data have been
collected?
• Have there been any changes to administrative alignment during this time?
• Are there any expected changes to administrative alignment during the
evaluation?
• Have there been any contextual disruptions that may have affected service
delivery, availability of required commodities (e.g., vaccines or therapeutics),
or data entry?
RHIS context questions

9. Understand
specific data
source(s)
• Data sources present their own
opportunities and challenges.
• RHIS are set up to meet a
country’s information needs and
not set up for research or
evaluation.
• Some challenges:
• Case-based versus visit-based
sources
• Ability to extract data for analysis
• Presence (or not) of unique identifiers
• Interoperability of multiple data
sources or of older and newer systems

10. Data quality in RHIS
Tools available Considerations
• Investment in improving
data quality and use
• Several tools available that
help assess
• Dimensions of data quality

11. Data
completeness
• Understand reporting deadlines.
• Review internal data quality
reports.
• Conduct data quality audits.
• Options for dealing with missing
values.
• Approach will depend on to what
extent data are missing and the
likely cause. One could:
• Ignore
• Exclude districts or facilities
• Impute

12. • Majority of the evaluations reviewed used
additional data sources—including primary data
collection—to answer all the evaluation questions.
• Primary data: Qualitative methods (interviews and
focus group discussions)
• Secondary data: Demographic and Health Surveys,
climatic data, service provision assessment data
Other sources of data

13. • Routine data can be a valuable source of information to
assess health program performance and impact on
outcomes.
• There is no perfect solution—each approach must be
tailored to the goals of the evaluation and the nature of
the data.
• Evaluators may need to first assess the data and present
a summary of it to donors who want to use it—so that all
parties are aware of possible limitations.
Conclusion

14. Strengthening Tuberculosis (TB) Control in Ukraine: Evaluation
of the Impact of the TB-HIV Integration Strategy on Treatment
Outcomes
Zola Allen, Ph.D., Palladium, Data for Impact
September 24, 2020

15. TB in Ukraine
• One of 10 countries
with a high burden of
multi-drug resistant TB
(MDR-TB) for 2016–
2020
• One of 20 countries
with the highest
estimated numbers of
MDR-TB cases
World Health Organization. Global Tuberculosis Report. 2019
https://apps.who.int/iris/bitstream/handle/10665/329368/9789241565714-
eng.pdf?ua=1.

16. • To improve the delivery of TB and HIV services,
with the goal of enhancing the timeliness of care
and the life expectancy of patients with TB-HIV
coinfection
Strengthening TB Control in Ukraine
(STbCU) project

17. • Examined the relationships among the strategies
for integrating TB and HIV services, the use of
TB-HIV services, and mortality outcomes
• Employed a mixed-methods approach
• A quasi-experimental quantitative evaluation design,
complemented by qualitative interviews to inform the
findings
Impact evaluation of the STbCU project

18. 1. Completion of TB-HIV service cascade
2. Impact of service integration on the time lag
between each step of service
3. Impact of service integration on all-cause
mortality
Questions addressed using routine data

19. • Sufficient to address the evaluation questions
• Medical chart data aligned well with the evaluation
period
• Used abstracted data from routine data sources in all
six study oblasts
Usability and quality of the data

20. • Data captured in electronic registers
• Electronic registers contained information by case, not
by patient.
• Necessary to de-duplicate data and include only the latest
case in the sampling frame
• Patient lists at AIDS centers included all HIV patients
registered.
• Necessary to exclude patients who do not receive treatment
in the AIDS centers (e.g., patients who were diagnosed with
HIV within the penitentiary system)
Challenges in the usability of routine
data for the evaluation

21. • Data availability
• Missing data on several HIV disease characteristics
(numbers of visits, clinical stage, and CD4 count)
• At the TB facilities
• At the AIDS centers (at baseline, >50% of the coinfected
patients had data missing)
• Missing data for several TB disease characteristics
(classification, clinical form, and treatment category)
• At the AIDS centers
• Data on status of injecting drug users not collected by
the facilities
Challenges in the usability of routine
data for the evaluation, cont.

22. • Data accuracy
• Analysis of the time lag between each step of service
• Dates of service incorrectly recorded according to the
expected time sequence
• Need for data cleaning
• Analysis on the number of planned and received
doses of treatment
• These numbers did not always correspond with the duration
of treatment
• Low variation on the proportion of doses completed
Challenges in the usability of routine
data for the evaluation, cont.

23. • Inconsistency in the use of data collection tools
across health facilities
• HIV control card modifications in 2012
• Difficult to find out which version was used by each facility
• Fields in the two forms meant different things [e.g., the code
(T6) for “treatment completed" in the first version of the form
was changed to “requires preventive treatment”]
Challenges in the usability of routine
data for the evaluation, cont.

24. • Missing data
• Missing data in the medical charts and electronic
registers
• Developed and documented imputation rules and other
decisions on how to handle missing data
Challenges in the usability of routine
data for the evaluation, cont.

25. • Constrained the analysis to the variables that
were available from the records
• Data from the records were better suited for the
analysis of service cascades than for analyzing the
effect of services on survival.
Limitations in using routine data for
evaluation

26. • Challenging and time-consuming fieldwork
• It was difficult sometimes to find the necessary medical
records in the archives.
• TB forms were not always kept with forms containing
HIV-related information.
• Intensive and follow-up TB treatment were usually
provided at different facilities.
Limitations in using routine data for
evaluation, cont.

27. • Inability to de-
duplicate patients
served by both types
of facilities
• Samples were collected
and analyzed separately
based on each patient’s
point of service
• Extra efforts to
abstract handwritten
information from
medical charts
Limitations in using routine data for
evaluation, cont.

28. • Most data needed for the evaluation were
available and accessible at the health facilities.
• The data collection and data management challenges
were expected and therefore sufficient time and
financial resources were planned for the work.
• Imputation rules and other decisions were
developed and documented on how to handle
missing and inconsistent data.
• Rules and decisions applied during the baseline and
end-line evaluation phases.
What worked well

29. • Routine data were successfully used to address
the evaluation questions.
• Good planning, detailed documentation, and
flexibility were important.
• The development and documentation of
imputation rules and other decisions on how to
handle missing and inconsistent data were
essential components.
• The use of routine data worked better for some
questions (e.g., service cascades), than for others
(e.g., the effect of services on survival).
Conclusion

30. Improving Maternal and Child Health Outcomes in Kenya:
Impact of the Free Maternity Service Policy on Healthcare
Use and Lives Saved
Lyubov Teplitskaya, Palladium, Data for Impact
Photo: Peter Kapuscinski / World Bank

31. 1989: Kenya introduces user fees
1990: Waiving policy for poor and
children under five introduced
1991-2003: User fees re-introduced
through phased approach
2004: Kenya introduces “10/20 policy”
2007: Removal of user fees for maternity
care at public facilities
June 2013: Kenya introduces Free
Maternity Service Policy, abolishing user
fees for maternal health at all public
facilities
The maternal mortality ratio in
Kenya remains high, well above
the Sustainable Development
Goal (SDG) target of 70
Background: User Fees in Kenya
513
342
0
100
200
300
400
500
600 Free Maternity Service
Policy
Source: WHO, UNICEF, UNFPA, World Bank Group, and
the United Nations Population Division. Trends in
Maternal Mortality: 2000 to 2017. Geneva, World Health
Organization, 2019

32. • What is the impact of the Free Maternity Service Policy on
maternal healthcare utilization?
• How many maternal and neonatal deaths were averted
due to the Free Maternity Service Policy?
Evaluation Questions

33. • Introduction of the Kenya Free Maternity Service Policy
served as a natural experiment, with potential for use of
quasi-experimental methods to evaluate the effect of the
policy.
• Interrupted time series (ITS) analysis is increasingly used
to evaluate the impact of public health policies.
• Monthly routine data is optimal for use in ITS and similar
analyses because data is collected frequently.
Rationale for Use of Routine Data

34. • Quasi-experimental
statistical analysis
• Used to quantify impact
of an intervention
• Requires sequential
measures of the
outcome variable
before/after intervention
• Frequently used in real
world settings when
RCTs are not possible
• May be difficult to
control for all time-
varying confounders
Interrupted Time
Series (ITS)
• Mathematical modeling tool
• Estimates impact of coverage
change on maternal and child
mortality in low- and middle-
income countries
• Used for advocacy, evaluation,
strategic planning
• “If I increase coverage of
intervention X, I could save Y
number of lives”

35. DHIS2 outcomes:
• Outpatient visits by
children under age 5
• Outpatient visits by
females over age 5
• Clients with more
than 4 ANC visits
• Normal deliveries at
facilities
• Postnatal care visits
Description of Data Consulted
Ownership:
• Ministry of Health
(MOH)
• Faith-based
organizations (FBO)
• Private-for-profit (PFP)
Aggregated all monthly national-level data from all 47 counties in Kenya

36. • Vetted specific coverage indicators for the analysis by
assessing reporting completeness rates
• Outliers
• Adjustments for seasonality
Assessment of Usability and Quality of Data
Facility Type Reporting Completeness (%)
MOH 69%
PFP 71%
FBO 65%
Average Reporting Completeness
(Jan 2011–Jul 2015)

37. • Time period
• Before intervention: January 2011–May 2013
• After intervention: June 2013–August 2015
• Interrupted Time Series
𝒀 = 𝜷 𝟎 + 𝜷 𝟏 𝒕𝒊𝒎𝒆(𝒕) + 𝜷 𝟐 𝒍𝒆𝒗𝒆𝒍(𝒊) + 𝜷 𝟑 𝒕𝒓𝒆𝒏𝒅(𝒊𝒕) + 𝜺
(“itsa” command in Stata 14)
• ITS utilization results used as inputs in LiST to estimate
number of maternal and newborn deaths averted due to
intervention
• Counterfactual projection accounted for pre-policy coverage
trend
Data Analysis

38. • Incomplete facility reporting
• Misalignment between DHIS2 intervention coverage and
LiST service coverage
Limitations
DHIS2 Coverage Indicators LiST Inputs
• Normal delivery at facility
• ANC ≥ 4 visits
• Outpatient visits by children
under age 5
• Outpatient visits by females
over age 5
• Postnatal care visits
• Skilled birth attendance
• Clean postnatal practices
• ANC ≥4 visits
• Iron supplementation
• Antibiotics for treatment of dysentery
• Zinc for treatment of diarrhea
• Oral antibiotics for pneumonia
• Artemesinin compounds (ACT) for
treatment of malaria
• Other extrapolation based on data available in previous
years

39. • Assumed normal delivery at facility coverage = coverage
of clean postnatal practices
• For SBA:
𝐻𝐹𝐷𝑡 =
𝑁𝐷𝐸𝐿𝑡
𝑏𝑖𝑟𝑡ℎ 𝑡
∗ 100%
𝑆𝐵𝐴 𝑡 = 𝐻𝐹𝐷𝑡 + 𝑓𝑡
Example of LiST input calculation using
ITS coverage indicator

40. • Aggregated DHIS2 data from facilities can be used:
• To assess differences in healthcare coverage following an
intervention, such as the initiation of a new policy
• As inputs into tools, such as LiST, to estimate impacts on health
outcomes, such as mortality
• Routine data such as those in DHIS2 have the advantage
of time series
• Recent approaches available to correct for incomplete
facility reporting (Maina et al., 2017)
Conclusions