Whast goes up must come down: challenges of getting evidence back to the ground
Data quality in remote monitoring (Mona Fetouh, UNOIOS)
1. Data Quality in Remote
Monitoring
A comparative analysis of
experiences in Somalia and
Eastern Burma
Mona Fetouh (Co-Author and Presenter), Christian Balslev-Olesen (Co-Author), and Volker Hüls (Co-
Author)
2. Remote Monitoring
Humanitarian space restricted in many situations; risks
have increased
Both delivery and monitoring of programmes affected
Remote monitoring and cross-border approaches
required in a number of countries in recent years
Increased reliance on local actors
Reduced ability to collect and verify information
Often results in compromises on data quantity & quality
Although remote management is increasingly common,
evidence on best practice is still emerging
3. Comparable Contexts
Somalia
Classic remote management situation
Difficulty of access due to insecurity, heavy reliance on
national civil society (NGOs, communities) for aid delivery
(still largely managed from neighbouring Kenya)
Risk to external monitors; monitoring relies heavily on local
partners – risk of bias
Eastern Burma
Hard to impossible to access from capital due to
government restrictions and long-running conflicts
between ethnic groups in the East and the government
Heavy reliance on civil society (LNGOs, communities) for
aid delivery that is managed from neighbouring Thailand
Little to no access for INGO staff, including national staff
Risk to external monitors, monitoring relies heavily on local
partners – risk of bias
4. Different Data Environments (1)
Somalia:
Main concern is availability of data.
Little opportunity to collect data regularly,
even for local partners
Local partners have varied capacity to
produce quality data, severe education gap
results in overall low staff capacity
Focus of strengthening objective monitoring
was through investment in independent
systems (―third party verification‖).
5. Different Data Environments (2)
Eastern Burma:
Main concern is management and quality of data
Data are abundant, both monitoring data and surveys
Local partner capacity is varied but good – strong
ethnic and professional exchange with Thai border
area; better access to education
Information remains in technical silos; local NGOs are
largely confined to ethnic areas; access opportunities
are often limited to a particular sector.
Few mechanisms for independent verification/
triangulation of data.
Focus of strengthening objective monitoring was on
the quality and verification of information
Access is improving due to ceasefire agreements
6. Comparative Analysis
Similar contexts require different
approaches
Presentation with details on
Third party verification in Somalia
Quality assurance of monitoring information in
Eastern Burma
Experiences presented are based on of
work of UNICEF (Somalia) and International
Rescue Committee and the Border
Consortium (Thailand/Burma)
7. Somalia – Third Party Verification (1)
First Level: Information from partners and
networks
Implementing Partner Reports are main source of
primary performance data
Reviewed against:
Previous track record of partner / confidence level
Specific concerns about partner performance
Reporting complete and realistic?
Specific issues flagged in or apparent from
requiring follow-up
Comparison to occasional information from staff
contacts in the filed (email, telephone)
8. Somalia – Third Party Verification (2)
Second Level: Third Party Verification
Flagged issues are scheduled for third party
monitoring
3rd Party Systems use field monitors that are
not affiliated with any implementing partner
often outside of the aid business
And therefore
can move more freely with less risk
are not as qualified to judge details of implementation
used mostly for verification of easily obtainable information
Third party monitors, are ‗blind‘ tasked to avoid
fabrication of reports
Were successfully tasked to track leakage of relief
goods into markets, including quantities and pricing.
9. Somalia – Third Party Verification (3)
Third Level: Follow up on concerns from Level 1 and 2
Third party information is kept confidential and assessed for
the risk level of a particular performance issue.
Depending on risk level, issues are taken up with the
partner:
without revealing source
e.g. dedicated open monitoring mission at next opportunity
Main reasons for staggering:
First level flags issues, but is in itself not sufficient for
reliable data
Second level is costly, and can be targeted to only flagged
issues
Third level is costly and not timely, and should only be used
with knowledge of problems
Low-level third party networks have worked well in
Somalia for other purposes, e.g. for food price monitoring
10. Eastern Burma – Strengthening
Monitoring Quality (1)
Correlating data in geographical information
systems
Main limitation to data correlation / triangulation is
sector-based systems (Health, Education, Relief
information management systems)
Sector IMS are basis for cross-sectoral GIS solution
Platform maps service delivery data of all sectors to
village location
Allows analysis of performance data from all sectors
per location
Allows sectors to engage in cross-monitoring and
data sharing
11. Eastern Burma – Strengthening
Monitoring Quality (2)
Regular surveys are expanded in scope
and feed into the information system
Key strength of the Eastern Burma
programmes is history of conducting regular
(sector) surveys.
All partners are now supporting the expansion
of the scope and the coverage of these
surveys.
Example: Annual Poverty Survey
12. Eastern Burma – Strengthening
Monitoring Quality (3)
Increased linkages between implementing
partners, and improved M&E capacity
Effortsin recent years to connect local
organizations across sector and ethnic group
Discussion on cross-monitoring
Comprehensive M&E training for local groups
Improved community feedback/village
monitoring (also used by Oxfam in Somalia
and Tearfund in Afghanistan)
13. Eastern Burma – Strengthening
Monitoring Quality (4)
Post-facto review of health centre
logbooks and patient files
Simple but innovative example:
Can be done remotely and is not time
sensitive
Reveals substantial information about quality
of support and services
Initiated by the IRC, and now being expanded
to whole sector
Variations conceivable for other sectors
14. Eastern Burma – Strengthening
Monitoring Quality (5)
Increased use of photographic and video
evidence of implementation
Where possible, local NGOs use
photography
video
to document their work.
Use of video started with success by one local
NGO
Provides better representation e.g. of
trainings and public awareness activities.
15. Lessons learned, and looking towards
the future
Similar contexts, different data environments
Good examples of when similar contexts warrant
different approaches
Both contexts are changing rapidly – more access in
Somalia, political reforms in Burma
Improved monitoring systems instill long-term effects
to adapt to these changes—stronger information,
increased local capacity
Changes in Somalia may make Eastern Burma
lessons applicable in near future
Experience in Somalia valuable for similar situations
elsewhere, e.g. Syria
16. Thank you!
For further questions:
Mona Fetouh - fetouh@un.org
Volker Hüls - volker@makingaidwork.com
Christian Balslev Olesen -
christianbalslev@gmail.com