This document summarizes an analysis of metadata from the CommCare mobile data platform, which is used by hundreds of humanitarian programs worldwide. The analysis examined CommCare usage data to better understand how frontline workers and programs utilize the technology. Key findings included: 1) Frontline workers develop proficiency quickly in their first year but then level off; 2) Worker activity levels remain relatively stable month-to-month; 3) Worker productivity does not follow a normal distribution. The metadata analysis can help programs monitor performance and identify factors that lead to improvement over time.
Mobile technology Usage by Humanitarian Programs: A Metadata Analysis
1. MOBILE TECHNOLOGY
USAGE BY HUMANITARIAN
PROGRAMS:
A METADATA ANALYSIS
Rashmi Dayalu
O P E N
D A T A
S C I E N C E
C O N F E R E N C E_
BOSTON 2015
@opendatasci
2. 1
Mobile technology usage by humanitarian programs:
a metadata analysis
Open Data Science Conference
May 31, 2015
Rashmi Dayalu
Data Scientist
Dimagi, Inc.
3. 2
#ODSC
We are called to work “where our greatest passion
meets the world's greatest need.”
- Frederick Buechner
12. 11
#ODSC
Open source mobile technology platform
Does not require software developers to configure or deploy mobile
applications
Can be used on feature phones, androids, tablets, on the web or over
SMS
Image: http://www.ictedge.org/projects/zeprs
X
13. 12
Data collection Client Counseling
Case management and workflow supportTraining reinforcement and supervision
The result? Stronger healthcare workers
and stronger communities…
#ODSC
14. 13
#ODSC
There are CommCare users in over 40 countries
4001 – 5000
3001 – 4000
2001 – 3000
1001 – 2000
1 - 1000
16. 15
#ODSC
• CommCare’s cloud server hosts data from hundreds of
humanitarian programs.
• We are using CommCare metadata to ask a variety of questions
that can aid programs and FLWs in their goals.
http://noble1solutions.com/wp-content/uploads/2014/06/what-is-big-data.jpg
17. 16
#ODSC
BREADTH
D
E
P
T
H
Program A Program B …. Program “X”
FLW 1
FLW 2
FLW 3
…
FLW “N”
How do programs and FLWs perform across the board?
Ismyprogramperformingwell?
AremyFLWsperformingwell?
22. 21
#ODSC
Cumulative # form submissions and # new cases registered with Commcare:
all programs, 2010 - 2014
23. 22
#ODSC
1. How quickly do FLWs develop proficiency
with CommCare?
Analysis by: Jeremy Wacksman
24. 23
#ODSC
We looked at 634 workers who used CommCare for at least one year and
were active for at least 10 months of their first year.
Q1
Q2
Quarterly
range
Median
change
Q1 – Q2 + 22.9%
Q2 – Q3 + 1.9%
Q3 – Q4 + 0.0%
Q3
Q4
26. 25
#ODSC
Intra-user consistency:
• After the 6 month adoption period, do FLWs
maintain stable levels of CommCare activity?
• We calculated the Pearson correlation coefficient for
all pairs of consecutive calendar months for
individual FLW activity levels
28. 27
#ODSC
Programs can use the hypothesis of intra-user consistency to
monitor unexpected changes in FLW activity levels:
e.g. (1) Are FLWs less active during certain seasons or
months of the year?
N = 5,303 monthly observations (from health programs in India)
29. 28
#ODSC
(2) Do FLWs show decreased activity levels prior to
attrition in CommCare activity (inactivity >= 90 days)?
N = 252 FLWs with at least one CommCare attrition event
30. 29
#ODSC
3. Do FLW activity levels follow a bell curve?
Analysis by: Mengji Chen
31. 30
#ODSC
1. Boyle, E., Aguinis, H. “The Best & the Rest: Revisiting the Norm of Normality and Individual Performance”, Personnel
Psychology, 2012, 65, 79-119.
2. Image from: http://www.marin.edu/~npsomas/Lectures/Ch_1/Section_03.htm
Normal distributions are the most commonly held assumption in performance
metrics1. Is this assumption valid for CommCare FLWs?
33. 32
#ODSC
Boyle, 2012. Personnel Psychology
Programs have larger number of FLWs that are either underperforming or hyper-
performing.
Workloads, performance ranking, training and compensation cannot assume the
norm of normality.
34. 33
#ODSC
4. Do FLWs use CommCare in real-time while
interacting with their clients?
35. 34
#ODSC
There is no way to confirm real-time data collection by FLWs using
metadata, but we can flag visit data that was unlikely to have been
entered in real-time:
1. Batch entry – visits entered consecutively in quick succession (e.g.
with < 10 minutes between visits)
2. Visit duration (e.g. < 1 minute)
3. Visit time of day (e.g. visits started at night, between 6pm – 6am)
36. 35
#ODSC
Batch visits (%) by program
Proportion of batch visits from 30 maternal and child health programs worldwide
37. 36
#ODSC
Programs with unexpectedly large daily visit volumes revealed that:
(1) Patient data was often uploaded automatically via CommCareHQ -
CommCare’s web interface (e.g. maternal registrations)
38. 37
#ODSC
(2) Manual batch entry might actually be part of regular work flow for FLWs in
clinical settings (e.g. immunizations, child anthropometrics, etc.)
We looked at batch entry rates for 9 programs that had at least one “travel visit”
component built into their apps.
39. 38
#ODSC
In conjunction with batch entry, visit duration and visit time of day can be used to
flag visit data that was unlikely collected in real-time.
Visit duration (Mood’s Median Test):
Batch visits for programs A, B and C combined were ~half the duration of non-batch
visits (median duration of batch visits = 3.8 minutes, median duration of non-batch
visits = 7.7 minutes, Z = 5.35, p < 0.001).
Visit time of day (Chi-square Test):
Batch were more likely to have been recorded in the night
(% night non-batch visits = 16.5% and % night batch visits = 20.8%, 2 = 178.99, df = 1, p
<0.001).
41. 40
#ODSC
Sustainable use of CommCare is evidence for CommCare’s value. Of 306 programs,
how many were still active in Q4 2014?
Distribution of # programs by # active months and activity status in Q4 2014
42. 41
#ODSC
176 (57.5%) programs stopped using CommCare for at least 3 months. Of those,
43% restarted their CommCare usage, though restart rates are dependent on the
age of the program.
43. 42
#ODSC
Programs with more active FLWs were more likely to be active through 2014
This could mean that programs with smaller numbers of users have limited
resources and sometimes cannot continue their activities - regardless of how
effective CommCare is.
44. 43
#ODSC
6. Which programs are improving over time?
Algorithm developed by:
Dag Holmboe
Dimagi’s Data Science Advisor
Founder of Klurig Analytics
http://www.kluriganalytics.com
46. 45
#ODSC
Preliminary validation: Program #60
Performance feedback to FLWs in the middle of the year could have
contributed to the continued improvement (beyond first 6 months).
47. 46
#ODSC
Some future investigations:
1. Do 20% of FLWs submit 80% of the data?
2. Do programs that use supervisory tools have
the most active FLWS?
3. Is CommCare activity correlated with socio-
economic indicators (GNP, literacy rates,
corruption index, etc.)?
4. How do CommCare crashes affect user
behavior?
48. 47
Thank you!
For questions or research opportunities, please contact:
Rashmi Dayalu
rdayalu@dimagi.com
#ODSC
49. 48
#ODSC
Activity metric by FLW
per calendar month
Definition
1. # forms Total number of electronic forms submitted
2. # visits Total number of visits made to all cases
3. # cases Total number of unique cases visited (either registered
or followed up)
4. # cases registered Total number of unique cases registered
5. # cases followed-up Total number of unique cases followed-up
6. % of active days Percentage of days in the month during which the CHW
submitted data
7. Total duration of
CommCare use (min)
Cumulative time using CommCare, i.e. sum of all visit
durations
CommCare activity metrics - Aggregated by calendar month
per FLW
50. 49
#ODSC
Calculating breakpoints:
• Rolling window of mean FLW activity levels over the program
lifetime
• If window mean is at least 3 SD’s higher then the previous window, it
is a candidate breakpoint
• t-test of means between the windows confirms the statistical
significance of the breakpoint
Hinweis der Redaktion
Open source
Supports longitudinal tracking
Designed for low-literate users
Runs on Java & Android
Runs offline
Supports SMS
Has an app builder designed for non-programmers
Last year we had 37 self starter programs!