Webinar organized the CGIAR Research Program on Policies, Institutions, and Markets, led by IFPRI, on July 13, 2021. Presentations: - Are we done yet? Response fatigue and rural livelihoods (Sylvan Herskowitz, Research Fellow, IFPRI) - Assessing response fatigue in phone survey: Experimental evidence on dietary diversity in Ethiopia (Kibrom Abay, Research Fellow, IFPRI) - Telescoping causes overstatement in recalled food consumption: Evidence from a survey experiment in Ethiopia (Kalle Hirvonen, Senior Research Fellow, IFPRI) Discussant: Andrew Dillon, Clinical Associate Professor of Development Economics within Kellogg's Public-Private Interface Initiative (KPPI); Director of Research Methods Cluster in the Global Poverty Research Lab, Northwestern University. Moderator: Kate Ambler, Research Fellow, International Food Policy Research Institute (IFPRI). More info and full recording: https://bit.ly/2TrpaNF

1. W E B I N AR
Measuring employment and consumption
in household surveys: Reflections from
three survey experiments
July 13, 2021 / 10:00 – 11:00 AM EDT
Presenters:
Sylvan Herskowitz (IFPRI)
Kibrom Abay (IFPRI)
Kalle Hirvonen (IFPRI)
Discussant:
Andrew Dillon (Northwestern University)
Moderator:
Kate Ambler (IFPRI)
Photo:
ILRI/Camille
Hanotte

2. Are We Done Yet?
Response Fatigue and Rural Livelihoods
PIM Webinar
Kate Ambler1 Sylvan Herskowitz1 Mywish Maredia2
1Markets, Trade, and Institutions Division
International Food Policy Research Institute
2Department of Agricultural, Food, and Resource Economics
Michigan State University
July 13, 2021
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 1 / 12

3. Response Fatigue and Rural Livelihoods
• 79% of the world’s poor live in rural areas (UN)
• Effective policy relies on understanding peoples’ livelihood strategies.
i. Employment for rural youth
ii. Feminization of agriculture
iii. Employment in agricultural value chains
• Heavy reliance on household surveys.
- Survey design could impact data quality
- May induce biases
• This paper looks at whether response fatigue...
a) ...exists and is economically meaningful....
b) ...has differential impacts for different groups...
c) ...the mechanisms driving these differences.
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 2 / 12

4. Collecting Labor Data in HH Surveys
Response Fatigue:
Deterioration of data quality over duration of interview.
• Surveys can be long/arduous spanning many hours (if not days!)
• Primary Respondents - HoH or member w/ senior standing:
i. Provide a household roster.
ii. Serve as proxies, answer questions about other members.
• We zoom in on fatigue induced within the labor module.
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 3 / 12

5. Standard Labor Modules
Common sequence of labor questions (based off LSMS):
1. Did you do any productive activities over the last year?
→ ... follow-up questions...
2. Did you do any other productive activities over the last year?
→ ... follow-up questions...
3. In the last week did you do any productive activities?
→ ... follow-up questions...
0-3 productive activities listed
Primary respondent:
• Answers about self, then repeats for each household member.
• Learns (quickly): report fewer activities → finish sooner
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 4 / 12

6. Setting - Rural survey in Northern Ghana
• Sample in Northern Ghana
• 1,117 households
• 4,817 non-respondents in labor
module
• Reported Labor Activites:
- 72% household on farm
- 14% wage work, 14% HH business
0
1
2
3
4
5
Density
0 1 2 3
Total Jobs Recorded
Total Jobs Recorded
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 5 / 12

7. Empirical Strategy
1. Survey Intro:
- Respondent provides HH roster → non-random order
2. Labor section:
- Respondent asked about other HH members → randomized order
3. Analysis:
- Exclude respondent: Not random, self-reports
- Controls: Household fixed effects, relation, gender, age, student status.
Causal effect of later response position on reported number of labor activities ...
ie. effect of exposure to greater response fatigue.
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 6 / 12

8. Main Result and Heterogeneity
-.25
-.2
-.15
-.1
-.05
0
.05
Total
Jobs
Listed
2 3 4 5 6 7 8 9 10+
Randomly Assigned Order
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 7 / 12

9. Main Result and Heterogeneity
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 8 / 12

10. Quantifying/Modeling Aggregate Fatigue Losses
• Results shown were marginal effects of fatigue.
- But we may care about aggregate effects.
- Overall average losses of ∼8.3%
• Estimation for different groups depends on:
1. Marginal effect AND
2. Natural/listed order position
- Women: ∼0.3 positions later
- Youth: ∼1-1.7 positions later
• Calculate percent losses:
- Order effects only (uniform effects of fatigue)
- Order + differential effects
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 9 / 12

11. Quantification of Impacts - by Gender
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 10 / 12

12. Quantification of Impacts by Age Group
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 11 / 12

13. Implications / Final Thoughts
1. Meaningful aggregate losses
- Likely a lower bound. Only within module variation.
2. Induces large biases - by gender and age
3. Two mechanisms:
i. Differential marginal impacts of fatigue
ii. Differential exposure (from roster construction)
4. Systematic ordering patterns in most household surveys (LSMS, IFLS, DHS)
5. Implications for other topics using similar module design
- e.g. consumption, expenditures, birth records, plot level inputs
Thank you!
Ambler, Herskowitz, and Maredia (IFPRI/MSU) Are We Done Yet? 12 / 12

14. Assessing Response Fatigue in Phone Surveys:
Experimental Evidence on Dietary Diversity in Ethiopia
Kibrom Abay, Research Fellow, IFPRI
(with Guush Berhane, IFPRI; John Hoddinott, Cornell; and Kibrom Tafere,
World Bank)
PIM Webinar| July 13, 2021

15. 1. Introduction
▪ The outbreak of pandemics and conflicts make monitoring welfare outcomes such
as food security particularly important.
▪ However, they also create substantial obstacles to face-to-face (FTF) interviews.
▪ This, has spurned interest in remote data collection tools including Computer-
Assisted Telephone Interviews (CATI).
▪ However, little is known about how this shift in the modality affects data quality
▪ The impact of fatigue is likely to be more pronounced in CATI for many reasons:
oThe cognitive burden and demand required for responding to survey questions
oEnumerators have limited control on the respondent and the interview
oIt is harder to control the interview environment in remote methods
▪ We evaluate the overall and differential impact of fatigue in phone survey

16. 2. Data and experimental design
▪ We use two rounds of phone surveys collected in June 2020 and December 2020.
▪ The primary respondent was the mother or caregiver of the young child.
▪ These build on previous FTF survey (2,551households in the August 2019 survey)
▪ In the first phone (CATI) survey, we reached out 1,497 households (59 percent)
▪ In December 2020, we interviewed 1,109 households (we lost those from Tigray)
▪ In our December 2020 survey, we introduced a randomized assignment of
respondents to one of two questionnaire types
▪ 50 percent of respondents were randomly assigned to receive the instrument on
women’s dietary diversity 15 minutes earlier in the interview.
▪ Mothers assigned to the control group were asked the same set of questions in the
middle of the interview, the same in the June 2020 survey.

17. 3. Summary statistics and descriptive results
▪ The August 2019 FTF survey contains detailed background information about the
sample households who were later interviewed in the phone (CATI) surveys.
▪ These serve two important purposes, mainly to:
oAssess the validity and balancing of the randomization
oFacilitate the identification of differential vulnerability to fatigue
▪ We also collected information on dietary diversity in the June 2020 CATI survey
▪ Almost all baseline characteristics and outcomes are balanced
oTreatment and control group mothers report statistically similar dietary
diversity score.
oBoth groups report consuming statistically comparable food groups.
▪ We also collected detailed information about the phone calls, including interview
date and time, number of call attempts made, and interviewer identifiers

18. Table 1: Balance of baseline characteristics
No
obs
Mean
Control
No
obs
Mean
Treatment
Mean
difference
Male headed household (dummy) 555 0.933 554 0.931 0.002
Age of household head(dummy) 555 37.286 554 37.827 -0.54
Education of household head (years) 555 3.616 554 3.599 0.017
Age of the mother (years) 551 29.216 553 28.429 0.787**
Education of mother (years) 555 3.117 554 3.255 -0.137
Fasting mother (dummy) 555 0.139 554 0.125 0.014
Age of the child (months) 555 30.773 554 31.191 -0.418
Household size 555 5.782 554 5.679 0.103
Livestock assets (TLU) 555 3.303 554 3.42 -0.117
Corrugated iron roof (dummy) 555 0.551 554 0.554 -0.003
Access to electricity (dummy) 555 0.427 554 0.403 0.024
Farm size (ha) 555 0.90 554 0.96 -0.06
Poor housing condition (dummy) 555 0.193 554 0.184 0.009
Food gap (in months) 555 2.485 554 2.558 -0.073
Food insecure household 555 0.773 554 0.756 0.017
Mothers’ dietary diversity (June 2020) 555 2.814 554 2.744 0.071
Mothers’ minimum dietary diversity (June 2020) 555 0.25 554 0.231 0.019
Mother consumed staples (June 2020) 555 0.944 554 0.926 0.018
Mother consumed animal source food (June 2020) 555 0.339 554 0.303 0.035
Mother consumed vegetable fruits (June 2020) 555 0.683 554 0.67 0.013
Children's dietary diversity (June 2020 survey) 554 1.939 554 2.002 -0.063
Amhara region 555 0.339 554 0.338 0.001
Oromia region 555 0.342 554 0.341 0.001
SNNP region 555 0.319 554 0.321 -0.002

19. 4. Empirical strategy
▪ We estimate a respondent fixed effects (FE) specification:
▪ Where 𝑌𝑚𝑡 stands for the dietary diversity outcomes of mother m in round t.
▪ Round is survey round indicator that takes value 1 for the December 2020 survey
and 0 for the June 2020 round.
▪ 𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑚𝑡 is equal to 1 for mothers’ receiving the dietary diversity module
early in the interview and 0 for those receiving the same module later.
o This variable takes the value 0 for all respondents in the baseline.
▪ 𝑋𝑚𝑡 stands for time-variant observable mother characteristics and interview
features
𝑌𝑚𝑡 = 𝛼ℎ + 𝛽0𝑅𝑜𝑢𝑛𝑑𝑡 + 𝛽1𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑚𝑡 + 𝛾𝑋𝑚𝑡 + 𝜀𝑚𝑡 (1)

20. 5. Results and Discussion: Main results
▪ Mothers in the treatment group report consumption of 0.25 more food groups
o This is equivalent to an 8.4 percent reduction in dietary diversity.
o Delay reduces women meeting a minimum of four-food groups by 28 percent.
Table 2: Impact of early placement on maternal diet diversity score, respondent fixed effects
estimates
(1) (2) (3) (4) (5) (6)
Diet
diversity
score
Diet
diversit
y score
Minimum diet
diversity
dummy
(five and
above)
Minimum diet
diversity
dummy
(five and
above)
Minimum
diet diversity
dummy
(four and
above)
Minimum diet
diversity
dummy
(four and
above)
Treatment: Early placement 0.229***
0.252***
0.022 0.025 0.072**
0.081**
(0.083) (0.083) (0.023) (0.023) (0.035) (0.034)
Round -0.029 -0.127 -0.022 -0.030 -0.002 -0.053
(0.065) (0.080) (0.017) (0.020) (0.027) (0.033)
Controls No Yes No Yes No Yes
Interview day No Yes No Yes No Yes
Enumerator fixed effect No Yes No Yes No Yes
Mean of dependent variable 2.985 2.985 0.090 0.090 0.292 0.292
R-squared 0.01 0.08 0.00 0.06 0.01 0.06
No. observations 2,234 2,234 2234 2234 2,234 2,234
Notes: Controls include a dummy variable indicating whether the mother was fasting, duration of interview, time of
interview, a dummy variable if interview was conducted in the afternoon, and the number of call attempts. Interview

21. ▪ Fatigue may entail differential impact by food groups depending on how
frequently a given food group is consumed.
▪ Delaying the module leads to 8.6 pp reduction in probability of reporting
consumption of ASF (40 percent decrease in the share of mothers)
Table 3: Fatigue effects on dietary diversity of mothers, by food groups, respondent fixed
effects estimates
(1) (2) (3) (4) (5) (6)
Staples,
beans and
nuts
Staples,
beans and
nuts
Animal
source
foods
Animal
source
foods
Vegetables
and fruits
Vegetable
s and fruits
Treatment: Early placement -0.014 -0.019 0.081**
0.086***
0.092***
0.086***
(0.012) (0.012) (0.032) (0.031) (0.030) (0.029)
Round 0.029***
0.031***
-0.032 -0.041 -0.013 -0.053*
(0.010) (0.012) (0.024) (0.029) (0.025) (0.028)
Controls No Yes No Yes No Yes
Interview day No Yes No Yes No Yes
Enumerator fixed effect No Yes No Yes No Yes
Mean of dependent variable 0.981 0.981 0.216 0.216 0.755 0.755
R-squared 0.01 0.06 0.01 0.06 0.01 0.07
No. observations 2,234 2,234 2,234 2,234 2,234 2,234
Standard errors clustered at the EA level in parentheses: *
p < 0.10, **
p < 0.05, ***
p < 0.01.

22. Why underestimation?
▪ One intuitive explanation relates to respondent incentives and responses to
lengthy interviews.
▪ How do respondents handle lengthy listing exercise or “yes/no” items?
▪ For instance, if each additional question involving “yes” response is perceived to
follow-up questions this may encourage fatigued respondents to respond “no”.
▪ This type of pattern is likely to be more so, for less frequent items or less
important plots or workers.
▪ Lengthy interviews may also lead to lack of attention and respondents may tend
to respond “no” to a question they have not grasped

23. 5.2 Heterogenous impacts by respondent characteristics
▪ Relatively older mothers suffer from significant response fatigue.
▪ Respondents with lower level of education are more vulnerable to fatigue.
▪ Mothers in larger households are more vulnerable to response fatigue
▪ Table 4: Heterogeneous effects of treatment on mothers’ diet diversity, respondent fixed effects estimates
(1) (2) (3) (4) (5) (6) (7) (8)
Maternal age Maternal
education
Household size Household wealth
Below
median
Above
median
Below
median
Above
median
Below
median
Above
median
Below
median
Above
median
Treatment: Early placement 0.060 0.514***
0.213*
0.134 0.065 0.462***
0.173 0.378***
(0.115) (0.135) (0.113) (0.123) (0.119) (0.127) (0.106) (0.132)
Round -0.051 -0.242**
-0.194*
0.050 -0.010 -0.247**
-0.118 -0.183
(0.109) (0.110) (0.103) (0.121) (0.117) (0.104) (0.112) (0.124)
Controls Yes Yes Yes Yes Yes Yes Yes Yes
Interview day Yes Yes Yes Yes Yes Yes Yes Yes
Enumerator fixed effect Yes Yes Yes Yes Yes Yes Yes Yes
Mean of dependent variable 2.979 2.991 2.832 3.167 2.945 3.023 2.786 3.194
R-squared 0.11 0.10 0.07 0.13 0.09 0.11 0.08 0.11
No. observations 1,170 1,050 1,223 1,001 1,098 1,136 1,146 1,088
Standard errors clustered at the EA level in parentheses: *
p < 0.10, **
p < 0.05, ***
p < 0.01.

24. 6. Concluding Remarks
▪ Delaying the timing of mothers’ food consumption module by 15 minutes leads to:
o 8-17 percent reduction in the dietary diversity score
o 40 percent decrease in the number of mothers who report consumption of ASF
Implications:
▪ Comparisons of statistics may be confounded by placement of modules
▪ Response fatigue exhibits a systematic pattern and will introduce non-classical
measurement
▪ Important trade-offs between volume of information collected and ensuring the
quality of data

25. Telescoping Causes Overstatement in
Recalled Food Consumption
Evidence from a Survey Experiment in Ethiopia
Kalle Hirvonen
International Food Policy Research Institute
Co-authors: Gashaw T. Abate, Alan de Brauw,
John Gibson, Abdulazize Wolle

26. What is telescoping?
▪ We often ask survey respondents to recall things that occurred in the past
oShock in the past 12 months
oWhether children had diarrhea in the past 14 days
oFood and non-food consumption (over past seven days typically)
▪ Question is whether respondents recall these accurately:
oForward telescoping: recalling more distant events as occurring more
recently.
oBackward telescoping: pushing recent events further back in time.

27. Why might telescoping matter?
▪ Measures of poverty and hunger require accurate
measurement of food consumption data.
▪ Might overstate overall value of consumption if
respondents systematically telescope in consumption
of some goods
oIf large enough, can then understate poverty
incidence or depth
oAlso food security/insecurity may be mismeasured
▪ This is a well-known problem but surprisingly few
attempts to quantify the degree of telescoping error
▪ Potential solution to telescoping is to use bounded
recall: “Since our last visit…”

28. Our experiment
▪ Conducted as part of an RCT endline in Addis Ababa in January-February
2020
▪ Sample includes 890 households, roughly representative of Addis Ababa
▪ Cross-randomized survey experiment with randomized trial as to not affect
results of RCT/results of survey experiment
▪ In “bounded” group- we sent a survey supervisor for a short visit precisely
7 days before the actual interview- just to announce they would come back
for an interview in 7 days
oWore a uniform so visit would be memorable
▪ 128 food item list in consumption survey (both bounded and unbounded)
▪ The two groups had similar characteristics and consumption levels in the
baseline survey conducted 3-4 months prior to our survey experiment

29. Distribution, Food Consumption, by Bounded/Unbounded
~16 percent
higher consumption
in unbounded recall,
on average

30. Further results
▪ Consumption of calories higher by 9 %, but 16 % for proteins in
unbounded group.
▪ The differences are larger for foods that are less frequently consumed,
particularly meat products.
▪ Overstates food security:
oHousehold Diet Diversity Score: 3 % increase.
oFood Consumption Score: 6 % increase.

31. Discussion
▪ Additional cost of bounded data collection was ~$3.50/household
▪ Big implications for less frequently consumed foods
▪ Can affect poverty computations as consumption overstated; however,
calculations of poverty lines may also be affected
▪ More research needed to see whether similar errors occur in rural areas
where costs of 2nd visit may be much higher but diets less diverse

32. Measuring employment and consumption in
household surveys:
Comments on three survey experiments
.
Andrew Dillon

33. Context
• De Weerdt et al. (2020) review the survey experiment literature which has
mostly focused on consumption and labor supply.
– Good discussion of identification challenges, objective standards, and
motivation for why measurement matters for research and policy.
• Carletto et al. (forthcoming) reviews measurement error and coverage bias
in agricultural data collection including innovations in survey methods.
• Dillon et al. (forthcoming) reviews innovation in agricultural questionnaire
design, updating Reardon and Glewwe (2000), and focuses on how shifts in
unit of analysis and agricultural panels have altered design choices.
2
Copyright or confidentiality statement.

34. Contributions
– What do we learn (for Ethiopia and Ghana)?
• Including dietary diversity modules earlier increases
dietary diversity by 0.25 food items.
• Later listed individuals report fewer jobs (~9%),
especially larger effects for women and youth.
• Telescoping increases consumption by a full day.

35. Contributions
– How does this change what we do in the field?
• Listing of household members often linked to hh roster listing.
– Does intentionally asking about certain household members in
other order or by job category ‘improve’ job reporting?
• Prioritize module ordering to reduce measurement error
– Which modules are the most sensitive to order effects?
• Assess feasibility of higher frequency consumption measurement,
especially where high seasonality.
– Tradeoffs with diaries, monitoring, consumption list items, etc.

36. Frontiers in Survey Experiments
• One size does not fit all
– Unpacking heterogeneity due to respondent characteristics
(observable and unobserved)
• Beware pairwise comparisons…objective truth is elusive.
• Methods are the hero that we didn’t know we needed….
– Telephone survey integration in response to COVID-19
– Household, remote sensing and telephone survey
interoperability will improve coverage, increase statistical power
and lower costs.
5
Copyright or confidentiality statement.

37. Thanks!
6
Copyright or confidentiality statement.