Monitoring and Evaluation Workshop for Grantees

Monitoring and Evaluation
Workshop for Grantees
City, Country
Dates

Session 1
Introduction to the Workshop
Adapted from: Module II M&E GBV Prevention and Mitigation Programs,
June 2009

• Introductions
• What do we have in common?
• Expectations and Learning Objectives
• Agenda

Workshop Objectives
By the end of this workshop, participants will be able
to:
• Differentiate between monitoring and evaluation
• Write/revise goals and smart outcomes for your VAW
program
• Demonstrate an understanding how to develop an
appropriate M & E plan for your program
• Identify criteria for indicator selection and information
sources for VAW indicators, and select appropriate
indicators for your program
• Discuss factors to consider when choosing an evaluation
design
• Demonstrate an understanding of the different methods
used to monitor and evaluate programs
Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009

Session 2
What is M & E?
What is the Purpose of M &E?
June 2009

Monitoring versus Evaluation
MONITORING =
Tracking changes in program performance over
time
EVALUATION =
Assessing whether objectives have been met
Assessing extent to which program is
responsible for observed changes

Illustration of Program Monitoring
Program
start
Program
end
TIME->
Program
indicator

Illustration of Program Impact
Program
start
Program
end
TIME->
Change
in
Program
Outcome
With program
Without program
Program
impact

Discussion: Is it Monitoring or Evaluation?
• The Ministry of Women’s Affairs wants to know if
programs carried out in Province A are reducing the
prevalence of intimate partner violence (IPV).
• The UN Trust Fund wants to know how many villages in
Region B have been reached with anti-VAW messages by
your program this year.

The Purpose of M & E
The purpose of monitoring and evaluation
is to measure program effectiveness
efficiency and to support evidence-
based decision making.

Is it monitoring? Is it evaluation?
• Were resources made available to the program in the quantities
and at the times specified by the program plan?
• Were the program activities carried out as planned?
• Which program activities were more effective and which were
less effective?
• Did the expected changes occur? How much change occurred?
• Can improved outcomes be attributed to program efforts?
• Did the target population benefit from the program and at what
cost?
Discussion:
Monitoring and Evaluation Questions

M&E Across Program Life Cycle
ASSESSMENT
What is the nature of the
problem?
EVALUATION
How Do I know that the strategy is working?
How do I judge if the intervention is making a
difference?
STRATEGIC PLANNING
What primary objectives should my
program pursue to address this problem?
IMPLEMENTATION/MONITORING
How do I know the activities are being implemented as
designed? How much does implementation vary from site
to site? How can be program become more efficient or
effective?
DESIGN
What strategy, interventions and
approaches should my program
use to achieve these priorities?
3
2
1
3
4
5

What is an M&E Plan?
An M&E Plan is a document that
describes a system which links strategic
information obtained from various data
collection systems to decisions that will
improve programs.

Functions of an M&E Plan
• Describes how the program is going to measure
achievements (ensure Accountability)
• Document consensus (encourage Transparency and
Responsibility)
• Guide M&E implementation (Standardization and
Coordination)
• Preserve Institutional Memory
M&E plan is a living document and needs to be adjusted when
necessary.

Elements of M&E Plan
1. Brief project description
2. Purpose(s) of M&E plan
3. Brief history of M&E plan development
4. Evaluation framework
5. Indicator system
6. Information system (Data sources)
7. Dissemination and utilization plan
8. Adjustments to M&E plan

Group Activity: M&E Plans
• Do you have M&E plans (or something
similar)?
• What are the components of the plans? What
sections do the plans include?
• Look at Handout 1: Template for M&E plan
• What is missing from your programs’ M&E
plans?
• What additional components do your
programs’ M&E plans include?

Session 3
Evidenced-based Programming
for VAW

What do we know about violence
against women?
• Around the world, at least one out of three
women is beaten, coerced into sex or
otherwise abused during her lifetime
• Women are most at risk at home and from
men they know, usually a family member
or spouse

Definition
“Any act of gender-based violence that
results in, or is likely to result in, physical,
sexual, or psychological harm or suffering for
women, including threats of such acts,
coercion, or arbitrary deprivations of liberty,
whether occurring in public or private life.”
United Nations General Assembly, 1993

Violence Against Women Includes:
• Physical violence
• Sexual violence
• Emotional violence
• Economic violence

Violence Against Women Includes:
• Violence that takes place in the family
– domestic abuse
– incest
• Violence that occurs in the community
– sexual assault
– sexual harassment at work and at school
• Violence that is organized or tolerated by governments

Violence against women begins early...
• female infanticide
• neglect, malnutrition
• child sexual abuse
• trafficking
• female genital
mutilation/cutting
• child marriage

And continues throughout the life cycle…
• physical and sexual partner abuse
• dowry and “honor” related violence
• coerced sex (within marriage or outside)
• forced prostitution & trafficking
• sexual harassment at work and at school
• rape as a weapon of war

Physical and sexual violence is extremely
common in women’s lives
Prevalence of physical or sexual violence against
women by anyone (partners and others), since age 15
years
0
20
40
60
80
100
B
angladesh
city
B
angladesh
province
B
razilcity
B
razilprovince
Ethiopia
province
Japan
city
N
am
ibia
cityPeru
city
Peru
provinceSam
oa
Serbia
and
M
onteneg..
Thailand
city
Thailand
province
U
nited
R
epublic
ofT...
U
nited
R
epublic
ofT...
percentage
Source: WHO Multi-country Study on Women’s Health and Domestic
Violence against Women, 2005

91
82
74
65 63 62 59 59 54
48 47 47
41 37
15
4
8
12
15 18 16 18 18
20 29 35
19 28
23
30
5 10 14
20 19 22 23 23 26 23 18
34 31
40
55
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Ethiopia
province
B
angladesh
province
Thailand
province
Thailand
city
U
.R
ep.Tanzania
province
B
angladesh
city
Japan
city
B
razil province
Serbia
&
M
ontenegro
city
N
am
ibia
city
Peru
province
U
.R
ep.Tanzania
city
Peru
city
B
razil city
Sam
oa
%
partner only partner and non-partner non-partner only
Women’s greatest risk of violence is
from a partner
Source: WHO Multi-country Study on Women’s Health and Domestic
Violence against Women, 2005

Domestic violence is prevalent in all countries,
but the levels can very greatly among settings
Japan and Serbia:
4% current violence
Bangladesh,
provinces in Ethiopia,
Peru and Tanzania:
30% to 54% current
violence

Prevalence of forced sexual initiation
48%
40%
37%
29% 28%
21%
19%
9%
7%
Female adolescents reporting forced sexual initiation, as % of those having
had sex (population-based surveys 1993-99)
Source: WHO’s World Report on Violence and Health, 2002

Violence against women is a violation of
human rights and a significant development
challenge
• UN Declaration on Violence against Women (1993)
• UN Human Rights Summit (Vienna, 1993)
• International Conference on Population and
Development (Cairo, 1994)
• Fourth World Conference on Women (1995)
• World Health Assembly resolution on violence as a
public health priority (1996)
• UN Secretary General’s In-depth Study on VAW
(2006)

Violence increases women’s
risk for …
 Fatal outcomes
 homicide
 suicide
 maternal deaths
 HIV/AIDS-related
deaths
 Non-fatal outcomes
 physical
 mental
 high risk behaviors
 reproductive health
For example:
 unwanted pregnancy
 chronic pain syndromes
 injury
 depression
 alcohol/drug use
 STDs/HIV
 gynecological disorders
 low birth weight

Violence against women
in the region
Group Exercise
• What are the most common forms of
violence against women and girls in the
region?
• What are the main barriers to
addressing them?

Key principles for evidence-
based programming on VAW

Guiding Principles in GBV
Programming
The underlying principle is “Do no harm”
• Respect survivors’ safety and autonomy
• Ensure relevance and appropriateness of interventions to
local setting
• Use a human rights perspective

Promising practices for addressing
VAW
• Increasing access to justice
• Providing support to survivors of
violence
• Prevention of VAW

Violence occurs at many levels…and
therefore must be addressed at many levels
Society
Community
Families/
household
Individual
woman/girlSociety
Laws that sanction VAW
and protect survivors
National plans on VAW
Media campaigns to change
gender norms
Address VAW in national
institutions (judiciary,
education, welfare, etc.)
Community based networks for
referrals and prevention
Multi-sector, integrated services
for survivors of VAW
Community mobilization to
change norms
Support for organizations that
protect rights of women
Engage men as
partners in prevention
VAW
Work with youth (boys
and girls) to promote
gender equity
Girls’ education,
Opportunities for economic
empowerment,
Information on civil/political rights
Increased mobility and autonomy

Improved policies and programs
– National Commissions and Plans against
Violence
– More services for victims of violence
(women’s police stations, NGO women’s
services)
– Policies and protocols for education and other
sectors

Promising practices in strengthening
women’s access to justice
• Improved laws on domestic and sexual violence
• Police procedures that require mandatory arrest of
abusers or “no drop” policies
• Protective/restraining orders that can be easily
accessed by women
• Specialized police stations for women
• Promoting coordination with traditional justice system

Lessons learned from the justice
system
• Train all justice personnel (judges, police, forensic
doctors, clerks, etc.); address attitudes as well as
skills
• Human and financial resources
• Multi-sectoral coordination
• Public education on how to use the laws
• Monitor implementation of the law to see if reporting
of violence and detention of abusers increases

Promising practices to increase
support for survivors
• Women’s safe houses or shelters
• Crisis centers (legal, medical, psychological
support – usually ngos.)
• One- Stop centers (all services in one place –
usually a hospital or police station)

Lessons learned from programs to
increase access to services
• Services should be integrated (with different types of
support in one place) and coordinated through
referral networks
• Specialized centers are mostly in urban centers and
may be too costly for rural areas
• Community support networks can provide safe haven
and support for survivors of violence in rural areas

Prevention of violence is key!
• Community mobilization
• Mass media / Communication for social change
programs
• Engaging men and boys in violence prevention
• Women’s economic empowerment

Communication, or “edutainment”
programs can help prevent violence
• Model new behaviors and ways
of thinking
• Create an enabling environment
for increased communication
about gender, sexuality
• Create synergies and enhance
interpersonal communication
• It is possible to document and
measure impact (GEM Scale)

Lessons learned for effective
communications campaigns
• “One-off” campaigns are less effective than ongoing
programs
• Multiple strategies and techniques and permanent
saturation of alternative messages should be used
• Importance of working with journalists to change the
way VAW is addressed in main stream media

Raising Voices and
CEDOVIP in Uganda
Promising practices for
community mobilization

Promising practices in engaging men
and boys as partners in violence
reduction
•
MenEngage.org

Promising practices in women’s
empowerment…
•Information and education
•Technologies and services
•Economic assets and income
•Social support and networks
•Leadership and political participation

Image Program in South Africa
Microfinancing and
training on violence,
together with
community
mobilization activities
reduced domestic
violence by 50% in
intervention group
over 2 years

Addressing VAW in schools
• Sexual harassment and violence are
widespread in schools, (both peers and
teachers) and an obstacle to girls educational
advancement
• Schools teach values around gender, conflict
resolution, sexual and reproductive health

48
Lesson Learned in Addressing
Violence in Education programs
• Anti violence and gender norms need to be
integrated into school curriculum, including
family life skills
• Address sexual violence in schools through
sexual harassment policies and codes of
conduct

We need to invest in evaluation!
• We need to know what works and what
doesn’t
• More rigorous designs and rethink success
indicators
• National level indicators to monitor overall
trends

References
• Barker, G., C. Ricardo, et al. (2007). Engaging Men and Boys in
Gender Equality: Evidence from Programme Interventions.
Geneva, World Health Organiztion: 70.
• Ellsberg, M. and L. Heise (2005). Researching violence against
women: A practical guide for researchers and activists. Geneva,
Switzerland, Program for Appropriate Technology in Health,
World Health Organization, .
• Morrison, A., M. Ellsberg, et al. (2007). "Addressing Gender-
Based Violence: A Critical Review of Interventions." The World
Bank Research Observer 22(1): 25-51.
• Michau, L. and D. Naker (2003). Mobilising Communities to
Prevent Domestic Violence. Kampala, Uganda, Raising Voices.
• Velzeboer, M., M. Ellsberg, et al. (2003). Violence against
Women: The Health Sector Responds. Washington, DC, Pan
American Health Organization, PATH.
• World Health Organization (2005). WHO Multi-Country Study on
Women's Health and Domestic Violence Against Women: Initial
Results on Prevalence, Health Outcomes and Women's
Responses. Geneva, Switzerland, WHO.

Session 4
Program Goals and Objectives
June 2009

Program Goals
A goal is a broad statement of a desired, long-term outcome of a
program. Goals represent general big picture statements of
desired results.
• Examples:
– To improve the lives of women who experience gender-
based violence
– To decrease injury and mortality due to gender-based
violence
– To eliminate gender-based violence

Program Objectives (Outcomes)
• Objectives: statements of desired, specific, realistic
and measurable program results
• SMART
– Specific: identifies concrete events or actions that will take place
– Measurable: quantifies the amount of resources, activity, or change
to be expended and achieved
– Appropriate: logically relates to the overall problem statement and
desired effects of the program
– Realistic: Provides a realistic dimension that can be achieved
with the available resources and plans for implementation
– Time-bound: specifies a time within which the objective
will be achieved
Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009: (GAP 2003)
NOTE: On UN Trust Fund logframes the “objective” is the “outcome”.

How to Write Program Objectives
(Outcomes)
To [action]
The
(what)
[specify what knowledge, attitudes, skills,
behaviors]
Among
(who)
[specific population or segment]
From – to [from baseline to desired level]
or by [ x percent ]
or to [specific level
By
(when)
[time frame]

Discussion: Is it a Goal or an
Objective (Outcome)?
Is it a goal or an objective? If an objective, is it
SMART? Why or Why not?
• To reduce violence against women
• To increase the % of men and women in beneficiary
population who believe that violence is not an acceptable
way of dealing with conflict from 40% in 2002 to 80% by
2007
• To increase the number of domestic violence clients
recommended for counseling who utilize counseling
services by 30%

Group Activity
In your groups:
• Choose two statements from the list on the
handout
• Decide whether the statement is a goal or an
outcome
• If a goal, is it properly stated? Why or why
not? If an outcome, is it SMART? Why or why
not?
• If an objective (outcome) is not SMART,
rewrite it in order to make it SMART.

Session 5
Conceptual Frameworks,
Logic Models and Logical
Frameworks
Adapted from: Module II M&E GBV Prevention and Mitigation
Programs, June 2009

Conceptual Frameworks
Diagram that identifies and illustrates the relationships between all relevant
systemic, organizational, individual, or other salient factors that may influence
program/project operation and the successful achievement of program or project
goals.
M&E Purpose:
• To show where program fits into wider context
• To clarify assumptions about causal relationships
• To show how program components will operate to influence outcomes
• To guide identification of indicators
• To guide impact analysis (causal pathways)

Illustrative Conceptual Model for IPV
Society
Community
Relationship
Individual
Perpetrator
•Norms
•Laws
•Notions of
masculinity
•Gender roles
•Poverty
•Unemployment
•Family
isolation
•Community
acceptance of
violence
•Marital conflict
•Male control of
wealth
•Male control of
decision-making
in family
•Being male
•Witnessing
marital violence
as child
•Being abused as
child
•Absent or
rejecting father
•Alcohol use Source: Heise 1998
Society

Logic Models
Logic models are diagrams connecting program
inputs to activities, outputs, outcome and impact
as they relate to a specific problem or situation.
Logic models show what resources your program
will need to accomplish its goals, what your
program will do, and what it hopes to achieve,
emphasizing the links between all these things.

Logic Model Components
• Input – Resources used in a program (e.g., money, staff,
curricula, and materials)
• Activity– Actions taken or work performed through
which inputs such as funds, technical assistance, and
other types of resources are mobilized to produce
specific outputs
• Output – Direct results of program activities (e.g., people
trained; materials distributed; couples counseled, etc.)
• Outcome – Program results at population level (e.g.,
knowledge, attitudes, behavior, coverage)
• Impact – Long-term change in morbidity, mortality, and
fertility
Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009 and UNAIDS MERG, 2008

Logic Model Shows Intended Results
6months – 1 year
After implementation
Inputs
Activities
(work plans)
Outputs
(Services)
Outcomes Goal
immediate immediate 5 years or more2 to 5 years
efficiency
effectiveness
indicators
Programme Level
Global Level
Source: UNIFEM RBM Curriculum: Module 3a & 3b:
The Results Chain; and The World Bank Global HIV/AIDS Programme and UNAIDS 62

If-Then Logic – Cause and Effect
Activity
Output
Outcome
Goal
then
If
If
If
if
if
if
then
then
Input if
then

64
INPUT
•Human and
financial
resources to
develop
training
materials &
implement
training
program
ACTIVITIES
•Develop
GBV training
curriculum
•Conduct
TOT
workshops
•Conduct
GBV training
for providers
OUTCOME
•Increased
awareness of
GBV
•Increased
disclosure of
GBV
•Increased
knowledge and
utilization of
GBV services
Illustrative Logic Model for
Provider Training Program
IMPACT
•Improved
well-being
and safety of
GBV
survivors
OUTPUT
•Providers
trained in GBV
•Improved
provider
attitudes toward
GBV
•Increased
provider ability
to identify,
counsel, care
for, and refer
GBV survivors

Role of Logic Model
• Program planning
– Helps one think through program strategy – where
you are & where you want to be
• Program management
– Helps one track and monitor operations to better
manage results
• Communication
– Shows stakeholders at a glance what program is
doing and what it expects to achieve

Role of Logic Model
• Consensus building
– Promotes common understanding about what
program is, how it works and what it is trying to
achieve
• Fundraising
– Can structure and streamline grant writing
• Monitoring and evaluation
– Provides thorough understanding of what
resources program has to work with, what it is
doing, and what it hopes to achieve.

67
Group Activity:
Identifying Logic Model Components
• Return to earlier groups
• Each of the scenarios in handout #3
corresponds to one of the five components in
the logic model:
– Input, activity, output, outcome, and
impact.
• In your groups, decide which component the
scenario illustrates.
• Discuss in plenary

Logical Frameworks
A logical framework (LogFrame) is a
tool for strategic planning, and
management of a program/project. It is
presented as a table and includes the
most pertinent information about the
project in a consise, logical and
systematic manner.

Logical frameworks
Logical frameworks answer the following
questions in a standardized way:
• What does the project aim to achieve?
• How will it be achieved
• What is needed to make the project a
success?
• How will progress and potential problems be
assessed?

Logical frameworks
• Summarize what the project plans to achieve and
why
• Summarize key hypotheses
• Summarize the results and products that were
monitored and evaluated

UNIFEM Definitions
Term Definition Example
Goal
The higher-order objective to which a
development intervention is intended
to contribute. The change is long-
term and achieved through collective
efforts of many partners.
Morbidity and mortality are reduced among
VAW/G survivors
Outcome
Intended or achieved short and
medium term effects of an
intervention’s outputs. Outcomes
represent changes in development
conditions which occur between the
completion of outputs and the
achievement of impact.
At least 75% of VAW/G survivors who
received appropriate care
Output
The products, services, and
capacities resulting from completion
of activities.
50 service providers trained to care for and
refer VAW/G survivors who demonstrate
increased skills
Activity
Actions taken or work performed
through which inputs are mobilised to
produce specific outputs.
Training of 50 service providers to care for
and refer VAW/G survivors
Source: UNIFEM (2005). Essential Guide to RBM; and UNIFEM RBM
Curriculum: Module 3a & 3b: The Results Chain 71

Project Team Group Work:
Developing a Logic Model/Framework
Part 1:
• Work in project teams
• Develop/revise a logic model for at least one of your program
outcomes
• Questions to address:
– Have you expressed your outcomes in terms of change?
– Do activities, outputs, and outcomes relate to each other
logically (the if-then relationship)?
– Does your program have adequate resources to implement
the activities and achieve the desired outcomes?
– Have you included all of the major activities needed to
implement your porgram and achieve the expected
outcome(s)
– Would the activities listed enable someone who is unfamiliar
with your program to understand its scope?

Project Team Group Work:
Developing a Logic Model continued
Part 2:
Compare the logic model to the logical
frameworks you have developed for your UN
Trust Fund program – What is the same?
What is different? How can creating a logic
model help when developing a logical
framework? What additional information is in
the logical framework?

Session 6
Choosing Appropriate Indicators for
Different Types of Interventions
June 2009

What is an indicator?
An indicator is a specific, observable and
measurable characteristic that can be
used to show changes or progress a
program is making toward achieving a
specific outcome.

Characteristics of Good Indicators
• Valid: accurate measure of a behavior, practice or task
• Reliable: consistently measurable in the same way by different
observers
• Precise: operationally defined in clear terms
• Measurable: quantifiable using available tools
and methods
• Timely: provides a measurement at time intervals relevant and
appropriate in terms of program goals and activities
• Programmatically important: linked to the program or to achieving
the program objectives that are needed for impact

Indicators do not specify a
particular level of achievement
-- the words “improved”,
“increased”, or “decreased” do
not belong in an indicator.

Common Indicator Metrics
• Counts
– # of legal and service organizations for VAW/G survivors
– # of women and children using VAW/G social welfare services
• Calculations: percentages, rates, ratios
– % of sites referring survivors of GBV with at least one service
provider trained to care for and refer survivors
– % of women aged 15-49 who have ever experienced physical
violence from an intimate partner
• Index, composite measures
– Gender-equitable men (GEM) scale
– Sexual relationship power scale
• Thresholds
– Presence, absence; pre-determined level or standard

Proxy Indicators
• Use a "proxy" indicator when you cannot directly
measure a result
• Result: Decreased trafficking of women and girls from
country X.
– Direct Indicator: # of women and girls moved across the border
against their will or under false pretences per year
– Proxy Indicator: # of women and girls reported missing / lost by
families in country x
– Proxy Indicator: # of arrests of persons operating trafficking rings
in country x
Source: Source: UNIFEM RBM Curriculum: Module 5: Indicators

Common Challenges in Indicator Selection
• Choosing an indicator that program activities
cannot affect
• Choosing an indicator that is too vague
• Indicators that do not currently exist and
cannot realistically be collected
• Selecting an indicator that does not accurately
represent the desired outcome
• Too many indicators

How Many Indicators Are Enough?
• No more than one or two indicators per key activity or
result
– (e.g., training, BCC)
– (ideally, from different data sources)
• Use a mix of data collection methods/source
• Use indicators that track change over the life of the
project

Factors to Consider When Selecting
Indicators
• Logic/link to framework
• Programmatic needs/information for
decision making
• Data availability
• Resources – HR capacity, data collection
forms, $$$
• External requirements (government,
donor, headquarters)
• Standardized indicators (if available)

Group Activity:
Selecting Program Indicators
• Return to groups from previous activities
• Select indicators that your group might use to measure progress
towards your program goals and outcomes with the following
considerations:
– To what extent are the indicators logically linked to your
logframe?
– Do the proposed indicators share the characteristics of good
indicators?
– How will the indicators be used for program decision-making
and for what decisions?
– Are data available to measure the indicators?
– Can the indicators be realistically collected given available
resources?

Where to Find Standardized Indicators for
GBV Programs
Bloom, S. 2008. Violence Against Women and Girls: A Compendium of Monitoring and
Evaluation Indicators. Ms.08-30. Chapel Hill, NC: Carolina Population Center, the
University of North Carolina and Chapel Hill.
http://www.cpc.unc.edu/measure/publications/pdf/ms-08-30.pdf
UN Division for the Advancement of Women (UNDAW), UN Economic Commission for
Europe (UNECE) and UN Statistical Division. 2008. Indicators to measure violence
against women. Report of the Expert Group Meeting, 8 to 10 October
2007, Geneva, Switzerland; UN Human Rights Council. 2008.
Report of the Special Rapporteur on violence against women, its causes and
consequences by Yakin Ertürk. Indicators on violence against women and State
response. 25 February 2008.
Friends of the Chair of the United Nations Statistical
Commission on the indicators on violence against women by the Secretary General. 11
September 2008.

Session 7
Indicators for VAW programs
June 2009

Illustrative Indicators:
Community Mobilization
• Proportion of people who would assist a woman
being beaten by her husband or partner
• Proportion of people who say that wife beating is
an acceptable way for husbands to discipline their
wives
• Proportion of people who agree that rape can take
place between a man and woman who are married
• Proportion of youth-serving organizations that
include training for beneficiaries on sexual and
physical VAW/G

Behavior Change Communication
• Proportion of people who have been exposed to
VAW/G prevention messages
• Proportion of girls who say that they would be
willing to report any experience of unwanted
sexual activity
• Proportion of girls that feel able to say no to
sexual activity
• Proportion of individuals who know any of the
legal sanctions for VAW/G

Service Delivery
• Proportion of rape survivors who received
comprehensive care
• Proportion of law enforcement units following a
nationally established protocol for VAW/G complaints
• Proportion of women who know of a local
organization that provides legal aid to VAW/G
survivors
• Number of women and children using VAW/G social
welfare services
Source: Bloom, S. 2008. Violence Against Women and Girls: A Compendium of Monitoring and
Evaluation Indicators.

Illustrative Indicators: Law and Public
Policy
• Number of law enforcement professionals
trained to respond to incidents of VAW/G
according to an established protocol
• Number of VAW/G complaints reported to the
police
• Proportion of VAW/G cases that were
investigated by the police
Source: Violence Against Women and Girls: A Compendium of Monitoring and Evaluation
Indicators. Ms.08-30. 2008

Activity: Matching indicators to
components in the logic model
• Return to small groups from previous
activities
• Look at the indicators that your group is
given by the facilitator and determine
what component of the logic model
each indicator fall into.
• Discussion: Which indicators fall where
and why?

Measuring Indicators
Indicator 1: # Number of teacher training programs that
include sexual and physical VAW/G in their
Curriculums
“trained” refers to attending every day of a five-day training
course on VAW/G

Measuring Indicators
Indicator 2: % Proportion of rape survivors who received
comprehensive care
Numerator: Number of rape survivors who received the
comprehensive package of care as defined by international
and national guidelines.
Denominator:
Record review: Total number of rape survivors who presented
at each facility under the program’s domain.
Exit interview: Total number of women interviewed.

Measuring Composite Indicators
• Example: How to measure, “attitudes toward
VAW” ?
• What question do you ask?
• How do you know any one question is a good
representation of general attitude?
• How do you scale each question (agree/disagree;
strongly disagree to strongly agree; yes/no)?
• How do you combine answers to the questions to
come up with one indicator?
• What value or weight do you assign to any one
question on GBV attitude in the survey?

Example of Measuring Attitudes Toward
Wife Beating in the DHS
• Sometimes a man is annoyed or angered by
things that his wife does. In your opinion, is a
man justified in hitting or beating his wife in
the following situations:
– If she goes out without telling him?
– If she neglects the children?
– If she argues with him?
– If she refuses to have sex with him?
– If she burns the food?

Project Team Activity:
Specifying Indicator Metrics
• Work in your project teams
• Look at the indicators that your group
has selected to measure progress
towards your goals and objectives
• Define the metrics for each indicator

Session 8
Quantitative Evaluation Designs

Guiding Questions for Choosing an
Evaluation Design
• What question does your program need to
answer?
• What do you want to measure (indicators)?
• How sure do you want to be? (VERY SURE =
GOOD )
– What is the cost of making a mistake (low,
medium, high)?
• When do you need the results? (FAST)
• How much are you willing to pay? (CHEAP)
• Has the program already started?

When monitoring and evaluating
VAW programs, the
confidentiality, privacy, safety,
and well-being of VAW survivors
must be top priority at all times.

Causality Requirements
• A (program) precedes B (outcome)
• B (outcome) is present only when A
(program) is present
• We can rule out all other possible
causes of B (outcome).

The Basic Experimental Principle
• The intervention is the only difference
between two groups
• This is achieved by random assignment

An Experimental Design
Experimental group
Control group
O1 X O2
O3 O4
RA

Factors that May Lead Us to Make
Invalid Conclusions
• Dropout: There may be loss to follow-up.
• Instrumentation effects: Occur when a
questionnaire is changed between pretest and
posttest.
• Testing effects: Occur because study participants
remember questions that were asked of them at
pretest and perform better at posttest because
they are familiar with the questions.

A Second Experimental Design
Experimental group
Control group
X O2
O4
RA

A Non-Experimental Design
Experimental
group
O1 X O2
Time
In this method of evaluation, only people who are
participating in the program get the pre- and post-test
Steps
1. Pre-test everyone in the program.
2. Deliver the intervention.
3. Post-test the same individuals.

A Non-Experimental Design-
Weaknesses
• Does not provide any information about
what kinds of results might have occurred
without the program and is the weakest in
terms of scientific rigor
• History effects: These occur when
extraneous events (events that occur
outside the study) influence outcomes that
are measured by a study.

A Second Non-Experimental Design:
Time Series
Experimental group O1 O2 O3 X O4 O5 O6
Time
Data are collected (typically using a
survey) multiple times - before,
during, and after a program

Time Series Design
Steps:
1. Select a program outcome measure that can be
used repeatedly.
2. Decide who will be in the experimental group.
Will it be the same group of people measured
many times, or successive groups of different
people?
3. Collect at least three measurements prior to the
intervention and made at regular intervals.
4. Check the implementation of the intervention.
5. Continue to collect measurements at least
through the duration of the program.

A Quasi-Experimental Design
Experimental group O1 X O2
Time
O3 O4Comparison group
---------------------------------
Two groups which are similar, but which were not
formed by random assignment, are measured both
before and after one of the groups gets the program
intervention

Summary Features of Different Study Designs
True experiment Quasi-experiment Non-experimental
Partial coverage/
new programs
Control group
Strongest design
Most expensive
Partial coverage/ new
programs
Comparison group
Weaker than
experimental design
Less expensive
Full coverage programs
--
Weakest design
Least expensive

I. Non-experimental (One-Group, Post-Only)
II. Non-experimental (One-Group, Pre- and Post-
Program)
IMPLEMENT PROGRAM ASSESS TARGET GROUP AFTER PROGRAM
ASSESS TARGET
GROUP BEFORE
PROGRAM
IMPLEMENT PROGRAM
ASSESS TARGET GROUP
AFTER PROGRAM

III. Experimental (Pre- and Post-Program with
Control Group)
RANDOMLY
ASSIGN PEOPLE
FROM THE SAME
TARGET
POPULATION TO
GROUP A OR
GROUP B
TARGET
GROUP A
CONTROL
GROUP B
ASSESS
TARGET
GROUP A
IMPLEMENT
PROGRAM
WITH TARGET
GROUP A
ASSESS
TARGET
GROUP A
ASSESS
CONTROL
GROUP B
ASSESS
CONTROL
GROUP B

IV. Quasi-Experimental (Pre- and Post-Program
with Comparison Group but not randomized)
ASSESS TARGER
GROUP BEFORE
PROGRAM
IMPLEMENT PROGRAM
ASSESS TARGET
GROUP AFTER
PROGRAM
ASSESS
COMPARISON
GROUP BEFORE
PROGRAM
ASSESS
COMPARISON
GROUP AFTER
PROGRAM

Summary Features of Different Study Designs-ctd
• The different designs vary in their capacity to
produce information that allows you to link
program outcomes to program activities.
• The more confident you want to be about making
these connections, the more rigorous the design
and costly the evaluation.
• Your evaluator will help determine which design
will maximize your program's resources and
answer your team's evaluation questions with the
greatest degree of certainty.

Important issues to consider
when selecting a design
• Complex evaluation designs are most costly, but allow
for greater confidence in a study's findings
• Complex evaluation designs are more difficult to
implement, and so require higher levels of expertise in
research methods and analysis
• Be prepared to encounter stakeholder resistance to the
use of comparison or control groups, such as a parent
wondering why one group of individuals will not receive a
potentially beneficial intervention
• No evaluation design is immune to threats to its validity;
there is a long list of possible complications associated
with any evaluation study. However, your evaluator will
help you maximize the quality of your evaluation study

Activity: Choosing a study design
Return to your small groups from previous activities.
Take 20 minutes to discuss the following questions
using the examples below:
1. What outcome indicators would you use?
2. What study design would you choose ?
Example 1: A family planning service wishes to determine if
carrying out a screening and referral program for violence will
lead to more women being able to successfully use family
planning methods.
Example 2: What study design would you recommend to test the
hypothesis that women who are offered counseling services for
violence during family planning counseling sessions are more
likely to use family planning than women are not offered
services?

Evaluation Questions
• Are the services available?
• Are services accessible?
• Is the quality of services provided
adequate?
• Are the services being used?
• Is the target population being reached?
• Were there improvements in patterns of
violence or behaviors?

Adequacy Assessment
• Adequacy studies only describe if a condition
is met or not
– Typically addresses provision, utilization or
coverage aspects. No need for control,
pre/post data in such cases
• Hypothesis tested: Are expected levels
achieved?
– Can also answer questions of impact
(magnitude of change) provided pre/post
data is available
• Hypothesis tested: Difference is equal
or greater than expected

Features of Adequacy Assessment
• Simplest (and cheapest) of evaluation models, as it
does not try to control for external effects. Data
needed only for outcomes
• If only input or output results are needed, then the
lack of controls is not a problem
• When measuring impact, however, it is not possible
to infer that the change is due to the program due to
lack of controls.
• Also, if there is no change, it will not be possible to
say whether the lack of change is due to program
inefficiency or if the program has impeded a further
deterioration

Adequacy Assessment Inferences
• Are objectives being met?
– Compares program performance with previously
established adequacy criteria
– No control group
– 2+ measurements to assess adequacy of change over
time
• Provision, utilization, coverage
– Are activities being performed as planned?
• Impact
– Are observed changes in of expected direction and
magnitude?
• Cross-sectional or longitudinal
Source: Habicht, Victora and Vaughan (1999)

Group Discussion
• What are the advantages of adequacy evaluations?
• What are the limitations of adequacy evaluations?
• If an adequacy evaluation shows a lack of change in
indicators, how can this be interpreted?
• Which of the study designs discussed earlier can be
used for adequacy evaluations?

References
• Adamchak S et al. (2000). A Guide to Monitoring and Evaluating
Adolescent Reproductive Health Programs. Focus on Young
Adults, Tool Series 5. Washington D.C.: Focus on Young Adults.
• Fisher A et al. (2002). Designing HIV/AIDS Intervention Studies.
An Operations Research Handbook. New York: The Population
Council.
• Habicht JP et al. (1999) Evaluation designs for adequacy,
plausibility and probability of public health programme
performance and impact. International Journal of Epidemiology,
28: 10-18.
• Rossi P et al. (1999). Evaluation. A systematic Approach.
Thousand Oaks: Sage Publications.

Session 9
Quantitative Methods and Tools
June 2009
Researching Violence Against Women: A Training Course for Researchers
and Activists, 2005

Types of Information Sources
• Quantitative data
– Useful for tracking trends accurately and
highlighting differences
• Qualitative data
– Useful for understanding the context in
which the trends/differences occur and to
interpret quantitative data accurately (e.g.
focus groups; in-depth interviews)

Quantitative Information Sources
• Population-based surveys
– Cross-sectional surveys
• DHS (domestic violence module); VAW survey
• Facility surveys
– Client exit interviews; provider surveys; observing
service delivery
• Program statistics
• Crime statistics (of poor quality in many LDCs)
• Behavioral surveillance systems

Group Activity:
Advantages and Disadvantages of
Quantitative Data Sources
• Return to groups from previous activities
• You will be given one of the data sources
from the previous slide
• Discuss:
– What are the advantages of using the
sources of information?
– What are the disadvantages of using this
source of information?
– Have your programs used this type of
information before? In what way?

Triangulating Data Sources: Soul City Source: Module II M&E GBV
Prevention and Mitigation Programs, June 2009
Individual
1. National Survey
Community
Society
2. Sentinel Site Studies
5. Media Monitoring
and Analysis
4. NNVAW Partnership
Study
6. Cost – Effectiveness Study
3. (National) Qualitative
Impact Assessment

Ethical and safety issues
Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
• Ethical Review Process
• Prioritise women’s safety
• Protect confidentiality: essential
to ensure women’s safety and
data quality
• Selection and specialised training
and on-going support for
research team
• Take actions to reduce any
possible distress to participants
(referrals, information).
• Ensure findings are properly
interpreted and used

Issues that influence how violence is
measured
• How to define the study population?
• How violence is defined
• How the interview is performed
Source: Researching Violence Against Women: A Training Course for Researchers and
Activists, 2005

Defining a study population
• Cutoff ages
• Marital experience
Activists, 2005

Effect of “study population” on reported
prevalence estimates in Nicaragua
Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
All
women
15-49
Ever
married
women
15-49
Currentl
y married
women
15-49
Formally
married
women
15-49
Current
physical
violence
20% 27% 30% 17%

Defining Violence
• Who defines: the researcher or the
respondent?
• Time frame
• Multiple perpetrators
• Frequency
• Types and severity of violence included
in the definition
Source: Researching Violence Against Women: A Training Course for Researchers and Activists,
2005

Factors that Affect Disclosure
• How the questions are phrased
• Number of opportunities to disclose
• Context in which questions are asked
• Characteristics and skill of interviewers
• Social stigma attached to issue

Suggestions for Measuring Domestic
Violence
• Define the study population broadly
• Use behaviorally specific questions
• Give multiple opportunities to disclose
• Specify discrete time frames (last year,
ever)
• Cue respondent to different contexts
and offenders

Quantitative Tools:
Standardized Questionnaire
• Developing the conceptual framework
• Operationalizing the main variables
• Formulating questions
• Formatting the questionnaire
• Translating the instrument
• Pre-testing the instrument

Developing a conceptual Framework
• What information do you need?
• How will you measure this information?
Activists, 2005

Formulating Questions
• Closed questions
• Open questions
Activists, 2005

Formatting the Questionnaire
• Sequence of questions
– Early questions should be easy and pleasant to answer
– Place difficult or sensitive questions toward the end
– Don’t assume you can predict what will be considered
sensitive if you are working in another culture
• Appropriate answer scales
– Be careful when using complex scales across different
populations and cultures
• Appropriate skip patterns
– Make sure skip patterns are correctly placed
Source: Researching Violence Against Women: A Training Course for Researchers and Activists,
2005

Translating the Instrument
• Accurate and precise translation
• Involve activists or service providers in
translating the test instrument
Activists, 2005

Pre-testing the Instrument
• Try the questionnaire on women who do not live in the
research area, but who are similar to the women you
are planning to study
• Try the test on women of different ages and
backgrounds, including women who are known to be
experiencing different forms of abuse
• Pre-test translated versions
Activists, 2005

Session 10
From Research to Action
Adapted from: Researching Violence Against Women: A Training Course for
Researchers and Activists, 2005

How to use evaluation findings for
advocacy
• Outreach to key constituencies
• Matching the message to the audience
• Sharing findings with the community
• Reaching beyond your borders
Activists, 2005

Let’s create harmony in intimate
relationships. Thailand

Silence for the Sake of Harmony,
Central Java, Indonesia

Candies in Hell …
After the beatings he
would court me and buy
my clothes, but my
grandmother would say
to me, child, what are
you going to with
candies in hell?

MOMMY, YOU LOOK LIKE A
MONSTER…
“When he would beat me, my
daughters would get involved
in the fight. Then he would
throw them around in his fury,
and this hurt me more than
when he beat me.
…and once, I was recovering
after he had beaten me, and
my daughter came up to me
and said “Mommy, you look
like a monster.” And she
began to cry, and what really
hurt me wasn’t so much the
blows, it was her sobbing and
the bitterness that she was
feeling…
Survivor of violence from
Nicaragua
49
23
10
5
0 10 20 30 40 50 60
Children
Husband's family
Husband's friends
Wife's friends
% of women who report witnesses to violence against them

From Insight to Action
• What experiences have you had in
presenting evaluation results to different
stakeholders?
• What are strategies that have been
most successful?

Monitoring and Evaluation Workshop for Grantees

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (8)

Andere mochten auch

Andere mochten auch (14)

Ähnlich wie Monitoring and Evaluation Workshop for Grantees

Ähnlich wie Monitoring and Evaluation Workshop for Grantees (20)

Monitoring and Evaluation Workshop for Grantees

Hinweis der Redaktion