Author: Kenny Meesters, Bartel van de walle Tilburg University

1. Designing towards an impact
evaluation framework for a
collaborative information supply chain
KENNY MEESTERS, BARTEL VAN DE WALLE
ISCRAM BADEN-BADEN, MAY 2013

2. Outline
Outline
Domain Processes
Evaluation
types
Systems
evaluation
Evaluation
perspectives
Scope Measurement
Concept Supply Usage
Findings
Objectives
Indicators
Conclusion
Results
V&TC
Design
Approach
Research
Future work

3. V&TC
Data collection
•Media
•Geo-location
•SMS
Data
processing
•Analysis
•Verification
Dissemenation
•Information
products
•Maps, reports, etc
Information
consumers
•Decision making
•Monitoring
Outline

4. Volunteer Training
Open*
SBTF
Transition
Impact Evaluation Decision Makers
Data Scramble
Data licensing
Preparedness
UN OCHA
IM
ISCRAM
IMMAP, Google,
Mapaction
Woodrow, UN,
Harvard, OSM
Mapaction,
GISCorp,
Munster
ISCRAM,
ICT4Peace,
Mapaction
SBTF, UNV
ISCRAM,
Harvard,
UvT
TBC
UNV,
Munster
“The challenge is to improve coordination between the structured humanitarian
system and the relatively loosely organized volunteer and technical
communities. ” -Valerie Amos, UN Under-Secretary-General
V&TC
ISCRAM,
Harvard,
UvT
Impact Evaluation
Impact Evaluation

5. What do we need?What do we want to know?What do we know?
BusinessV&TC
Practice
Theory
Impact Evaluation

6. What do we need?In general…
BusinessV&TC
Practice
Theory (2) Measure
Status quo
New situation
(1) Define
Indicators
Situations
(3) Analyze
Comparison
Conclusion

7. In general…Applications
BusinessV&TC
Practice
Theory
ProjectA
1. Impact evaluation
2. Impact assessment
3. Program evaluation
Project B

8. BusinessV&TC
Practice
Theory
•Determine how well specific initiatives perform
•Adjust and fine tune specific decisions/projects
•Determine ‘best’ response
•Manage provided solutions
•Secure resources
•AdvocateV&TC
• Impact evaluation
• Impact assessment
• Program evaluation
Use for V&TCApplications

9. BusinessV&TC
Practice
Theory
Use forV&TC
• Design principles of frameworks
• Types
• Measurements
• Indicators
• V&TC
• Objective
• Scope and focus
• Indicators
• Evaluate the framework
• Case studies
• Refine-able, usable tools
Next steps

10. Next stepsEvaluation types
Evaluation
Perspective
Systems
General Formative
Resource
centered
Efficiency
oriented
Goal centered
Effectiveness
oriented
Summative

11. Evaluation typesSystem evaluation
Evaluation
Perspective
Efficiency
oriented
Effectiveness
oriented
Systems
Resource
investment
Production
capability
Resource
consumption
Organization
Organizational
performance
User performance
System
performance

12. System evaluationEvaluation implementation
Organizational performance
Department A
(Sub)
Project A
(Sub)
Project B
(Sub)
Project C
Department B
Evaluation
framework
Project Management
Efficiency oriented
Effectiveness oriented

13. Scope
Overall impact of
the response to a crisis
Impact of the decision making
process on crisis
Impact of information products
on the decision making process
Effect of data processing
on information products
Impact of data collection
on data processing
Soft- en hardware impact
on the system performance
1.
2.
3.
4.
5.
6.
SUPPLIERCONSUMER
Evaluation implementation

14. level 0:
Request / definition
level 1:
Resource allocation
level 2:
Team capability
level 3:
Investments
Impact
level 1:
Support & information
level 2:
Decision making
level 3:
Response effectiveness
ScopeMeasurements
Efficiency-oriented
perspective
Effectiveness-
oriented
perspective
System
implementation
Product generation
efficiency
Response
effectiveness
V&TC deployment

15. MeasurementsIndicators
level 1:
Resource allocation
level 2:
Team capability
level 1:
Support & information
level 2:
Decision making
Impact
Objective Performance measure Applied toV&TC
System
development
Facilities allocation Availability of required (tech.) facilities
Schedule compliance Time required to setup required systems
Requirements definition The clarity of requested products
Operational
resources
Data collection efforts Time/effort required to analyze data
System maintenance Time/effort required to maintain system
Training/support/comm Efforts for user assistance.
Objective Performance measure Applied toV&TC
Team capacity
Productivity rate Level ofV&TC body deployment
Required man-hours The total amount of hours used
Operational
capability
Throughput Products delivered/users served
Utilization rate Hours to product ratio
Response time Turn-around time on specific requests
Objective Performance measure Applied toV&TC
System quality
Usability Ease of use of information products
System features Customization of information products
Access / Availability Ease to reach information products
Information quality
Understandability Presentation of gathered information
Consistency Provided information is consistent
Importance / Relevance Relevance of provided information
Objective Performance measure Applied toV&TC
Individual impact
Awareness / Recall Better situational awareness
Decision effectiveness Enhanced effectiveness of job
Individual productivity Increased personal productivity
Organizational
impact
Cost-effective Information products save resources
Increased capacity Increased effectiveness of operations
Overall productivity Potentially improved outcomes

16. 4 suppliers, 7 consumers7 suppliers, 12 consumers
IndicatorsCase Study
Developpers and entry team
Specific knowledge
Time critical
No budget
Geographically seperate
Users are ‘unkown’
Developpers vs data entry
Expertise available
Time limited
Limited budget
Located in 1 office
Direct contact w. users
=
=
≈
≈
≠
≠
NGODHN

17. Case StudyInformation Supply
System NGO V&TC
FA:
SC:
RD:
Facilities
allocation
Schedule
compliance
Requirements
definition
Data collection
efforts
System
maintenance
Training, suppor
t and
communication
Operational resources System development
NGO Development V&TC Deployement
Resource NGO V&TC
DC:
SM:
TS:
Productivit
y
Required
man-hours
Throughpu
t
Utilization
rate
Response
time
Operational capability Team capacity
NGO Development V&TC Deployement
System NGO V&TC
PD:
MH:
Resource NGO V&TC
TP:
UR:
RT:
Level 2: Capabilities
Level 1: Resources

18. Information SupplyInformation use
System NGO V&TC
SF:
AV:
US:
Inform. NGO V&TC
US:
CO:
IM:
Individ. NGO V&TC
US:
AW:
EF:
PR:
Organ. NGO V&TC
OP:
CI:
CE:
US:
Level 2: Processes
Level 1: Information
Availability
Usability
System
features
Understandabi
lity
Consistency
Importance
NGO Development V&TC Deployement
System quality Information quality
Usage
Awareness
Effectiveness
ProductivityUsage
Cost-effective
Capacity increase
Overall…
NGO Development V&TC Deployement
Individual impact Organization impact

19. Information useFindings
•Agile vs.Waterfall
•Organizational use
•Strong integration
•Requirement analysis
•Sample selection
•Identifying population
•Also for other information supply chains
• Difference in system use
• Increasing impact
• Improving evaluation

20. Future work
V&TC
Feedback:
Increase
deployment
impact
Advocacy:
Secure
resources
Manage:
Improve
products
Findings
•Historical data
•Feedback loops
•Add/remove variables
•Scope of evaluation
• Refinements
• Framework design
• Application
Apply framework
Select case
Impact Evaluation
Framework
Select
particiapants
Control
group
Conducted
Interview
Apply framework
Select
particiapants
Control
group
Conducted
Interview
Verify results Verify results
Refine
Framework
Statistical
analysis
Data store
Model Refinement loop
Impact evaluation
outcome
Evaluation approach
V&TC
Feedback:
Increase
deployment
impact
Advocacy:
Secure
resources
Manage:
Improve
products
Feedback:
Manage
pool of
resources
Advocacy:
Common
understan-
diing of IS
impact
Manage:
Identify
gaps, ensur
e good fit
Coordination
Feedback:
Manage
pool of
resources
Advocacy:
Common
understan-
diing of IS
impact
Manage:
Identify
gaps, ensur
e good fit
Coordination
Feedback:
Improve
effectivenss
by IS use
Advocacy:
Articulate
needs, and
require-
ments
Manage:
Improve IS
use in future
responses
Decision Makers
Feedback:
Improve
effectivenss
by IS use
Advocacy:
Articulate
needs, and
require-
ments
Manage:
Improve IS
use in future
responses
Decision Makers
Impact Evaluation for theV&TC:
Communicate, Learn, Advocate