Ideas on how social services can be improved using data mining and business intelligence.

1. Proposal Note
Smarter, Better, Faster Program Evaluation and Monitoring
Proposal Note
Smarter, Better, Faster Program Evaluation and
Monitoring
Case for Strengthening Project Monitoring and Evaluation
Using Performance Analytics In Social Sector Development
Prepared by
Noor Mohammed Khan
Email: eJirga@gmail.com

2. Proposal Note
Smarter, Better, Faster Program Evaluation and Monitoring
Background
Governments and non-state actors have
increasing concerns for the cost effectiveness on
their programs and strategies. They are
concerned for the development programs
implemented by them and justify value for
taxpayer money. They have deployed monitoring
and evaluation systems for such programs and
projects, hoping that such systems could help
them better control and improve the efficiency of
these interventions. New developments in
software for business intelligence and data
mining have added enhanced capacity of data
analytics for improving the state of monitoring and
evaluation in social sector programs. These
include, Predictive Analytics, Rapid Cycle
Evaluations and Advance Dashboards for Project
Monitoring. A report written by Hamilton Project
for Brookings Institute in 2014 advocates strongly
for the deployment of these tools in order to
improve program efficiency.
Sustainable Development Challenge
The Sustainable Development Goals created by
the UN General Assembly dictates rapid action in
all areas where citizens of underdeveloped
countries face time bound challenges. OECD
countries and rich economies of the world assist
third world countries through grants and aid in
some of these sectors. Within the limited
resources available for social development
projects, governments all around the world are
keen to improve their effectiveness in service
provision to its citizens.
Productivity in social sector is of keen
interest to governments and donors alike. There
is an ongoing search for innovative ideas and
enablers for achieving these goals. The growth of
information and communication technology has
increased efficiency and productivity in some
sectors like education, health and governance.
One of the important domains of this innovation is
the ability of these sectors to deploy management
information systems for effective control. The
widespread availability of internet and social
media has opened new possibilities for using
abundant data for planning and understanding
citizen’s preferences. Advances in information
systems have enabled development of tools for
data analytics and decision support systems. The
purpose of this paper is to propose the ability of
new developments in Business Intelligence to
help in improving program effectiveness and
evaluation.
New Approach
The adoption rate of performance management
systems in public sector / government institutions
has been slow and many theories offer
themselves in explaining the gap between the
public and private sector. “Performance
indicators, performance appraisal, performance
management, and other such terms have
become part of the discourse of public
management and the reality on the ground. The
use of these ideas has attracted some criticism
along the lines that performance cannot be
measured adequately in government, that
performance indicators are inherently flawed, and
that performance appraisal of individual civil
servants is unfair as public sector staff are hard
to compare in terms of their competence and skill.
It is often claimed that a public agency has no
measure comparable to that of profit in the private
sector. Surely this means that performance
measurement in government is inherently
problematic.” [1] Others will argue that the
incentives are not well aligned in the public sector
but given all the critique, the most important
drivers are outweighing the challenges
confronting the adoption of performance
management in public institutions. As an OECD
(2005) paper argues, “Over the past two
decades, enhancing public sector performance
has taken on a new urgency in OECD member
countries as governments face mounting
demands on public expenditure, calls for higher
quality services and, in some countries, a public
increasingly unwilling to pay higher taxes.”
Governments all around the world are
looking at e-Governance as an enabler for
improving service delivery. “Information and
Communication Technology (ICT) in recent years
has presented an opportunity for the IT managers
and the senior officials in government to change
the way organizations leverage and value their
information assets. In contrast to the private
sector businesses, government organizations are
measured not by profits and losses, but by their
ability to deliver upon their mission.” [2].
Data analytics and business intelligence in
combination with data mining offers the potential
of using data generated by organizations for

3. Proposal Note
Smarter, Better, Faster Program Evaluation and Monitoring
improving their performance management and
plan better resource allocation. “Data mining is
becoming an increasingly hot research field, but
a large gap remains between the research of data
mining and its application in real-world business.”
[3]. One of the driving factor of the gap between
the theory and practice of data mining is the
absence of supporting artifacts in the form of
tailored design frameworks and toolkits tailored to
facilitate the adoption of data-mining to particular
public sector institutions.
Social protection programs usually involve large
quantum of resources and if the program is run
inefficiently, some stakeholders may benefit more
at the cost of others. Furthermore, programs like
these are subject to fraud and misappropriation.
The justification for application of data mining and
business intelligence in social sector protection
projects is driven by the need for ensuring that the
program meets its objectives. Critics who argue
for the limitations of performance management in
public sector must acknowledge the fact that
those who will lose in the process of resource
rationalization in program execution will offer
resistance to the adoption of data mining and
business intelligence in social protection sector.
As resources are devolved to subnational levels
of government in social protection programs, it
becomes the responsibility of these levels to
design programs which meets their local needs.
It has been observed that due to limited technical
assistance available to these levels of
governments, it becomes hard for these
governments to design an integrated M&E
system which can show holistic impacts on the
incidence of poverty locally and in the country.
This lack of vertical and horizontal linkage can
only be addressed with a tool that can help
different programs in designing an M&E /
Performance Management System. Social
protection programs get complex as most of the
interventions form part of a value chain where the
graduation of citizens under poverty line becomes
the ultimate objective. This will require managing
large quantities of data for knowledge discovery
and improving program efficiency. This means
understanding relationship between business
data and the management of data or data
governance. “Mining social security/welfare data
is challenging. The challenges arise from
business, data, and the mining of the data. Social
security data is very complex, involving all the
major issues that are discussed in the data quality
and engineering field, such as sparseness,
dynamics, and distribution. Key aspects
contributing to challenges in mining social
security data are many, e.g
1) Specific business objectives in social security
and government objectives.
2) Specific business processes and outcomes.
3) Heterogeneous data sources.
4) Interactions between customers and
government officers.
5) Customer / Citizen behavioral dynamics.
6) General challenges in handling enterprise
data, such as data imbalance, high dimension,
and so on.” [4]
State of the art data analytics technology
can offer predictive analytics and support rapid
cycle evaluation. “Predictive analytics refers to
a broad range of methods used to anticipate an
outcome. For many types of government
programs, predictive analytics can be used to
anticipate how individuals will respond to
interventions, including new services, targeted
prompts to participants, and even automated
actions by transactional systems. With
information from predictive analytics,
administrators can identify who is likely to benefit
from an intervention and find ways to formulate
better interventions. Predictive analytics can also
be embedded in agency operational systems to
guide real-time decision making."[5]. The
decision tree shown in Fig 1 shows the probability
of a response given a prior condition is met. Such
analytics makes the process of decision making
much smarter on programs and projects.
Fig 1
Monitoring Dashboards
As complex social development programs are
executed in geographically disbursed settings
and levels of management control, the need for

4. Proposal Note
Smarter, Better, Faster Program Evaluation and Monitoring
effective project management and monitoring
increases substantially. Stakeholders and actors
need to know about different aspects of project
performance. There is no best way other than to
visualize the state of key performance indicators
other than on a dashboard. A monitoring
dashboard would serve specific group of decision
makers at various levels and generate reports by
department or cross department to assist
decision making on program policies and
operations. The design of dashboards usually
follows a cognitive map for decision making.
Fig 2
Dash Boards can offer summative visualization of
key performance indicators as show above in the
figure. “We believe that these techniques can be
used to help government programs including
social service programs serving low-income
individuals to improve program services while
efficiently allocating limited resources. We
believe that the use of predictive modeling and
rapid-cycle evaluation both individually and
together holds significant promise to improve
programs in an increasingly fast-paced policy and
political environment.” [5]
Examples;
There is a wide spectrum of use for performance
analytics. The approach is being tested in many
different projects all around the world and shows
promising results. Some of the following
examples are being listed from different social
development sectors;
Health;
Health care. FAMS (Fraud and Abuse
Management System) is assisting health
insurance organizations dealing with fraud and
abuse: detection, investigation, settlement, and
prevention of recurrence. [6]
Education;
Dropout rates from schools is a common
problem. Understanding the patterns for a
specific group of students who are likely to go to
drop schools, the management now can focus
their energies on this group and take
countermeasures in time before the child drops
out of school.
Social Protection;
CentreLink in Australia undertakes fraud centric
analysis of the social sector data to identify
patterns and inform policy making process.
The Way Forward
Having established that government programs
can improve by deploying performance analytics
we will now discuss the approach and steps in
this direction.
Need Assessment
“We propose that social service agencies take
two actions. First, agency departments with
planning and oversight responsibilities should
encourage the staff of individual programs to
conduct a thorough needs assessment. This
assessment should identify where predictive
analytics and rapid-cycle evaluation can be used
to improve service delivery and program
management. The assessment should also
evaluate whether the benefits of adopting these
tools outweigh the costs, resulting in a
recommendation of whether and how these tools
should be deployed”.[5].
Monitoring and Evaluation
“Rapid-cycle evaluation, another decision-
support approach, uses evaluation research
methods to quickly determine whether an
intervention is effective, and enables program
administrators to continuously improve their
programs by experimenting with different
interventions. Like predictive analytics, rapid-
cycle evaluation leverages the data available in
administrative records.” [5].

5. Proposal Note
Smarter, Better, Faster Program Evaluation and Monitoring
The approach can be deployed for new and
running projects as significant resources can be
saved using the available technology.
Revisiting Program Performance Matrix
It is important to scope performance
management in a given program or project.
Traditionally Log-frames, and performance
matrices are used for spelling out the key
performance indicators. This approach can be
viewed with the enterprise architecture where a
there needs to be an alignment between the
business objectives, operational strategy and
organizational capacity.
The approach will help establish key performance
indicators and map existing data sets to generate
the desired reports. Also the analysis provides for
viewing the gaps in data collection with can enrich
the rapid cycle evaluation and help identify the
processes which may need to be re-engineered
for delivering better outcomes. The strength of
the approach lies in its holistic view and the
integration of all aspects of program operations to
deliver higher value for taxpayer money.
Data Warehousing
“It is important to note that predictive analytics
require historic observations of key outcomes.
This means that programs developing new
systems may not be able to perform predictive
analytics until the system has captured enough
history. Similarly, programs extracting data from
transactional systems would need to extract a
sufficiently large volume of historical data in order
for predictive analytics to be effective”.[5].
Data Warehouse is the most important artifact of
institutional memory and a step towards
managing the enterprise knowledge.
Fig 3
Fig 3 shows the integration of data from various
sources into a data warehouse for further
analysis and reporting.
The data warehouse will collect all information
generated by various processes in the program
and organize it in a structured format. This makes
sure that the data becomes available for rapid
cycle evaluation and predictive analysis. This will
include the development of Meta standards which
can help connect the data within and outside the
organization through common standards. Also
data dictionaries are developed to provide better
understanding of the data for those who intend to
use it for analysis.
Data Mining & Business Intelligence
Technology
Data Mining and Business Intelligence has now
become a specialized field on its own but has its
origins in database design and development.
Today, specialized software is available for
desktop and cloud environments. Some of the
most prominent include, Microsoft SQL Business
Intelligence Servers, Oracle, SAP Lumira,
Tableau, IBM Cognos and IBM Watson etc.
References
[1] F. S. Berry, M. P. Aristigueta, and K.
Yang, “International Handbook of Practice-
Based Performance Management Performance :
A New Public Management Perspective,” 2013.
[2] Coman, M. 2009. Business Intelligence
and E-governance. Lex ET Scientia International
Journal, 17, 484-491
[3] Y. Zhao, H. Zhang, L. Cao, H.
Bohlscheid, and Y. Ou, “Data Mining Applications
in Social Security,” pp. 81–96, 2009.
[4] L. Cao, “Social security and social
welfare data mining: An overview,” IEEE Trans.
Syst. Man Cybern. Part C Appl. Rev., vol. 42, no.
6, pp. 837–853, 2012.
[5] [1] S. Cody and A. Asher, “Proposal
14 : Smarter , Better , Faster : The Potential for
Predictive Analytics and Rapid-Cycle Evaluation
to Improve Program Development and
Outcomes,” 2014. [Online]. Available:
http://www.brookings.edu/research/papers/2014/
06/19-predictive-analytics-rapid-cycle-
evaluation-improve-program-cody-asher.
[6] S. N. S. S.Sumathi, Introduction to Data
Mining and its Applications. Verlag Berlin
Heidelberg: Springer, 2006.