This document outlines a chapter on policy evaluation from an educational textbook. The chapter discusses various methods of policy evaluation including assessing impact on target groups, experimental research designs, and challenges such as political and bureaucratic influences. It also describes the roles of organizations like the Office of Management and Budget and General Accountability Office in evaluating federal programs. The overall goal of the chapter is to examine how governments assess the effectiveness of public policies.
Point 1- Overview of policy evaluation
Learning about consequences of public policy
Stated goals of a program not always clear
Notion of policy evaluation thus limited to their achievements
Point 2- Policy impact
Effects of a policy on real-world conditions
Includes its impact on target situations or groups
Includes its impact on future and immediate conditions
All of its benefits and costs at present and in the future measured
4.1.1: Measuring Impact, Not Output
Point 1- Definition of policy output
Measure of a government activity
Important for describing the determinants of a public policy
Examples include welfare benefits paid, criminal arrests and prosecutions, Medicare payments, and school enrolments
Point 2- Difference between policy impact and policy output
Policy output does not tell much about poverty, crime, health, or educational achievement
Identifying the changes in society that are associated with measures of government activity necessary in assessing policy impact
4.1.2: Target Groups
Point 1- Overview
Part of the population for whom a program is intended
Should be first identified before determining the desired effects of the program on them
Point 2- Various effects of a program on target groups
To change knowledge, attitudes, awareness, interests, or behavior
To change physical or economic circumstances
Other unintended, or side, effects
4.1.3: Nontarget Groups
Point 1- Definition
Various other segments of society influenced by a program or policy unintentionally
Identifying important nontarget groups is difficult
Point 2- Nontarget effects
Could be expressed as beneficial such as the benefits to the construction industry of public housing projects
Could also be expressed as costly
4.1.4: Short-Term and Long-Term Effects
Point 1- Concerns about the effects of a program
Time period before the benefits or costs are felt
Whether a program is designed for short-time emergencies or as a long-term development effort
Issues that will prevent the processes of incrementalism and bureaucratization from turning it into a long-term program
Point 2- Conclusions drawn from impact studies
New or innovative programs have short-term positive effects that disappear as the novelty and enthusiasm wear off
Other programs experience difficulties at first but turn out to have “sleeper” effects
4.1.5: Calculating Net Benefits and Costs
Point 1- Formula
All of the benefits of a policy, both immediate and long range, minus all the costs, both immediate and future
Point 2- Difficulties in coming up with a net balance
Knowing all the costs and benefits
Everyone agreeing on what is a benefit and what is a cost
Point 1- Overview
Symbolic impact of a policy deals with perceptions individuals have of government action and their attitudes toward it
Public policies frequently judged in terms of good intentions
Policies help hold people together and maintain an orderly state
Point 2- Systematic policy analysis
Concentrates on what governments do, why they do it, and what difference it makes
Devotes less attention to what governments say
4.3.1: Hearings and Reports
Point 1- Overview
Most common type of program review
Government administrators asked by chief executives or legislators to report on the accomplishments of their own programs
Program administrators provide written annual reports
Point 2- Disadvantages
Not considered very objective means of program evaluation
Benefits are frequently magnified and costs are subsequently minimized
4.3.2: Site Visits
Point 1- Overview
Teams of high ranking officers visit agencies or conduct inspections in the field
Teams include high-ranking administrators, expert consultants, legislators, or some combination of these people
Point 2- Issues that are looked into
Management of a program
Whether specific guidelines are being followed
Availability of competent staffs
Whether or not clients are pleased with the services
4.3.3: Program Measures
Point 1- Overview
Output measures are covered in the data developed by the agencies themselves
Does not indicate what impact the policies have on society
Point 2- Examples of policy output measures
Number of recipients in various welfare programs
Number of persons in work-force training programs
Number of public hospital beds available
Tons of garbage collected
Number of pupils enrolled
4.3.4: Comparison with Professional Standards
Point 1- Overview
Comparing actual government outputs with ideal outputs
Desired levels of outputs are developed by professional associations
Point 2- Disadvantages
Does not focus on the impact of government activities on the conditions of target or nontarget groups
Standards are developed by professionals who are only guessing at what ideal levels of benefits and services should be
4.3.5: Evaluation of Citizens' Complaints
Point 1- Overview
Analyzing the complaints of citizens
Administrators occasionally develop questionnaires for participants in their program to learn whether they are satisfied or not
Point 2- Disadvantages
Not all citizens voluntarily submit complaints
Critics of government programs are self-selected and rarely represent the public
Complaints of the vocal few are not always shared by the many more who have not spoken up
Does not focus on the impact of the program on the lives of its participants
4.3.6: Surveys of Public Opinion
Point 1- Overview
Governments survey citizens about their satisfaction or dissatisfaction with various programs and services
Common at the local level of government
Point 2- Polls
Reflect general satisfaction or dissatisfaction with federal programs
Usually instructive
4.3.7: Congressional Investigations
Point 1- Overview
An implied constitutional power
Derives from Congressional power to legislate
Point 2- Process
Extensive background research
Questioning of witnesses, usually in public
Point 3- Role of congressional committees
To open investigations to probe particular issues and report on them
Make policy reports based on the investigation
Can issue subpoenas compelling witnesses to testify
Can hold witnesses in contempt of Congress if they refuse to testify
Point 4- Effects
Mostly publicize an issue or attract publicity to members of Congress
Sometimes result in the evaluation of policies and programs
Point 1- Failure of the common evaluative methods
Does not attempt to weigh costs against benefits
Difficult to calculate the costs of making specific changes in society
More complex and costly methods of program evaluation are required
Point 2- Systematic program evaluation
Requires measuring what happened and what would have happened without the program
Comparing the two conditions of the society
4.4.1: Before Versus After Comparisons
Point 1- Overview
Most common research design in program evaluation
Compares results in a jurisdiction at two times—one before program implementation and the other after
Point 2- Disadvantages
Difficult to know whether the changes observed came about as a result of the program or as a result of other changes occurring at the same time
Usually only target groups are examined
Design 1 – Before vs. After
Design 2 – Projected vs. Postprogram
Design 3 – With vs. Without Program
Design 4 – Control vs. Experimental Groups
4.4.2: Projected Trend Line Versus Postprogram Comparisons
Point 1- Process of evaluation
Project preprogram trends into the postprogram time period
Projections can be compared with what actually happened in society after program implementation
Difference between the projections based on preprogram trends and actual postprogram data can be attributed to the program
Point 2- Establishing a trend line
Data on target groups or conditions must be obtained for several time periods before the program was initiated
Requires tremendous effort by program evaluators
4.4.3: Comparisons Between Jurisdictions With and Without Programs
Point 1- Overview
Comparing individuals who have participated in programs with those who have not
Comparing cities, states, or nations that have programs with those that do not
Point 2- Disadvantages
Difficult to attribute differences in participants’ conditions to differences in government programs
Sometimes comparisons are made only in the postprogram period
Point 3- Alternate design
Observing both kinds of jurisdictions before and after program introduction
Estimates differences between jurisdictions before program efforts are considered
Provides some protection against attributing differences to a particular program
4.4.4: Comparisons Between Control and Experimental Groups Before and After Program Implementation
Point 1- Process
Careful selection of control and experimental groups that are identical in every way
Application of the policy to the experimental group only
Comparing changes in the experimental group with changes in the control group after the application of the policy
Point 2- Initial requirements
Control and experimental groups must be identical
Preprogram performance of each group must be measured and found to be the same
Program must be applied only to the experimental group
Point 3- Advantage
Estimates changes derived from the effects of other forces in society
4.5.1: A Bias Toward Positive Results
Point 1- Relevant questions
Are government-sponsored research projects predisposed to produce results supportive of popular reform proposals?
Are social scientists inclined to produce findings in support of liberal reform measures?
Point 2- Consequences of unsupportive or negative results
Go back and recode data
Redesign research
Reevaluate results
4.5.2: The Hawthorne Effect
Point 1- Overview
Taken from early experiments at the Hawthorne plant of Western Electric Company in Chicago in 1927
Worker output increased with any change in routine
Point 2- Conclusion
People behave differently when they know they are being watched
Example: Students
4.5.3: Generalizing Results to the Nation
Point 1- Overview
Results obtained with small-scale experiments differ from large-scale ones
Example: Brief experiment involving small number of families
Point 2- Guarantee of minimum annual income
Cultural standards might be changed nationwide
Might seriously understate expected cost of an economy-wide program
4.5.4: Ethical and Legal Issues
Point 1- Relevant fields
Government research
Medicine
Social experimentation
4.5.5: Political Interpretations of Results
Point 1- Overview
Political milieu shapes policy research
Politics helps decide what policies and policy alternatives will be studied
Point 2- Relevant fields
Education
Welfare
Housing
Health
4.6.1: Benefit–Cost Analysis of Federal Regulations
Point 1- Overview
Army Corps of Engineers undertook in 1936
Development of government benefit–cost analysis
Point 2- Reasoned determination
Benefits justify the costs
Regulatory action maximized net societal benefits
4.6.2: Value of a Statistical Life
Point 1- Controversies
Valuation of a human life
Required in the design of evaluations of health and safety regulations
Point 2- Relevant agencies
Environmental Protection Agency
Department of Transportation
Food and Drug Administration
Occupational Safety and Health Administration
Department of Homeland Security
Point 1- Overview
Arm of Congress
Audits operations and finances of federal agencies
Evaluates programs
Reports findings to Congress
Point 2- Criticized agencies
Defense Department
Environmental Protection Agency
Social Security
Point 1- Overview
To determine goals of programs
To not actually change the conditions of target groups
Government agencies usually have a heavy investment
Point 2- Requirements
Funds
Facilities
Time
Personnel
Point 1- Positive arguments
Effects of the program are long range and cannot be measured at the present time
Effects of the program are diffuse and general in nature
Effects of the program are subtle and cannot be identified by crude measures or statistics
Experimental research cannot be carried out effectively
Point 2- General laws
Wilson’s First Law
Wilson’s Second Law
4.10.1: Concentrated Benefits, Dispersed Costs
Point 1- Overview
Limited benefits concentrated in a small, well-organized constituency
Greater costs dispersed over a large, unorganized, uninformed public
Point 2- Characteristics of beneficiaries
Concerned
Well-informed
Active in their support
4.10.2: Legislative and Bureaucratic Interests
Point 1- Incentives for bureaucrats
To resist or undermine negative evaluations of programs
To respond to public criticism by making only marginal changes in programs
To claim programs are failing because not enough is being spent
Point 2- Actions taken by legislators
To protect failed programs
To minimize reform
To block termination
4.10.3: Incrementalism at Work
Point 1- Relevant considerations
Attention focused on proposed changes in existing programs
Value of programs in their entirety not considered
Point 2- Role of negative evaluative studies
Limiting increases for failed programs
Identifying programs ripe for budget cutting
Point 1- Requirements
Some agreement on what problems the government should undertake to resolve
Some agreement on the nature of societal benefits and costs and the weights to be given
Some agreement on the formulation of a research design
Point 2- Emphasis
Search for common concerns that might form the basis for identification of societal problems
Reasonable trade-offs among conflicting values at each stage of the policymaking process
Search for mutually beneficial outcomes for diverse groups
Bargaining among participants, even in separate policy areas, to win allies
Compromise and conciliation and a willingness to accept modest net gains
Point 1- Americans’ expectations of governments
Eliminate poverty
End racism
Ensure peace
Prevent crime
Restore cities
Point 2- Limitations of policy
Some societal problems are incapable of solution
Expectations may always outrace the capabilities of governments
Policies that solve problems of one group in society may create problems for other groups
Possibility that some societal forces cannot be harnessed by governments