Policy evaluation in criminology — the systematic assessment of whether criminal justice programs and policies achieve their intended effects — is both a methodological enterprise and a political one, requiring not only the technical capacity to design and execute rigorous evaluations but the institutional relationships, research independence, and communication skills to ensure that evaluation findings reach the decision-makers who can act on them. The evidence-based criminology movement has elevated policy evaluation to central importance within applied criminological practice, creating expectations among funders, policymakers, and practitioners that significant criminal justice investments will be evaluated rigorously and that evaluation findings will inform continuation, modification, or discontinuation decisions. Meeting those expectations with the technical rigor that credible evaluation requires, while navigating the institutional politics that surround programs with invested constituencies, is the central challenge of criminological evaluation practice.
Criminology and Public Policy engages policy evaluation not only as a methodological topic but as a policy topic in its own right: the institutional arrangements for evaluation — who funds it, who conducts it, what independence it has, how findings are communicated and used — determine whether evaluation produces the knowledge improvements in criminal justice practice that the investment in evaluation is intended to achieve. A field that produces technically rigorous evaluations that are never used because they challenge the interests of the institutions they evaluate has not solved the evaluation problem even if it has solved the methodology problem. Understanding evaluation’s institutional dimensions alongside its methodological ones is essential for criminologists who aspire to produce knowledge that makes a difference.
Introduction
The evaluation question — does this program work? — conceals at least four distinct questions that policy evaluation must address separately. Does the program produce the intended behavioral or institutional changes that constitute its proximate mechanisms? Do those proximate changes produce the ultimate outcomes — crime reduction, recidivism reduction, victim satisfaction, institutional efficiency — that the program is designed to achieve? Do the effects, if found, persist over time and generalize across the populations and contexts beyond those directly studied? And do the benefits exceed the costs when both program costs and comparison costs are calculated with appropriate comprehensiveness? Each question requires different data, different methods, and different analytical frameworks, and the best-designed evaluations address all four rather than treating evidence of any single element as sufficient demonstration of program effectiveness.
The institutional landscape of criminological policy evaluation includes academic researchers, government research agencies, independent policy research organizations, and internal agency research units whose different organizational positions generate different capacities and different risks for producing credible, useful evaluation knowledge. Academic researchers bring methodological expertise and institutional independence but may lack access to administrative data and operational knowledge. Government research agencies — the National Institute of Justice, the Bureau of Justice Statistics, and state research offices — bring data access and policy proximity but may face political pressures on research design and findings. Independent policy research organizations — RAND, MDRC, the Urban Institute, Vera Institute of Justice — occupy a middle position with both methodological capacity and operational knowledge, though their dependence on government and foundation funding creates its own potential for research agenda distortion. Internal agency research units provide operational access and institutional knowledge but face the most direct pressure to produce findings favorable to their parent agency.
The most important structural requirement for credible policy evaluation is independence — the evaluation researcher’s freedom to follow the evidence wherever it leads, including to findings that challenge the program being evaluated, the agency running it, and the funder supporting it. Evaluation research conducted without this independence — where the agency being evaluated controls the research design, data access, analytical choices, or publication decisions — cannot be treated as credible evidence regardless of its methodological sophistication, because the selection pressures that produce favorable findings operate at every stage of the research process in ways that methodological quality cannot correct.
Research Design and Causal Inference
The Experimental Standard and Its Alternatives
The randomized controlled trial (RCT) — in which eligible individuals, places, or agencies are randomly assigned to receive the intervention being evaluated or to a control condition — provides the strongest available basis for causal inference in program evaluation, because random assignment ensures that treatment and control groups are statistically equivalent before the intervention on all characteristics, measured and unmeasured, that predict the outcome. The logic is straightforward: if randomization succeeded, post-intervention differences between groups can be attributed to the intervention rather than to pre-existing differences, because those pre-existing differences were eliminated by the randomization procedure. The Minneapolis Domestic Violence Experiment (1984), the Moving to Opportunity housing experiment (1994–1998), and the Multisystemic Therapy RCTs represent landmark criminological experiments whose findings have shaped policy and whose methodological quality has withstood sustained scrutiny (Sherman & Berk, 1984; Kling et al., 2007).
The practical and ethical constraints on randomized criminological evaluation are real and significant. Random assignment of individuals to incarceration versus community supervision raises obvious ethical concerns when the evaluator controls actual justice dispositions. Random assignment of individuals to different police patrol strategies raises logistical challenges when officers exercise discretion in ways that contaminate assignment. And random assignment of communities to different policy environments raises theoretical challenges when the treatment effects operate through social processes that individual-level randomization cannot capture. These constraints have motivated the development of quasi-experimental alternatives that approximate the causal inference benefits of randomization using statistical methods and natural variation in treatment assignment.
Regression discontinuity designs exploit arbitrary eligibility thresholds — age cutoffs determining juvenile versus adult court jurisdiction, score thresholds triggering mandatory treatment, population thresholds determining program eligibility — to compare individuals or units just above and below the threshold who are assumed equivalent in all relevant characteristics. The identifying assumption is that the relationship between the threshold-determining variable and outcomes is continuous at the threshold except for the discontinuity produced by the policy, so that any observed discontinuity in outcomes at the threshold can be attributed to the policy treatment rather than to underlying differences in the threshold-determining variable. Regression discontinuity has been applied to evaluate the effects of incarceration length on recidivism, the effects of juvenile versus adult court processing on criminal careers, and the effects of specific sentencing thresholds on deterrence.
Process Evaluation and Implementation Science
Impact evaluation — assessing whether a program produced its intended outcomes — is necessary but not sufficient for the policy-relevant knowledge that effective criminal justice practice requires. Process evaluation — assessing whether a program was implemented as designed, reached its intended population, and delivered its services with sufficient quality and dosage — is equally essential, because the most common source of impact evaluation null findings is program failure (the program was not implemented adequately to test the theory) rather than theory failure (the theory of change was wrong). A null finding from a poorly implemented program cannot be interpreted as evidence that the program theory is incorrect; it can only be interpreted as evidence that this implementation failed to produce the predicted effects.
Implementation science — the systematic study of the factors that determine whether evidence-based interventions are implemented with the fidelity, quality, and sustainability that produce outcomes comparable to those achieved in controlled evaluations — has developed specific frameworks for assessing implementation quality that criminological evaluation practice has increasingly adopted. The National Implementation Research Network (NIRN) at the University of North Carolina‘s Fixsen-Blase implementation framework identifies the implementation drivers — competency development, organizational supports, and leadership — that predict implementation quality, and has been applied in corrections, juvenile justice, and community supervision settings to diagnose and address implementation failures before they undermine evidence-based program effectiveness.
Table 1. Evaluation Designs in Criminological Policy Research
| Design | Causal Inference | Feasibility | Best Application | Key Limitation |
|---|---|---|---|---|
| Randomized controlled trial | Highest | Limited by ethical/practical constraints | Program evaluation with eligible population; policing tactics | Cannot always randomize; external validity concerns |
| Regression discontinuity | High if threshold met | Requires clear eligibility threshold | Sentencing policy; program eligibility cutoffs | Estimates effect only at threshold; narrow generalizability |
| Difference-in-differences | Moderate-high | Requires comparison jurisdiction/group | Policy adoption across jurisdictions; legislative changes | Parallel trends assumption may not hold |
| Interrupted time series | Moderate | Requires adequate pre-policy data | Policy changes with clear implementation date | Confounding from contemporaneous changes |
| Propensity score matching | Moderate | Requires rich covariate data | Large administrative dataset comparisons | Selection on unobservables unaddressed |
| Systematic review / meta-analysis | Highest across literature | Requires sufficient primary studies | Synthesizing evidence across programs | Publication bias; heterogeneity of contexts |
| Process evaluation | N/A (describes, not estimates effects) | High | Implementation quality assessment; program monitoring | Cannot assess impact without impact component |
| Cost-benefit analysis | N/A (requires impact estimate input) | Moderate | Comparing alternatives; budget justification | Monetization assumptions; discount rate sensitivity |
Systematic Review and Meta-Analysis
The Campbell Collaboration and Evidence Synthesis
The systematic review and meta-analysis tradition — which aggregates evidence across multiple primary studies using transparent, pre-specified methods to produce cumulative estimates of program effectiveness — has transformed the criminological evidence base from a collection of individual studies whose findings are difficult to integrate into a synthesized literature whose cumulative implications for policy are substantially clearer. The Campbell Collaboration’s Crime and Justice Review Group, established in 2000, has produced systematic reviews of criminal justice interventions covering hot spots policing, community supervision, drug courts, sex offender treatment, juvenile diversion, domestic violence interventions, and dozens of other policy-relevant topics using the rigorous systematic review methods developed by the Cochrane Collaboration for medicine and adapted for social science.
The methodological requirements of systematic review — comprehensive literature search, explicit inclusion and exclusion criteria, quality assessment of primary studies, quantitative synthesis of effect sizes using meta-analytic methods, and transparent reporting following the PRISMA guidelines — address the limitations of the narrative reviews that preceded them: selective citation favoring familiar studies, implicit weighting of studies by researcher preference, and conclusions that reflect the reviewer’s prior beliefs as much as the evidence. The transparency requirements of systematic review make it possible for other researchers to reproduce and update the review as new primary studies accumulate, creating a living evidence base that narrative reviews cannot provide.
The policy utility of systematic reviews depends on both their methodological quality and their communication quality. Reviews written primarily for methodologist audiences — emphasizing heterogeneity statistics, funnel plot asymmetry, and moderator analysis without translating these into policy-relevant conclusions — do not produce the evidence-based guidance that practitioners and policymakers can use. The What Works Clearinghouse in education and the What Works Centre for Crime Reduction in policing have developed practitioner-accessible evidence summaries that distill systematic review findings into practice guidance calibrated to operational audiences — demonstrating that methodological rigor and policy relevance are compatible, if the communication investment is made to achieve both simultaneously.
Publication Bias and the Evidence Base
Publication bias — the systematic tendency for journals to accept and publish studies with statistically significant positive findings while rejecting studies with null or negative findings — threatens the validity of both individual meta-analyses and the cumulative evidence base they synthesize. If the studies that reach publication are systematically those that found positive effects, the average effect size estimated across published studies will overstate the true average effect in the population of all studies actually conducted. Meta-analyses that do not explicitly assess and correct for publication bias will produce inflated estimates of program effectiveness — and if those estimates inform investment decisions, the resulting programs will produce smaller effects in real-world implementation than the meta-analytic evidence predicted.
Pre-registration — committing study hypotheses, design, and analysis plan to a public registry like the Open Science Framework before data collection begins — reduces but does not eliminate publication bias by making the universe of conducted studies traceable independent of publication status. Criminological pre-registration has increased substantially since the Society of Criminology and Society for Evidence-Based Policing began encouraging it, but it remains far less common in criminology than in psychology and medicine, leaving the criminological evidence base more vulnerable to publication bias than the evidence-based criminology movement’s aspirations require.
Cost-Benefit Analysis in Criminal Justice
The Economic Case for Evidence-Based Policy
Cost-benefit analysis — comparing the monetized costs and benefits of alternative programs or policies — provides the framework through which criminological evaluation evidence can be incorporated into the budget decisions and policy tradeoffs that resource-constrained governments must make. The most consequential applications of cost-benefit analysis in criminal justice have provided the economic rationale for prevention and rehabilitation investment over incarceration — by documenting that the costs of imprisonment substantially exceed the costs of the prevention and treatment alternatives that produce comparable or better crime reduction outcomes, while generating fewer collateral costs in family disruption, community destabilization, and human capital loss.
The Washington State Institute for Public Policy (WSIPP) has developed the most sophisticated and most influential cost-benefit analysis framework for criminal justice programs in the United States, producing benefit-cost estimates for hundreds of programs across criminal justice, education, health, and social services that the Washington state legislature has used as the primary tool for evidence-based budget allocation. WSIPP’s methodology — converting outcome effects to monetized values using willingness-to-pay estimates, discounting future benefits to present value, and comparing net benefits across alternative investments — has been adopted or adapted by many other state governments and has influenced federal program evaluation frameworks. Its consistent finding that well-designed prevention and treatment programs generate positive benefit-cost ratios while many incarceration expenditures generate negative ratios (costs exceed benefits) has provided the economic argument for evidence-based reform that legislative appropriators find more compelling than effect size statistics alone.
Monetizing Crime Costs
The benefit side of criminal justice cost-benefit analyses depends critically on the monetization of crime costs — the translation of crime reduction effects into dollar values that can be compared to program cost dollars. Crime costs encompass victim costs (tangible costs including medical treatment, lost earnings, and property damage, plus intangible costs representing the pain, suffering, and reduced quality of life that crime victims experience) and criminal justice system costs (police, prosecution, courts, and corrections expenditures associated with each crime). The most widely cited estimates of crime costs were developed by Miller, Cohen, and Wiersema in the mid-1990s using jury verdict data to estimate willingness-to-pay for crime reduction, a methodology that has been updated by Cohen and Piquero and by Wickramasekera and colleagues using more recent data.
The intangible victim cost component — which attempts to monetize the pain, suffering, and lost quality of life that crime victims experience, even when those experiences produce no directly measurable economic loss — is simultaneously the most important component for crimes involving personal violence and the most methodologically contested, because it rests on willingness-to-pay estimates that reflect the preferences of survey respondents rather than actual victim experience. The range of estimates in the literature for specific crime types is wide enough that cost-benefit conclusions for programs with modest effect sizes are sensitive to which cost estimates are used, creating opportunities for motivated reasoning in cost-benefit presentations that evaluators must address through transparent sensitivity analysis.
Natural Experiments and Quasi-Experimental Advances
The past two decades have seen substantial methodological advances in quasi-experimental evaluation that have expanded the range of criminological policy questions that can be addressed with credible causal evidence. Difference-in-differences designs — comparing outcome changes in jurisdictions that adopted a policy to contemporaneous changes in jurisdictions that did not, using the untreated jurisdictions to control for secular trends — have been applied to evaluating the effects of marijuana legalization, stand-your-ground laws, body-worn camera adoption, and dozens of other criminal justice policy changes that occurred at different times in different jurisdictions, generating natural experimental variation that statistical methods can exploit.
The synthetic control method — which constructs a weighted combination of comparison units that best matches the treated unit’s pre-treatment outcome trajectory, providing a more credible counterfactual than simple comparison groups — has been applied to high-stakes criminal justice policy evaluations where the small number of treated units (states, cities) makes conventional difference-in-differences estimation difficult. Research evaluating the crime consequences of California’s Proposition 47 — which reclassified several felonies to misdemeanors and substantially reduced incarceration — and of police department reforms following federal consent decrees have used synthetic control designs to estimate causal effects with greater credibility than simple before-after comparisons permit.
Instrumental variable designs — which use sources of as-if-random variation in treatment assignment as instruments to identify causal effects net of selection bias — have been applied to evaluate the causal effects of incarceration length, prosecutorial charging decisions, and immigration enforcement through instruments including judge assignment lotteries, weather conditions affecting criminal activity, and policy thresholds that affect treatment probability discontinuously. These designs require identifying instruments that satisfy the exclusion restriction — that the instrument affects outcomes only through the treatment channel — an assumption that must be argued theoretically and tested empirically rather than assumed, but that when satisfied provides causal identification in observational data that cannot be achieved through covariate adjustment alone.
Evaluation and the Politics of Knowledge
When Evidence Is Inconvenient
The institutional politics of policy evaluation — who commissions it, who conducts it, who controls publication, and what happens when findings are unfavorable — shape the knowledge that evaluation produces in ways that methodological quality cannot fully correct. Programs with invested constituencies, large budgets, and strong political backing are more likely to produce favorable evaluations than programs without these characteristics — not primarily because they are more effective but because the institutional pressures that surround their evaluation create conditions that favor favorable findings through selection at every stage: which programs get evaluated, which evaluators are selected, what comparison conditions are used, which outcomes are prioritized, and how results are communicated.
The Drug Abuse Resistance Education (D.A.R.E.) program — evaluated in multiple rigorous studies that consistently found no significant effects on drug use, ultimately prompting a curriculum redesign that also has not demonstrated effectiveness — exemplifies the persistence of programs with null evidence when institutional constituencies are strong enough to maintain funding regardless of evidence. D.A.R.E. was implemented in three-quarters of American school districts at its peak, with strong institutional support from law enforcement associations, school administrators, and parent organizations, and it continued receiving federal support for years after the null evidence was clear and widely publicized. The D.A.R.E. experience demonstrates both the importance of rigorous independent evaluation and the limits of evaluation evidence alone in driving policy change when institutional interests favor continuation.
Practical Significance vs. Statistical Significance
The distinction between statistical significance — the probability that an observed effect could have occurred by chance if the null hypothesis were true — and practical significance — whether the effect is large enough to matter for policy — is among the most important methodological issues in criminological policy evaluation, and it is frequently conflated in both research reports and policy discussions. A program that produces a statistically significant recidivism reduction of 1–2 percentage points may meet conventional significance thresholds in a large evaluation while producing effects too small to justify its costs; conversely, a program that produces practically meaningful 15% recidivism reductions may fail to reach statistical significance in an underpowered evaluation. Both types of error — treating small statistically significant effects as practically important and treating large but statistically non-significant effects as null — have distorted the criminal justice evaluation literature and produced policy recommendations that the data cannot actually support.
The growing emphasis on effect size reporting, confidence intervals, and power analysis in criminological evaluation research — institutionalized through reporting standards at journals including Criminology & Public Policy and through funders’ requirements for minimum detectable effect size calculations in evaluation proposals — reflects the field’s increasing methodological sophistication about the distinction between statistical and practical significance. The policy-relevant question is not “is the p-value below 0.05?” but “is the effect large enough and reliable enough to justify this program’s cost over the best available alternative?” Evaluators who cannot answer the second question, even when they can answer the first, are not providing the policy-relevant information that evaluation is intended to generate.
Evidence Utilization and the Research-Practice Gap
Why Good Evaluations Don’t Always Change Practice
The research-practice gap in criminal justice — the persistent failure of well-established evaluation evidence to change practitioner behavior — is one of the most important and most studied problems in applied criminology. Programs with strong evidence of ineffectiveness continue operating; programs with strong evidence of effectiveness fail to be adopted at scale; and the decision processes of most criminal justice agencies are more heavily influenced by professional tradition, resource constraints, political pressure, and organizational inertia than by the findings of rigorous evaluations. Understanding why this gap persists — and what structural changes would narrow it — is as important for evidence-based criminological practice as understanding how to conduct rigorous evaluations.
Research on evidence utilization in criminal justice has identified several mechanisms through which the gap is maintained. Awareness barriers: most practitioners are not regularly exposed to evaluation research findings, and the channels through which research reaches practice — journal articles, conference presentations, professional training — reach a small fraction of the operational workforce that research findings need to influence. Comprehension barriers: evaluation research is frequently written in technical language that practitioners cannot interpret without statistical training that most do not have, even when findings are substantively clear. Credibility barriers: practitioners who have witnessed ineffective programs survive repeated null findings are rationally skeptical that the next evaluation finding will produce the practice change that previous findings did not. And institutional barriers: organizations that would need to change their practices to implement evidence-based approaches face the full weight of organizational inertia, staff resistance to change, and resource constraints that make program modification difficult regardless of the evidence for doing so.
The Police Executive Research Forum (PERF)’s work on translating research into police practice, the National Institute of Corrections‘ training and technical assistance programs for correctional agencies, and the Arnold Ventures Evidence-Based Policy team’s work on supporting evidence-based criminal justice reform all represent institutional investments in the research-practice translation function that the evaluation field requires but that individual evaluators cannot provide. Building sustained partnerships between researchers and practitioners — in which evaluation is embedded in ongoing organizational learning rather than conducted as an external assessment — represents the most promising structural solution to the research-practice gap, and the one most consistent with the implementation science insight that organizational change requires active engagement rather than passive evidence provision.
The Role of Evaluator Independence
The structural independence of evaluation researchers from the programs and agencies they evaluate is the single most important determinant of evaluation credibility, and it is also among the most difficult to maintain in the institutional contexts where criminal justice evaluation typically occurs. Government funding of evaluation research creates dependency relationships between evaluators and funders that can influence research design, analysis, and reporting in favor of findings acceptable to the funder — particularly when funders have political stakes in the programs being evaluated. Program operators who control data access can restrict evaluators’ ability to examine program records, administrative databases, and the participant characteristics that rigorous evaluation requires. And publication control — the ability to delay, modify, or suppress unfavorable evaluation findings before they reach policy audiences — can prevent the negative findings that honest evaluation produces from informing the policy decisions they should influence.
The Government Performance and Results Act (GPRA) and subsequent performance management legislation have created statutory requirements for federal program evaluation that represent an institutional response to the independence problem: by requiring that agencies conduct evaluations through independent researchers or government evaluation offices with specific independence protections, GPRA-mandated evaluation provides structural protection against some of the most direct forms of evaluation capture. The What Works Clearinghouse‘s role in the Department of Education and the Evidence-Based Policy team within the Office of Management and Budget represent institutional investments in independent evidence review that provide structural protection against agency self-evaluation bias. Extending equivalent institutional independence to criminal justice program evaluation — through independent research offices, researcher data rights protections, and publication independence guarantees — would substantially improve the credibility of the evaluation enterprise across the criminal justice system.
Conclusion
Criminological policy evaluation has matured substantially from the narrative literature reviews of the mid-twentieth century to the experimental, quasi-experimental, and systematic review methods of contemporary evidence-based criminology. The research designs available to criminological evaluators — randomized trials, regression discontinuity, difference-in-differences, synthetic control, instrumental variables, and their combinations — provide the methodological toolkit for credible causal inference across a wide range of policy questions, and the systematic review and meta-analysis infrastructure of the Campbell Collaboration and allied organizations provides the synthesis framework that translates accumulated primary studies into cumulative policy guidance. This methodological progress is real and consequential.
Policy evaluation in criminology is the mechanism through which the criminal justice system learns from its own practice — identifying what works, what fails, and what the conditions are under which promising approaches can be reliably reproduced. The methodological sophistication of criminological evaluation has increased substantially over the past three decades, from the narrative literature reviews of the Martinson era through the meta-analytic revolution through the experimental criminology movement through the current emphasis on implementation science and cost-benefit analysis. This methodological progress has produced a knowledge base substantially more reliable and more actionable than the criminal justice field had access to in any prior period.
The institutional challenges — independence, communication, utilization — remain more difficult than the methodological ones and ultimately more determinative of whether evaluation knowledge changes practice. Evaluation research that is technically rigorous but institutionally captured, poorly communicated, or structurally disconnected from the decision processes it is meant to inform will not improve criminal justice practice regardless of its methodological quality. Building the evaluation infrastructure that is genuinely independent, well-resourced, effectively communicated, and structurally connected to decision-making — rather than conducted as a compliance exercise that meets funder requirements without informing operational choices — is the central institutional challenge for criminological evaluation practice, and the one whose resolution is most consequential for the field’s capacity to fulfill its evidence-based policy aspirations.
References
- Farrington, D. P., Gottfredson, D. C., Sherman, L. W., & Welsh, B. C. (2002). The Maryland Scientific Methods Scale. In L. W. Sherman, D. P. Farrington, B. C. Welsh, & D. L. MacKenzie (Eds.), Evidence-based crime prevention (pp. 13–21). Routledge.
- Kling, J. R., Liebman, J. B., & Katz, L. F. (2007). Experimental analysis of neighborhood effects. Econometrica, 75(1), 83–119. https://doi.org/10.1111/j.1468-0262.2007.00733.x
- Lipsey, M. W., & Cullen, F. T. (2007). The effectiveness of correctional rehabilitation: A review of systematic reviews. Annual Review of Law and Social Science, 3, 297–320. https://doi.org/10.1146/annurev.lawsocsci.3.081806.112833
- Sherman, L. W., & Berk, R. A. (1984). The specific deterrent effects of arrest for domestic assault. American Sociological Review, 49(2), 261–272. https://doi.org/10.2307/2095575
- Washington State Institute for Public Policy. (2024). Benefit-cost results: Criminal justice. https://www.wsipp.wa.gov/BenefitCost
- Welsh, B. C., & Farrington, D. P. (2010). The utility of cost-benefit analysis in crime prevention research. In B. C. Welsh & D. P. Farrington (Eds.), The Oxford handbook of crime prevention (pp. 505–523). Oxford University Press.