V. Weisburd’s Principles to Overcome Ethical, Political, and Practical Problems in Experimentation
Randomized experiments are often excluded in criminal justice and criminological research for either ethical, political, or practical concerns. However, in reality randomized experiments are possible and appropriate in many circumstances. For experimenters, the challenge is to identify the conditions under which experiments can be successfully implemented in criminal justice settings. Weisburd (2000) identified eight principles to help practitioners and researchers assess when experimentation is most feasible. This section presents Weisburd’s principles and summarizes his discussion of each. The first two principles involve ethical concerns, the next three principles involve political concerns, and the final three principles involve practical problems in criminal justice experimentation.
Principle 1: In the case of experiments that add additional resources to particular criminal justice agencies or communities or provide treatments for subjects, there are generally fewer ethical barriers to experimental research (Weisburd, 2000, p. 184). It is important to differentiate the nature of the criminological intervention to be evaluated at the outset of the evaluation. Ethical problems are not likely to be raised when researchers provide new resources to offenders, such as rehabilitative services, or to communities, such as additional police patrols. The assumption is that the control group will continue to receive traditional levels of criminological intervention. Criminal justice experiments are often framed as tests of whether a new intervention is better than an existing one. However, when treatment is withdrawn from control subjects, serious ethical questions will arise; thus, Weisburd suggested that crime and justice experiments can often be defined as including treatment and comparison groups, rather than treatment and control groups.
Principle 2: Experiments that test sanctions that are more lenient than existing penalties are likely to face fewer barriers than those that test sanctions more severe than existing penalties (Weisburd, 2000, p. 185). So-called sanctioning experiments have produced the most serious ethical problems in criminal justice experimental study. In these evaluations, random allocation rather than the traditional decision-making power of criminal justice practitioners is used to make decisions about the processing of individual offenders (Weisburd, 2000). Arrest, sentence, and imprisonment decisions are based on random allocation. It is important to remember that sanctioning experiments allocate sanctions that are legally legitimate to impose on offenders. Ethical concerns are raised in connection with how the sanction is applied rather than to the harshness of the sanction itself. Clearly, there needs to be a balance between the criminal justice system’s need to find answer to important policy questions and its commitment to equity in allocating sanctions. Weisburd suggested that, when designing sanctioning experiments, questions be framed in a way that allows ethical barriers to be removed—for instance, by using the experiment to test whether the criminal justice can be lenient in the allocation of sanctions rather than being harsh. The California Reduced Prison Experiment released some offenders from prison earlier than their sentenced release date (Berecochea & Jaman, 1981). Leniency for a few generated no major ethical objections despite thousands of offenders who were left in prison for longer periods of time on the basis of a random allocation scheme. However, the end result was two distinct groups that received more or less punitive sanctions.
Principle 3: Experiments that have lower public visibility will generally be easier to implement (Weisburd, 2000, p. 186). Obviously, in additional to ethical concerns there are political costs to criminal justice experimentation. Although reducing penalties for certain offenders may not generate ethical objection, the approach may generate strong political resistance to the experiment. Citizens may not want offenders to return to the community before their natural sentence expiration date. Citizens may also exert political pressure to halt an experiment if additional criminal justice resources are randomly allocated. For instance, in the Jersey City Drug Market Analysis Experiment (Weisburd & Green, 1995), citizens in comparison drug market hot-spot areas were very concerned that they were not receiving the increased police attention given to the treatment drug market hot-spot areas. As Weisburd (2000) observed, this problem is similar to those encountered in medical research where interest groups fight to have experiments abandoned so medication will be provided to all who might benefit from it. These political problems are less likely to emerge when experiments are less visible to the public. As such, researchers should resist the temptation to publicize experiments before they are completed.
Principle 4: In cases where treatment resources are limited, there is generally less political resistance to random allocation (Weisburd, 2000, p. 186). There are circumstances in which it can be easier for researchers to defend random allocation in the context of the politics of the allocation of treatments (Weisburd, 2000). Often, treatments and new programs can be applied to only a few areas or a small number of individuals. When communities or individuals understand that they have not been systematically excluded from additional resources, experiments do not provide larger political problems when compared with nonexperimental evaluation designs. For instance, in the High Intensity Drug Trafficking Area drug treatment experiment (Weisburd & Taxman, 2000), practitioners were much less resistant to an experimental design because they could not provide treatment to all eligible subjects. As Weisburd suggested, random allocation can serve as a type of pressure valve in the allocation of scarce criminal justice resources. Random allocation can be a politically safer basis on which to apply treatment when compared with other criteria.
Principle 5: Randomized experiments are likely to be easier to develop if the subjects of the intervention represent less serious threats to community safety (Weisburd, 2000, p. 187). When the potential risks to the community are minimized, it is much easier for policymakers and practitioners to defend the use of randomization (Weisburd, 2000). Very few experiments have involved high-risk violent offenders that would generate serious threats to community safety.
Principle 6: Experiments will be most difficult to implement when the researcher attempts to limit the discretion of criminal justice agents who generally operate with a great degree of autonomy and authority (Weisburd, 2000, p. 187). Relative to ethical and political concerns, practical barriers have generally been more significant in explaining resistance to criminal justice experimentation (Weisburd, 2000). Even though there is a wide range of methodological issues facing experimenters, it can be very difficult to get practitioners to agree to random allocation. Although this problem is related to the ethical and political concerns already discussed, it is important to recognize that random allocation interferes with the daily operations of the affected agencies. Judges are generally more resistant to random allocation when compared with other criminal justice practitioners. They have been known to subvert experiments by not properly assigning subjects even when they have agreed to random allocation of sanctions and programs. For example, in the Denver Drunk Driving Experiment (Ross & Blumenthal, 1974) judges were supposed to randomly allocate fines and two different types of probation to convicted drunk drivers. Unfortunately, in more than half the cases judges circumvented the randomization process in response to defense attorney pleas for their clients to receive fines rather than probation. Weisburd observed that the likelihood of success in randomization is linked to the nature of the decisions being made. He suggested a subprinciple in developing randomization procedures: “Where treatment conditions are perceived as similar in leniency to control conditions, it will be easier to carry out a randomized study involving high-authority and high-autonomy criminal justice agents” (Weisburd, 2000, p. 188).
In Project Muster, a probation experiment in New Jersey, Weisburd (1991) found that the judges correctly randomized nearly all study subjects. In this evaluation, judges were asked to sentence selected probationers who violated release conditions by not paying their fines to a program that involved intensive probation and job counseling. No restraint was placed on their sentencing decisions for other violated probationers. Because few violated offenders would have been sentenced to jail for failure to pay fines, judges did not feel that their discretion was overly compromised in selecting Muster instead of traditional probation.
Principle 7: Systems in which there is a strong degree of hierarchical control will be conducive to experimentation even when individual actors are asked to constrain temporarily areas where they have a considerable degree of autonomy (Weisburd, 2000, p. 188). Weisburd suggested that, in militaristic hierarchical agencies, such as the police and certain correctional agencies, it is often easier to execute experimental designs because such agencies have rigid organizational structures. This is particularly true when the discretion is limited for the targets selected rather than the choice of action or decision. Experiments in Minneapolis, Minnesota (Sherman & Weisburd, 1995), and Jersey City, New Jersey (Braga et al., 1999; Weisburd & Green, 1995), were executed with success when police officers were focused on treatment crime hot spots and restricted from operating in control hot spot areas. In policing, hierarchical control also explains why it has been possible to implement experiments in which treatment and control conditions vary significantly and the line-level agent has traditionally exercised considerable autonomy (Weisburd, 2000). This was evident in the six domestic violence experiments supported by the National Institute of Justice in which misdemeanor spouse abusers were randomly assigned to either arrest or nonarrest conditions (Sherman, 1992). These studies did not show the extensive subversion to randomization seen in other criminal justice experiments, such as the Denver Drunk Driving Experiment.
Principle 8: Where treatments are relatively complex, involving multiple actions on the part of criminal justice agents or actions that they would not traditionally take, experiments can become prohibitively cumbersome and expensive (Weisburd, 2000, p. 190). Once randomization has been successfully achieved, maintaining the integrity of the treatment is the most difficult task for experimenters (Boruch, 1997; Weisburd, 2000). Experiments cannot be simply a before-and-after effort by researchers; it is very important to document and analyze what is actually happening in the treatment and control groups (Weisburd, 2000). Developing methods to monitor and ensure the integrity of the treatment is crucial. If the treatment is not implemented properly, it would not be surprising to find that the intervention did not generate an effect. For studies that involve one-shot interventions, this process can be relatively simple (e.g., documenting whether a subject was properly placed in a condition such as arrest, violation, or incarceration). However, if experimental treatments are complex, it will be correspondingly more difficult, time-consuming and costly to track and ensure the integrity of the treatment.
There is now a large, and growing, literature indicating that ethical, political, and practical barriers can be overcome and that randomized experiments are appropriate in a very diverse group of circumstances and across many aspects of decision making in the criminal justice system (Boruch, Snyder, & DeMoya, 2000; Petrosino et al., 2001; Weisburd, 2000, 2003). To some observers (e.g., Weisburd, 2003), the failure of crime and justice funders and evaluators to develop a comprehensive infrastructure for experimental evaluation represents a serious violation of professional standards:
A related line of argument here is that a failure to discover whether a program is effective is unethical. That is, if one relies solely on nonrandomized assessments to make judgments about the efficacy of a program, subsequent decisions may be entirely inappropriate. Insofar as a failure to obtain unequivocal data on effects then leads to decisions which are wrong and ultimately damaging, that failure may violate good standards of both social and professional ethics. Even if the decisions are “correct” in the sense of coinciding with those one might make based on randomized experiment data, ethical problems persist. The right action taken for the wrong reasons is not especially attractive if we are to learn anything about how to effectively handle the child abuser, the chronically ill, . . . and so forth. (Boruch, 1975, p. 135)
According to Weisburd (2003), the key question is why a randomized experiment should not be used: “The burden here is on the researcher to explain why a less valid method should be the basis for coming to conclusions about treatment and practice” (p. 352).