IV. Critiques of Experimentation in Criminology
Randomized experiments present many challenges. For instance, there are often problems of getting permission and cooperation from policymakers and practitioners that lead to case flow problems and difficulties in successfully achieving randomization. Although there is a large literature examining the barriers to experimentation (e.g., Baunach, 1980; Heckman & Smith, 1995; Petersilia, 1989), Clarke and Cornish (1972) raised several concerns with experimentation in crime and justice that had a major chilling effect on the development of experimental research in England during the 1970s (Farrington &Welsh, 2006). Although several experimental criminologists have responded to these concerns (e.g., Farrington, 2003; Weisburd, 2003), and the number of crime and justice experiments have increased in England and the United States over the last 25 years (Farrington & Welsh, 2006), the issues raised by Clarke and Cornish (1972) continue to be influential in resisting experimental methods in crime and justice today (see Pawson & Tilley, 1997).
Clarke and Cornish’s (1972) critique of experimentation is based on their experience in implementing a large-scale randomized experiment to evaluate a therapeutic community at a training school for delinquent boys in England. One major concern was that practitioners undermined the experiment by limiting the number of boys who could be considered for random allocation. Practitioners were very concerned that the boys would not receive the treatment that was most suitable for them. They felt that it was unethical for the boys to receive anything less than the most appropriate treatment. This led to the research being extended for a much longer time period and eventually stopped before the desired number of cases for the study was achieved. However, in response, experimental criminologists suggested that the ethical questions raised by the practitioners had more to do with contrasting belief systems between practitioners and researchers rather than the ethics of experimentation in crime and justice (Weisburd, 2003). The practitioners believed they knew which treatments worked best for the boys. Researchers, however, thought that the effectiveness of treatment was not clear and implemented a randomized study to determine what worked. As a result, the practitioners undermined the experimental evaluation.
Another concern put forth by Clarke and Cornish (1972) referred to the difficulty of generalizing from experimental studies (i.e., the problem of external validity). They argued that the unique institutional settings at the training school were difficult to disentangle from the treatment itself. Clarke and Cornish further argued that institutions that agree to experimentation are a self-selected group that are not representative of the general population of institutions; as such, experiments that include them tell one little about the workings of the treatment and their outcomes in the real world. In response, experimental criminologists argue that support for experimentation from larger governmental agencies, such as the U.S. National Institute of Justice, would encourage broader involvement of institutions in experimentation (Weisburd, 2003). Although this larger group is still likely to be self-selected and generalizability may still be limited, encouragement rather than discouragement of experimental study in crime and justice by funders would lead to the development of more generalizable experiments (Weisburd, 2003).
The strongest criticism of randomized experiments raised by Clarke and Cornish (1972) was that experimental studies are too rigid to address the complexity of crime and justice contexts. The treatment at the training school involved many components that varied over the course of the experiment; thus, it was impossible to clearly define the treatment being tested. Whatever evaluation results were obtained would not have explainable. In essence, the experiment might have been able to say what happened, but not be able to answer how or why it happened (Weisburd, 2003, p. 349). Pawson and Tilley (1997) argued that experiments tend to be inflexible and use broad categorical treatments; as a result, experimental designs miss the important interaction between the nature of the treatment and the nature of the subjects being studied. Experimental criminologists acknowledge that the use of experimental approaches in complex social settings requires the development of experimental methods that are capable of addressing the complexity of crime and justice treatments, settings, and subjects (Weisburd, 2003). Of course, this requires a commitment to institutionalize experimental methods in the crime and justice field.