B. Quasi-Experimental Designs
The quasi-experiment is a research design that has some, but not all, of the characteristics of a true experiment (Cook & Campbell, 1979). As such, quasi-experiments do not have the same high degree of internal validity as randomized controlled trials. Although there are many types of quasi-experimental research designs, the element most frequently missing is the random assignment of subjects to the treatment and control conditions. In developing an equivalent control group, the researcher often uses matching instead of randomization. For example, a researcher interested in investigating the effects of a new juvenile curfew on crime in a particular city would try to find a city with similar crime rates and citizen demographics in the same geographic region. This matching strategy is sometimes called a nonequivalent group comparison design because the treatment and control cities will not be exactly the same. In the statistical analysis of quasi-experimental data, researchers will often attempt to isolate treatment effects by including covariates to account for any measurable factors that could also influence observed differences in the dependent variable (e.g., poverty levels, youth population size, and the like). This results in less confidence in study findings than true experimental approaches because it is possible that the difference in outcome may be due to some preexisting difference between the treatment and control groups that was not taken into account by the evaluators.
Quasi-experimental interrupted time series analysis, involving before-and-after measurements for a particular dependent variable, represents a common type of evaluation research found in criminology and criminal justice. One of the intended purposes for doing this type of quasi-experimental research is to capture longer time periods and a sufficient number of different events to control for various threats to validity and reliability (Cook & Campbell, 1979). Long series of observations are made before and after the treatment. The established before-treatment trend allows researchers to predict what may have happened without the intervention. The difference between what actually happened after the intervention and the predicted outcome determines the treatment effect. These approaches are often criticized for not accounting for other confounding actors that may have caused the observed differences. It can also be difficult to model the trend in the time series so the treatment effect can be properly estimated. For instance, in their evaluation of the 1975 Massachusetts Bartley–Fox gun control law that mandated a year in prison for illegal carrying of firearms, Deutsch and Alt (1977) used an interrupted time series quasi-experimental design and found that the passage of the law was associated with a statistically significant reduction in armed robbery in Boston. However, Hay and McCleary (1979) reanalyzed these data using a different quasi-experimental time series modeling approach and found no statistically significant reduction in Boston armed robberies associated with the passage of the law. In contrast, Pierce and Bowers (1981) found statistically significant violence reductions associated with the passage of the law using quasi-experimental interrupted time series analysis with multiple control group comparisons.
Although these designs are still likely to have lower internal validity than randomized experimental evaluations, quasi-experiments that combine the use of a control group with time series data can sometimes produce results that are of similar quality to randomized controlled trials (Lipsey &Wilson, 1993). Some researchers, however, have found that even strongly designed quasi-experiments produce less valid outcomes when compared with well-executed randomized controlled trials (see Weisburd, Lum, & Petrosino, 2001). In general, the persuasiveness of quasi-experiments should be judged on a case-by-case basis (Weisburd et al., 2001). For experimental criminology, the implication is that randomized controlled trials are necessary to produce the most valid and unbiased estimates of the effects of criminal justice interventions.
In their evaluation of the Operation Ceasefire gang violence reduction strategy, Braga and his colleagues (Braga, Kennedy, Waring, & Piehl, 2001) used a quasi-experimental interrupted time series analysis with multiple comparison groups to compare youth homicide trends in Boston with youth homicide trends in other major U.S. cities. They found a statistically significant 63% reduction in youth homicides associated with the implementation of the Ceasefire strategy. The evaluation also suggested that Boston’s significant youth homicide reduction associated with Operation Ceasefire was distinct when compared with youth homicide trends in most major U.S. and New England cities (Braga et al., 2001).
The National Academies Panel on Improving Information and Data on Firearms (Wellford, Pepper, & Petrie, 2005) concluded that the Ceasefire evaluation was compelling in associating the intervention with the subsequent decline in youth homicide (see also Morgan & Winship, 2007). However, the panel also suggested that many complex factors affect youth homicide trends and that it was difficult to specify the exact relationship between the Ceasefire intervention and subsequent changes in youth offending behaviors (see also Ludwig, 2005). Although the Ceasefire evaluation controlled for existing violence trends and certain rival causal factors, such as changes in the youth population, drug markets, and employment in Boston, there could be complex interaction effects among these factors not measured by the evaluation that could account for some meaningful portion of the decrease. The evaluation was not a randomized controlled experiment; therefore, the nonrandomized control group research design cannot rule out these internal threats to the conclusion that Ceasefire was the key factor in the youth homicide decline. Other quasi-experimental evaluations face similar critiques when attempting to unravel cause and effect associated with the implementation of specific criminal justice intervention.
Another type of quasi-experimental design is known as a natural experiment, whereby nature, or some event, has created treatment and control groups. In contrast to laboratory experiments, these events are not created by scientists, but they yield scientific data nontheless. The classic example is the comparison of crime rates in areas after the passage of a new law or implementation of a crime prevention initiative that affects one area and not another. For instance, the 1994 Brady Handgun Violence Prevention Act established a nationwide requirement that licensed firearms dealers observe a waiting period and initiate a background check for handgun sales. To assess the impact of the Brady Law on violence, Ludwig and Cook (2000) examined trends in homicide and suicide rates, controlling for population age, race, poverty, and other covariates, in the 32 “treatment” states directly affected by the Brady Act requirements and compared them with the 18 “control” states and the District of Columbia, which had equivalent legislation already in place. They found that the Brady Act appeared to be associated with reductions in the firearm suicide rate for persons age 55 years or older but not with reductions in homicide rates or overall suicide rates.
C. Nonexperimental Designs
Studies that rely only on statistical controls are often seen as representing the weakest level of confidence in research findings (Cook & Campbell, 1979; Sherman et al., 1997). These studies are typically called nonexperimental or observational research designs. In these studies, researchers do not vary treatments to observe their effects on outcomes; instead, they examine natural variation in a dependent variable of interest, such as crime, and estimate the effect of an independent variable, such as police staffing levels, on the basis of its covariation with the dependent variable. Additional covariates related to variation in the dependent variable will be included in the model as statistical controls to isolate the effect of the key independent variable of interest. The difficulty of this approach is that there could easily be excluded factors related to both the key independent variable and dependent variable that bias the estimated relationship between these variables. Unfortunately, for some sensitive areas in crime and justice, nonexperimental research designs are the only method of investigation possible. Although some scholars argue that it is possible to develop statistical models that provide highly valid results (e.g., Heckman & Smith, 1995), it is generally agreed that causes unknown or unmeasured by the researcher are likely to be a serious threat to the internal validity of nonexperimental research designs (Cook & Campbell, 1979).