|
« AAUP Homepage
|
Dual Review: Scientific Research in Education & Evidence Matters
Reviewed by Yvonna S. Lincoln
Scientific Research in Education.
Richard J. Shavelson and Lisa Towne, eds.
Washington, D.C.: National Research Council, National Academy Press, 2002
Evidence Matters: Randomized Trials in Education Research
Frederick Mosteller and Robert Boruch, eds.
Washington, D.C.: Brookings Institution Press, 2002
For nearly three decades, the research community in the social sciences has engaged in arguments and conversations regarding the proper roles and rules of academic research. It has seemed clear that the major theoretical challenges emanating from Europe and North America, as well as the rising tide of new indigenous and postcolonial voices, would provide new, emerging models for reconsidering the primacy of positivist and postpositivist research in the United States and elsewhere. In part, these conversations have sought ways in which positivist, or conventional experimental, research could coexist with postpositivist methodologies such as phenomenological work and the work of critical theorists. Together, these three approaches might simultaneously contribute to a more precise understanding of social life and provide power, robustness, and subtlety to social policy decisions. In 1977, anthropologist Ray Rist claimed that a certain methodological détente had begun to emerge, along with a sophisticated discourse regarding what the various models of research might reasonably deliver in terms of knowledge and understanding.
These conversations and controversies revolved about the extended and extensive critiques of experimentalism's insistence on being the single best model for achieving reliable, valid, and so-called objective knowledge about social life and the social world. Experimentalism, the best known form of conventional "scientific method," is often based on random assignments of subjects to treatments, programs, or experiments. Proponents of alternative epistemologies challenged its claim to primacy and hegemony, highlighting philosophers' rejection of the possibility of genuine objectivity and the existence of multiple, sometimes conflicting, epistemologies. A sense of rapprochement had begun to settle within the social sciences, as scientists and researchers opened a dialogue around the contributions that several, rather than singular, models might make to comprehending complexity in various social and education contexts. Détente seemed achievable. With the publication of Scientific Research in Education, the 2002 report of the National Research Council's Committee on Scientific Principles for Educational Research, and Evidence Matters: Randomized Trials in Education Research, by Frederick Mosteller and Robert Boruch, it seems as though the conversation is once again closing down.
These two new books, in conjunction with provisions of the Bush presidency's national education initiative, the No Child Left Behind Act, appear to foreclose the possibility that the federal government will support research and evaluation projects that propose multiple views of knowledge and multiple research strategies to obtain that knowledge. The No Child Left Behind Act requires "rigorous testing" of its own effects, primarily through randomized field trials, or experiments in which participants, in a natural, everyday context such as a school, are randomly assigned to receive different treatments or programs.
The National Research Council committee's aim is to delineate what is science and what is not science, and one of its fundamental assumptions is that the principles it proposes in Scientific Research in Education "provide guidance for what constitutes rigorous scientific research," largely by "identify[ing] a set of principles that apply to physical and social science research and to science-based education research," with the purpose of helping "define the domain of scientific research . . . roughly delineating what is in the domain and what is not." This would be less troublesome did the committee not "reject the postmodernist school of thought when it posits that social science research can never generate objective or trustworthy knowledge," thereby discounting most of the new theoretical critiques of Western epistemologies proposed in the past quarter century. It would also be less troubling did the committee not take several examples of contemporary scholarship and theoretical and empirical work and assign them to the realm of non-science or "unscientific" work.
The argument is far broader than education. The committee adamantly insists that many forms of scholarship and inquiry do not fit the committee's criteria and are thus "not science," when in fact most of the criteria fit comfortably with multiple forms of systematic, disciplined inquiry. For the social sciences and humanities (for example, history), the committee's limited definition of science has enormous implications, virtually eliminating scholars other than those deemed "scientific" from participating in inquiry and evaluation of social and educational programs, curriculum development efforts, or policy arenas, particularly federally sponsored or funded research efforts. The general tone of Scientific Research in Education is authoritative and authoritarian; readers are left wondering why, in light of the major theoretical critiques of positivist and postpositivist science of the past decades, the volume appears to recommit to what sociologist Anthony Giddens has called "high modernism," and to return to the now-discredited fundamental philosophies and scientific practices of the modernist era.
Mosteller and Boruch's Evidence Matters sets a rather different tone. The editors have assembled a set of essays that make more or less cogent arguments for mounting randomized field trials in education. Such trials are said to produce "estimates of relative effects that are statistically unbiased" about which "one can make legitimate statistical statements about one's confidence in the results." The editors' intent is to address a concern originally raised in Great Britain "about ideology parading as intellectual inquiry." Randomized experiments solve the problems of ideology, relevance, and intelligibility of results, as sociologist Thomas Cook and graduate student Monique Payne observe in one chapter, because they are "considered the causal 'gold standard.'" The labeling of this particular form of experimentalism as a "gold standard" for scientific research proceeds from the assumption that the medical-clinical and pharmacological model for drug testing is appropriate for use in schools and classrooms—an assertion that is frequently contested by scientists and laypersons.
Several chapters of the Mosteller and Boruch work stand out as particularly useful, clear-sighted, and accessible. Research consultant Judith Gueron's work on the politics of random assignment, directed specifically toward convincing program implementers of the worth of a randomized field trial and the imperative to inform policy in a sophisticated and data-based manner, is easily read and full of practical political wisdom. Readers who are themselves practitioners or contractors in the research or evaluation arena will immediately recognize the rueful smile behind her stories of wins and losses. Boruch, research association executive Dorothy de Moya, and PhD candidate Brooke Snyder mount a strong argument for utilizing randomized field trials in both education and other national policy arenas. It is not necessary to believe they are correct in order to understand the logic and clarity of their arguments, which are well organized and well within the technical reach of a wide variety of readers.
More complex is the Cook and Payne chapter, "Objecting to the Objections to Using Random Assignment in Educational Research." As I am one of the individuals named as having "influenced the practice of many generations of young educational evaluators" and as being "probably a major cause of the impoverished current state of knowledge about what reform initiatives in American education have actually achieved," I approached the chapter with trepidation, even though the remainder of the individuals named are distinguished, amiable, and honorable company. Cook and Payne outline nine major objections, to which they have prepared nine well-reasoned, if not entirely persuasive, counterobjections. One they fail to mention—indeed, one which is never mentioned in the book—is that most randomized field trials, in clinical trials, in manpower and workforce development, in law, social welfare, and other community action sites, are targeted to adults. It may well be that more and broader obstacles are posed when the targets of even curricular experiments are very young, minor school children. As a society, Americans tend to be more concerned about the participation of minor children than about the participation of consenting adults in social experiments and proposed social policy initiatives. While educational reforms hardly pose the risk of clinical trials, most parents believe randomized field trials are best left to clinical trials and agricultural applications (for which they were first developed).
Public policy professor Carol Weiss makes a strong argument for strategies that do not necessitate randomized field trials in evaluation efforts, and points out that multiple evaluation efforts can be mounted while waiting for the researcher who makes the random assignment. Three other chapters are intrinsically interesting, although not central to the arguments posed by the other authors. David Cohen, Stephen Raudenbush, and Deborah Lowenberg Ball, well-recognized authorities in teacher education, reframe the findings of sociologists James Coleman and Christopher Jencks about factors contributing to school achievement. In the process, they suggest new ways in which traditional critical questions might be recast to be amenable to randomized experiments, and introduce a level of subtlety and sophistication to enduring questions of what works to enable achievement in all children. Public policy professor Maris Vinovskis traces a somewhat cranky history of evaluation efforts at the U.S. Department of Education, proffering some reasons why inquiry has been insufficiently scientific at the agency, and Gary Burtless, a senior fellow at the Brookings Institution, ponders why randomized field trials have been utilized somewhat successfully for policy evaluations, but rarely in education.
While similar in their calls for more rigorous "scientific" research and evaluation in the social sciences and education, the books take radically dissimilar postures on alternative forms of research. The National Research Council committee's stance, for example, is that:
It is common to see quantitative and qualitative methods described as being fundamentally different modes of inquiry—even as being different paradigms embodying quite different epistemologies. . . . We regard this view as mistaken . . . because we see quantitative and qualitative scientific inquiry as being epistemologically quite similar . . . [A]s we recognize that both can be pursued rigorously, we do not distinguish between them as being different forms of inquiry.
Cook and Payne, however, do understand that they are different, and acknowledge that the controversies proceed from alternative epistemologies. As a consequence, Cook and Payne recommend the use of qualitative methods both as adjuncts to randomized field trials and as alternatives when they are infeasible.
The most worrisome aspect of the two books is the implication thattheir stance on experimental research emanates from the federal government. The National Research Council's committee is a federally funded working group. Most, if not all, of the contributors to the Mosteller and Boruch book are large-scale contractors for federal funds. More important, the No Child Left Behind Act, a federally funded large-scale school reform effort, specifies quite clearly that primarily or solely experimental methods will be utilized in all federally financed evaluations of the school and district efforts. Common practice has been for the federal and state governments to let contracts for research and evaluation that depend on the expertise of the professional community of practice for design, and other members of the same expert professional community for vetting in a peer-review process. The expertise of blind peer review has been a cornerstone of scholarly integrity for well over a hundred years.
It is particularly ominous when the federal government specifies the particular methods and models of research to be used in pursuing answers to major policy questions.
Such specification has two untoward consequences. First, it limits potential contractors to those who practice such methods, or who believe that experimental models are the single best method for deter-mining program effects, effectively shutting out alternative epistemologies or rival social understandings. Second, the pre-specification of research methodologies permits the federal government to usurp the roles of professional disciplinary communities, effectively limiting the voices that are heard in policy circles, and narrowing broad peer review—a fundamental of free and open scholarly inquiry—to a small band of federally "approved" reviewers. This is hardly a circumstance that should be welcomed in the academic disciplines, as it echoes the partisan and highly politicized award process set up at the National Endowment for the Humanities a dozen years ago. Far from assuring objective knowledge about social programs, such constriction virtually assures a tight oligarchy of researchers, ideas, understandings, and new knowledge, with few of the insights forthcoming from alternative models of scholarly inquiry.
If these two books are taken as harbingers of continuing federal policy toward research, then discriminating, critical modulation of educational and social policy research as well as nuanced interpretation of deep social processes and hidden organizational infrastructures will disappear from the federal funding landscape for the near future. Randomized field trials have a place in policy research. The question is whether they will be the only seat in the house.
Yvonna Lincoln holds the Ruth Harrington Chair of Educational Leadership and is a Distinguished Professor at Texas A&M University.
|