The Thrust and Parry of Assesment

Faculty should learn to stop before damage is done, like a good fencer, when dealing with the move toward standardized testing.

By Sharon J. Hamilton and Trudy W. Banta

Since the publication of A Nation at Risk: The Imperative for Educational Reform by the U.S. Department of Education in 1983, the Damocles sword of assessment has hung over higher education: either develop an effective means to assess student learning or the government will impose a standardized means of assessing learning. Last year, that sword came a little closer to our vital organs when Secretary of Education Margaret Spellings’s Commission on the Future of Higher Education concluded that “higher education institutions should measure student learning using quality assessment data,” and that “the results of student learning assessment, including value-added measurements that indicate how much students’ skills have improved over time, should be made available to students and reported in the aggregate publicly.”

These recommendations have spurred a whirlwind of activity involving increased development of standardized tests, challenges to accrediting bodies, and responses from major higher education associations. This article considers two of the many resultant questions that are shaping the national conversation: Where are we nationally in relation to the assessment of student learning? And what should colleges and universities be doing to retain control of the evaluation of student learning?

National Context

Many of our faculty colleagues declare, “We assess our students’ work to ensure that they have learned the course material, and we give them grades.” That response helps to demonstrate where we are nationally. We are more aware now of the evaluative work that faculty have been doing for years. But we are also more aware that the evaluation of student learning in individual courses must be aggregated across groups of students and across programs to provide evidence of group strengths and weaknesses. This kind of information helps us to know where we can improve in instruction and curricular design. We also realize that we have to document our assessment of student learning to the vast array of people, on campus and off, who have a stake in higher education. These two functions of assessment—improving learning and documenting the results to external stakeholders— are a frequent source of tension between faculty, who are primarily interested in using assessment to improve the learning of individual students in their individual courses, and administrators, who are held accountable by various constituencies for student learning at the institutional level.

The accountability movement emerged during the 1980s as colleges and universities were compelled to compete with highways, prisons, and various health and human services programs for rapidly diminishing public funds. No longer an untouchable and irreproachable “ivory tower,” higher education needed to demonstrate that it provided sufficient value for precious tax dollars. By 1985, Virginia, New Jersey, Colorado, and Tennessee had enacted mandates for colleges and universities to document student learning.

The push for accountability through assessment continued into the 1990s. This push was driven partly by the Continuous Quality Improvement Movement, which was adapted for higher education from Masaaki Imai’s Kaizen: The Key to Japan’s Competitive Success (1986). Imai’s book develops the notion of kaizen (literally, good change or improvement) as a “meta” process for managing improvement based on continuous incremental progress. Also contributing to the push for assessment in the 1990s were accreditors, who began to require that institutions of higher learning demonstrate accountability.

Approaches to Assessment

Common sense and decades of education research would indicate that a rich and context-based approach to assessing learning would work better than a sterile and distant standardized approach. It is challenging enough to try to create a picture of all the different ways faculty assess individual learning at the department level; how much more challenging, then, to aggregate evidence of student learning for the purpose of comparing institutions?

Standardized tests can look enormously appealing as a comparatively easy, albeit expensive, way to aggregate and compare student learning within and across institutions. Since stakeholders are most interested in how students can perform in their major fields, standardized tests of disciplinary learning will pay far greater dividends than standardized tests of general intellectual skills. However, the most prolific development and implementation of standardized tests in the past two years has been in the area of general intellectual skills.

The National Association of State Universities and Land-Grant Colleges and the American Association of State Colleges and Universities have responded to the call for greater accountability by jointly developing the Voluntary System of Accountability (VSA). The VSA will include standardized measures of student engagement and the general intellectual skills of critical thinking, analytic reasoning, and written communication with the aim of demonstrating the value added by a college education at the undergraduate level.

In his first public presentation of the VSA at a symposium in Indianapolis in September 2007, George L. Mehaffy of the American Association of State Colleges and Universities demonstrated the template to some 150 faculty members. Damocles’s sword dropped even further, as the faculty members envisioned how what was now “voluntary” could very easily become mandatory. While they agreed with Mehaffy that much of the VSA incorporated information already available on most institutions’ Web sites, they articulated grave concerns about standardized tests for critical thinking, communication, and analytic reasoning. Their reservations ranged widely: some worried about the potential for error when comparing institutional aggregates of student learning between and among institutions with different missions and contexts for testing; some were concerned about the need for authentic, context-based evidence of what students actually know and can do. Out of their concerns and ensuing discussions emerged a possible solution.

To retain control of assessment at the institutional level, faculty and administrators are going to have to learn as much as possible about multiple modes of assessment and what each mode can tell us. A standardized test is merely one mode of assessment that can provide information about some aspects of student learning. It will grow into a bête noire only if it becomes the key modality. We need to develop a multimodal approach that uses the best instruments available to answer the most urgent and important questions about learning. In her article in the spring 2007 issue of Peer Review, Trudy W. Banta describes a host of other modes of assessment, including the National Survey of Student Engagement; surveys of alumni and employer satisfaction; retention and graduation statistics; and measures of the proportion of students engaged in experience-based learning opportunities such as undergraduate research, study abroad, and service learning. All of these have the potential to be standardized to provide data for institutional comparisons.

Electronic Portfolios

Electronic portfolios, which provide digital collections of evidence to document the achievement of specified goals and outcomes, are the ultimate tool for combining multiple assessment modes into a coherent presentation of learning. In 1998, the Pew Charitable Trusts funded a $2.5-million grant to six urban universities across the nation to develop electronic institutional portfolios to document how effectively they were achieving their missions. The challenge to document improvement and achievement in student learning for these institutional portfolios, together with the need to document student learning for regional and disciplinary accrediting agencies, spurred the development of electronic student portfolios. For almost five years, the National Coalition for ePortfolio Research and other portfolio consortia have been exploring how electronic portfolios can relieve the tension between assessing for accountability and assessing for improvement by integrating the two. Too many different models have been developed to outline here, so we will focus on just one: the student electronic portfolio in use at Indiana University–Purdue University Indianapolis.

Student electronic portfolios at IUPUI are designed to document both achievement and improvement in student learning by integrating campus learning outcomes of general intellectual abilities (which we call principles of undergraduate learning) with learning in the major. The principles of under - graduate learning include the three intellectual skills being assessed by the standardized tests recommended for the VSA: critical thinking, written communication, and analytic reasoning.

Seven departments are currently developing different models for integrating the principles with targeted assignments that will be assessed in discipline-specific coursework so that they don’t add to the faculty workload. For example, the Department of Secondary Education used to require lesson plans in several different courses throughout the baccalaureate program, with no coordination to encourage improved conceptual understanding of lesson planning from year to year in the program. With the grant, faculty have identified key stages in the undergraduate program where lesson plans will be required, and presented in the electronic student portfolio, with increasingly higher conceptual levels of understanding described in the evaluative rubric. Part of the rubric used to assess these classroom assignments will address how students apply critical thinking, written communication, or understanding of diverse cultures within the assignments. These assessments of general intellectual skills can be aggregated and reported in the institutional portfolio, which is designed to demonstrate how IUPUI is achieving its mission and goals. Additionally, test scores on national board exams can be aggregated for the institutional portfolio, as can surveys of alumni, employers, and graduate schools that receive our students. In this model, students build an individual portfolio full of authentic evidence of their learning that they can transform into resumes, scholarship applications, and graduate school applications. Concurrently, the institution can build an institutional portrait of student learning based on aggregated reports of authentic evidence of student learning, with the rubric and sample evidence available for scrutiny and amplified by standardized surveys.

Within their disciplines, faculty can develop rubrics to help them to measure what students are learning. Rubrics can be shared across campuses, and, through blind scoring of selected assignments, they can be used for comparison.

Course or Program?

The conceptual frameworks for good assessment are being established, but there is still much more pioneering work to be done. One major aspect of the work involves a change of thinking from “my course” to “our program” as faculty come to consensus on rubrics to help them assess what their students are learning. Additionally, faculty will need to come to consensus across disciplines on how to score students’ written reflections on the skills that cross disciplines, the common learning outcomes. Finally, if we want to move toward multi-institutional accountability, the faculty from different institutions will need to agree about what they mean by critical thinking, written communication, and analytic reasoning. This is important faculty work, but, particularly at the beginning, it is additional work that requires additional resources.

Ultimately, faculty will highlight their vested interest in improving student learning as they accommodate the need for demonstrating accountability for student learning. Therein lies the basis for the fencing allusion in our title. Fencers are asked to arrêt à bon temps—to stop before damage is done. With diminishing resources of faculty time and funding for higher education, do we want to spend millions of dollars and millions of hours of faculty, staff, and student time purchasing, administering, and scoring standardized tests? Or might it be preferable to develop an electronic portfolio system capable of respecting the work and assessment of individual faculty members and students while concurrently providing the capability for comparisons based on faculty-developed rubrics for learning? Shall we arrêt à bon temps the increasing momentum toward standardized testing to provide at least equal resources for portfolio assessment? Might we then envision a time when faculty approach assessment with less of an attitude of arrêt à bon temps and more of an enthusiastic laissez les bon temps rouler because assessment will be used both to improve the learning of students in their classrooms and to document that learning to others in formats that allow for comparison?

Sharon J. Hamilton is associate vice chancellor for academic affairs and chancellor’s professor of English at Indiana University–Purdue University Indianapolis. Trudy W. Banta, also at IUPUI, is professor of higher education and senior adviser to the chancellor for academic planning and evaluation.

 Printer-Friendly Page