Article Surfing Archive

Educational Evaluation may be inherently a process of professional judgment.

The first principle, according to Cann, is that professional judgment is the foundation for evaluation and, as such, is needed to properly understand and use all aspects of evaluation. The measurement of student performance may seem "objective" with such practices as machine scoring and multiple-choice test items, but even these approaches are based on professional assumptions and values. Whether that judgment occurs in constructing test questions, scoring essays, creating rubrics, grading participation, combining scores, or interpreting standardized test scores, the essence of the process is making professional interpretations and decisions. Understanding this principle helps teachers and administrators realize the importance of their own judgments and those of others in evaluating the quality of evalution and the meaning of the results.

To Shadish, evaluation is based on separate but related principles of measurement evidence and evaluation.

To Cann, It is quite important to understand the difference between measurement evidence (differentiating degrees of a trait by description or by assigning scores) and evaluation (interpretation of the description or scores). Essential measurement evidence skills would include the ability to understand and interpret the meaning of descriptive statistical procedures, including variability, correlation, percentiles, standard scores, growth-scale scores, norming, and principles of combining scores for grading. A conceptual understanding of these techniques, to her, is needed (not necessarily knowing how to compute statistics) for such tasks as interpreting student strengths and weaknesses, reliability and validity evidence, grade determination, and making admissions decisions. This author has indicated that these concepts and techniques comprise part of an essential language for educators. They also provide a common basis for communication about "results," interpretation of evidence, and appropriate use of data. This is increasingly important given the pervasiveness of standards-based, high-stakes, large-scale assessments.

Another point of view, offered by Shadish considerates evaluation concerns merit and worth of the data as applied to a specific use or context. It involves a systematic analysis of evidence. Like students, teachers and administrators need analysis skills to effectively interpret evidence and make value judgments about the meaning of the results.

Evaluation decision-making is influenced by a series of tensions to Cook. His basement parts of idea that competing purposes, uses, and pressures result in tension for teachers and administrators as they make assessment-related decisions. For example, good teaching could be characterized by assessments that motivate and engage students in ways that are consistent with their philosophies of teaching and learning and with theories of development, learning and motivation. Most teachers want to use constructed-response evaluation because they believe this kind of testing is best to ascertain student understanding. On the other hand, factors external to the classroom, such as mandated large-scale testing, promote different evaluation strategies, such as using selected-response tests and providing practice in objective test-taking.

These tensions, to the same author suggest that decisions about evaluation are best made with a full understanding of how different factors influence the nature of the assessment. Once all the alternatives understood, priorities need to be made; trade-offs are inevitable. With an appreciation of the tensions teachers and administrators will hopefully make better informed, better justified assessment decisions.

Evaluation influences student motivation and learning. Wilde and Sockey have used the term 'educative evaluation' to describe techniques and issues that educators should consider when they design and use evaluation methods. Their message is that the nature of evaluation influences what is learned and the degree of meaningful engagement by students in the learning process. While Wiggins contends that evaluation tools should be authentic, with feedback and opportunities for revision to improve rather than simply audit learning, the more general principle is understanding how different evaluations affect students. Will students be more engaged if evaluation tasks are problem-based? How do students study when they know the test consists of multiple-choice items? What is the nature of feedback, and when is it given to students? How does evaluation affect student effort? Answers to such questions help teachers and administrators understand that evaluation has powerful effects on motivation and learning.

Teachers and administrators, to Shadish, need to not only know that there is error in all classroom and standardized evaluation, but also more specifically how reliability is determined and how much error is likely. With so much emphasis today on high-stakes testing for promotion, graduation, teacher and administrator accountability, and school accreditation, it is critical that all educators understand concepts like standard error of measurement, reliability coefficients, confidence intervals, and standard setting.

To Cann two reliability principles deserve special attention. The first is that reliability refers to scores, not instruments. Second, teachers and administrators need to understand that, typically, error is underestimated.

COOK, J. Evaluating Knowledge Technology Resources. LTSN Generic Centre, 2002.

CANN. E et al. English Language Arts: A Curriculum Guide for the Middle Level (Grades 6-9). Saskatchewan Education. 1998.

HIRSCHMAN, L; THOMPSON, H. Overview of Evaluation in Speech and Natural Language Processing. In J. and Mariani, editor, State of the Art in Natural Language Processing, pages 475 -- 518.

SHADISH, W. Some evaluation questions. Practical Assessment, Research & Evaluation, 6(3), 1998.

WILDE, J.; SOCKEY, S. Evaluation Handbook. Clearinghouse. 2000.