CHAPTER 15

Automated Testing

Of the research in learning, no strategies have led to better outcomes than mastery learning. In mastery learning, the learner is required to master material to a specified level before they can go forward. In Human Characteristics and School Learning, Bloom [1976] makes an excellent case for mastery learning.

If you are interested in automated testing, you'll need more than casual computer support. The testing support available within courseware packages (Chapter 3) is still too limited to support mastery learning.

Mastery learning involves having students demonstrate mastery of material at a defined, high level before they proceed with new material. Two essential elements of this instructional strategy are corrective, enriching feedback and congruence among the instructional components. In Keller Plan courses, quizzing is used for the corrective feedback. Mastery learning is reviewed in Implementing Mastery Learning [Guskey, 1996]. Bloom [1984] suggests that one-to-one tutoring is the most effective strategy known, generally yielding two standard deviations better performance than traditional instruction. He suggests further that mastery learning approaches one-to-one instruction in terms of measured learner gains. When students adopt self-regulatory strategies aimed at mastery learning, these have been associated with positive, achievement oriented activities [Elliot and Dweck, 1988]. Incorporating mastery systems into computer-based instruction has led to excellent learning outcomes [Montazemi & Wang, 1995]. When objective assessments are employed, prior knowledge is an exceptionally strong predictor of learning [Dochy et al, 1999].

Should students make their own decisions about monitoring their progress, or should control be retained by computerized testing? As noted in Chapter 11, Garbin finds that offering lots of information gives students with different styles the opportunity to learn in their own way. Penn & Nedeff [2000] describe a Web-based system for organic chemistry. They demonstrate higher earned test scores for those students making the largest number of attempts on Web-based practice system. Garhart & Hannafin [1986] report that students who control their own decisions about when to terminate instruction often do so prematurely.

The concern that many faculty have about Web testing relates to ensuring who actually completes the test items. UNL has built a testing laboratory. Students can practice test over the Web but, in many courses, they must come to the testing laboratory in order to take tests that count. Testing laboratories are becoming a part of the modern campus landscape.

In a graduate statistics course, Charles Ansorge has students identify a proctor who meets certain criteria. Distant students are then tested in the proctor's presence. Ansorge’s procedure includes having the proctor both sign on and sign off for the test. Faculty report using the strategy of phoning students during the test — but this has serious limitations. How do you handle calling into the one-line household in which the one line is connecting the student to the Web?

Assessment and Evaluation of Distance Learners was a PBS Teleconference (12/99), addressed this issue. There is tremendous interest in this area among educators. It often is discussed in journals that deal with online teaching [for example, see Carlson, 2000].

Orr's software is not restricted to any particular discipline; it is being used widely throughout the University of Nebraska—Lincoln campus. Orr responds to collegial suggestions and, as a result, the power of his system improves incrementally.

We have done research on automatic testing on the Web. This research is less concerned with traditional testing for assigning grades to students, and more with mastery learning by practice. The testing is designed to keep learners working on a question topic until they "get it right." The questions are generated by the computer, as opposed to pulling them from a bank of questions. The items have great variety. The information is stored in a database, and the questions are generated with encrypted codes. Students respond to quiz items that may be presented in any of ten formats. Upon receipt, the feedback provided to a student’s response is not only context-related, but it may be shaped by features in the student’s response. In a few cases, responses are checked for common misconceptions, and tutoring to address those misconceptions is provided in the student's feedback.

The verification strategy to ensure academic integrity is to sample small portions of the required material after students have completed the entire package. If the small sample indicates that successful learning has occurred, the assumption is the student has learned all of the material as well.

Although Web-based testing is possible at remote sites, many faculty want to administer proctored tests. UNL has established a testing laboratory. In this laboratory, students are proctored while they take tests. The testing student sits at a terminal, indicates the test to be taken, and logs in to the test. A proctor then verifies identity using the student's photo ID. Upon identification of the testing student, the proctor completes the log-in procedure. At the end of the test, a proctor logs the student out before the test grade and full results are made available. The laboratory shown in Figure 15.04 is being expanded to include about 60 stations.