Fein Associates

Test Development for Certification Programs

By Melissa D. Fein

Test development for certification programs follows the same process as for any other mastery test development situation. The first step involves defining a body of knowledge from which measurable objectives can then be developed. A table of specifications can be outlined to define the degree of emphasis that should be put on specific objectives. Using the table of specifications, items, along with any scoring tools needed to grade the items, can be developed from the objectives. After the items have been developed and finalized, they can be examined for quality. The final step in the development of tests for certification programs is to determine the score, indicating whether the examinee should earn certification.

Writing Measurable Objectives
Objectives, when properly written, specify measurable and observable behaviors; they are not just a summary or topical list of the elements comprising the content of the body of knowledge. Properly written objectives specify both content and a behavior. Some behaviors that are commonly elicited include recalling or paraphrasing facts, analyzing, making comparisons, making inferences, making predictions, synthesizing, judging, and critiquing. There are a number of different taxonomies that have been developed to help organize objectives. What these taxonomies share is the differentiation of rote memorization of facts and the use of high-level thinking skills.

Item Writing
Multiple-choice items can be a very efficient way to cover a large number of objectives. Carefully constructed multiple-choice items can be used to test higher level thinking skills as well as the knowledge of basic facts. It is important to craft high quality distracters. The term "distracters" refers to the incorrect item options. They are designed to "distract" the examinee who has not achieved complete mastery of the objective. Statistical measures that are used to examine the quality and effectiveness of an item include:

• item difficulty
• item discrimination coefficient
• distracter effectiveness coefficient
• point biserial correlation

Some common pitfalls that can affect the quality of an item are giving grammatical clues to the answer within the question or the distracters, and making the "correct" choice much longer than the distracters.

Validity and Reliability
Once a test of high-quality items has been constructed, test validity and reliability measures can be used to assess the quality of the test. Validity pertains to the extent to which the items in a test reflect the objectives they are intended cover. Reliability pertains to consistency of the results (test scores). Although subject matter experts can assess validity, both validity and reliability can be measured with the use of correlation coefficients.

Writing Rubrics
Another type of reliability is rater reliability. When multiple choice or true-false test items are used, the scoring of exams is easily accomplished. When items are performance-based, it becomes more important to invest time in writing scoring rubrics. The appropriate time to think about scoring rubrics is when the items are being written. There should be a clear correspondence among the objective, item, and scoring rubric. If the exam item and the scoring rubric are clearly structured, multiple raters should be able to use the rubric and come up with the same, or a very similar score. Sometimes raters need to be trained in order to use a particular scoring rubric.

Determining Mastery
A "Modified Angoff Procedure" can be used to set a mastery cut point. The mastery cut point indicates the lowest possible passing score. This procedure is not purely statistical, but incorporates the use of judgments by subject matter experts. CM

With Permission, As printed in June 2002 /"Contract Management" page 37

About the author: MELISSA D. FEIN, PH.D. is an affiliated assistant professor with the Maryland Assessment Research Center for Education Success at the University of Maryland. She received her Ph.D. in Applied Statistics from the University of Maryland.
Send comments on this article to cm@ncmahq.org.