Test Validation & Reliability

A well-designed training program must enable students to return to the job following their training and successfully apply the knowledge and skills addressed during the training program. Achieving this goal requires completion of a thorough job task analysis and the development of course objectives. Test items can then be written that measure whether or not the students have accomplished the course objectives.

PTG's Level 2 online system for test development and reliability verification (TDRS) uses Worksheet Wizards to perform job analyses, build and validate test instruments, and estimate reliability of test results. Tests developed using PTG's automated approach to job task analysis, objective determination, and analysis of test items, are assumed to have met the requirements for face and content validity. The system also makes available the appropriate coefficients to calculate and determine criterion-related validity, if appropriate for the planned use of the test.

TDRS has five inter-related components:

• A test development system (including job task analysis, objective verification and test item analysis)
• The capability to administer pilot tests online and to import test results for automated Item Analysis using Item Difficulty and Discrimination Indices, and Distractor Analysis
• A test repository
• A test results reliability verification system.
• A test administration and automated scoring feature

TDRS allows content developers and SMEs to perform job task analyses, write, validate and store tests, and conduct reliability estimates of the test, based on test results. Outside auditors trained in psychometric theory can review job task analyses and test elements by reviewing the worksheets associated with their development, to verify that acceptable procedures have been followed for test validity.

Tests developed and validated as described above, can be stored in a test repository for later administration. Once the test has been administered and scores are available, the Test Reliability Estimation System can be used to estimate the reliability of test results overall, based on actual test results.

If there is no measure of accomplishment, there is no way to document the worth of the training event. However, for worth to be measured properly a test must be valid, and it must also produce reliable test results. The lack of these characteristics diminishes the value of testing, per se, and can even have legal ramifications. The method selected for reliability determination is determined based on the structure (tests with a range of scores vs. binary tests to determine mastery / non-mastery), and use (recruitment, promotion, separation, course improvement, trainee progress development) of the test. Through is relationships with trained psychometric professionals in the evaluation field, PTG can offer consulting expertise in developing more effective and reliable testing programs.