Background
Performance-based assessments, including those that employ simulation, are subject to many sources of error. The purposes of this exercise are a) to get you to think about reliability (of assessment scores) from the standpoint of sources of measurement error, some that can be controlled and others that cannot, and b) to make you more aware of the importance of validity evidence in judging the adequacy of assessments.
In each situation indicate whether the event described contributes to random or systematic measurement error in an examinee’s assessment score. (Explain your answer and provide some suggestion, where applicable, to mitigate the potential sources of measurement error):
An observer rating communication skills during a simulation exercise tends to rate female physicians more highly than males
An examinee who is interviewing a standardized patient is distracted by noise in the hallway and forgets to ask pertinent history questions
Dr. Frank becomes so anxious in testing situations that he often skips procedural steps
An observer looking at a performance video skips over the first few minutes of a patient interview before providing communication ratings
A student in a physical examination class misinterprets the instructions and later performs poorly on an OSCE station
A hospital plans to institute a minimal competency testing program for hiring new nursing staff. A study is undertaken to assess the adequacy of 2 commercially available assessments to decide which is more appropriate for assessing performance on the nursing skills and competencies valued by this hospital. A panel of “experts” matched the items on each assessment to a list of required skills. Each assessment was found to consist of 50% of items that matched one or more of the skills. What additional data might be useful in deciding which assessment to use?
IMPORTANT*******
IN YOUR ANSWERS, YOU SHOULD USE CITATIONS TO JUSTIFY YOUR RESPONSES. THE SOURCES THAT YOU SHOULD USE FOR YOUR CITATIONS ARE LISTED BELOW.
CITATIONS MUST BE GIVEN IN APA FORMAT (AND INCLUDE A REFERENCE PAGE LISTING THE SOURCES).
THE QUESTIONS THAT YOU NEED TO EXPLICITLY ANSWER ARE ON THE LINK LABELED ASSIGNMENT #2.
