Due to the nature of the practice of anaesthesia, more objective measures of performance, such as the OSCE, have found limited application in anaesthesia prior to the advent of simulators. Because full-scale simulators can realistically and repeatedly replicate complex scenarios in the operating room environment, they have the potential to be an ideal tool to evaluate clinical competence in anaesthesia through performance-based assessments and evaluations.
In contrast to the abundant literature on the use of the high-fidelity simulator in teaching and training, research on its use for evaluation, particularly in high-stakes evaluation, is not as well developed. Part of the problem has been related to the difficulty in establishing a standardised curriculum or definitive “gold standards” of evaluations in anaesthesia. Murray suggests that in order to measure the efficacy of a simulation training programme, a simulation curriculum must be created that has well defined training objectives.
According to a framework proposed by Wass et al., the first step is clearly to define whether the test is for formative (educational) or summative (pass/fail) purposes. The second step is to plan the test against the learning objectives of the competency being tested (blue printing). The third step is to determine the test’s reliability and validity. Finally, the last step in designing a test of clinical competency is standard setting in which the mode of scoring and the minimum standard required to pass is determined. The rigour with which an evaluative test fulfils all these criteria depends on the purpose of the test. Summative evaluation in a high stakes context such as the Royal College certification will require much more rigour than formative evaluation used to provide information and feedback on progress in training.
Byrne and Greaves reviewed the literature pertaining to the use of simulators for performance-based assessment in anaesthesia between the years 1980-2000. Of 13 studies in this category, only four specifically addressed the reliability or validity of the assessments used. Since that review, a number of studies addressing psychometrics of performance using the simulator have emerged. Forrest et al. have published a study looking at the inter-rater reliability and validity of technical performance assessment of novice anaesthetists using a simulator. Devitt et al. published a study to further investigate the construct validity of the test developed in his previous study. Morgan et al. followed up on their pilot study of 2000 in order to further study the validity and reliability of the evaluation of medical students using the simulator. Schwid et al. carried out a multi-institutional study of 99 residents to evaluate the validity and reliability of simulator testing in four different scenarios using two checklist scoring systems. More recently, Murray et al. described their methodology for developing simulation-based assessments of residents using different scoring systems. Weller et al. described their experiences in assessing anaesthesia trainees using different cases, judges and self-assessment and found that it required 12-15 cases to reliably rank the trainees.
The development of performance-based evaluations in anaesthesia using the simulator has lagged behind its use as a teaching tool. However, knowledge and work in this area have increased over recent years. Based on the review of the current literature, the area of performance assessment in anaesthesia still needs further clarification, definition, focus and development before it can be used routinely in high-stakes examination.
AUK would like to thank Dr Anne Wong M.D., M.Ed., F.R.C.P.(C), for her contribution to this section based on her work; Wong AK. Full scale computer simulators in anesthesia training and evaluation. Can J Anesth 2004; 51 (5): 455-464
Bryne AJ, Greaves JD. Assessment instruments used during anesthetic simulation: review of published studies. Br J Anaesth 2001; 86: 445-50.
Devitt JH, Kurrek MM, Cohen M et al. Testing internal consistency and construct validity during evaluation of performance in a patient simulator. Anesth Analg 1998; 86: 1160-4.
Devitt JH, Kurrek MM, Cohen M, Cleave-Hogg D. The validity of performance assessments using simulation. Anesthesiology 2001; 95: 36-42.
Forrest FC, Taylor MA, Postlethwaite K, Aspinall R. Use of high-fidelity simulator to develop testing of the technical performance of novice anaesthetists. Br J Anaesth 2002; 88: 338-44.
Gaba DM, Howard SK, Flanagan B, Smith B, Fish K, Botney R. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998; 89: 8-18.
Morgan PJ, Cleave-Hogg D. Evaluation of medical students’ performance using the anaesthesia simulator. Med Educ 2000; 34: 42-5.
Morgan PJ, Cleave-Hogg D, Guest CB, Herold J. Validity and reliability of undergraduate performance assessments in an anesthesia simulator. Can J Anesth 2001; 48: 225-33.
Murray D. Clinical simulation: measuring the efficacy of training. Curr Opin Anaesthesiol 2005; 18: 645-8.
Murray DJ, Boulet JR, Kras JF, Woodhouse JA, Cox T, McAllister JD. Acute care skills in anesthesia practice: a simulation-based resident performance assessment. Anesthesiology 2004; 101 (5): 1084-95.
Schwid HA, Rooke GA, Carline J et al. Evaluation of anesthesia residents using mannequin-based simulation. Anesthesiology 2002; 97: 1434-44.
Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001; 357: 945-9.
Weller JM, Robinson BJ, Watterson LM et al. Psychometric characteristics of simulation-based assessment in anaesthesia and accuracy of self-assessed scores. Anaesthesia 2006; 60: 245-50.