Quantitative Measurement of the Progression of Clinical Expertise University of California at San Diego, CA

The Specific Aims of the original proposal were:

1. Examine the progression of residents' technical skills over the course of anesthesia residency by measuring actual clinical performance;
2. Examine how cognitive models of a complex clinical decision problem evolve throughout residency training; and
3. Define the critical differences in technical and cognitive performance among anesthesia residents, CRNA, and Board- Certified anesthesiologists.

In addition to addressing all of the original Project aims, this team of researchers have also successfully undertook several important related studies that help to delineate the nature of the anesthesia task and critical factors related to optimal clinical performance.

These kinds of data may permit more objective consideration of the non-monetary costs of technical anesthesia procedures. The potential clinical benefits of the use of more complex airway management techniques may be partially offset by the impact of increased workload on other clinical demands. This study has also refined a Workload Density technique that measures clinical workload continuously, with high-resolution in real-time, in a way that accounts for the contribution of multiple individual task components of the job of administering anesthesia.

Behavioral Task Analysis
Behavioral task analysis allows study of how clinicians alter their behavior under different real-world conditions. The researchers examined the intra- and inter-observer reliability of this methodology and how task patterns, workload, and vigilance during actual cases change over the course of anesthesia residency training.
The preliminary results support the hypothesis that as clinicians gain experience they can more efficiently accomplish critical clinical tasks and avoid distraction while maintaining low workload and "spare capacity" to respond to new task demands. The new data identifies the critical behavioral attributes of clinical performance that distinguish experts from novices. This research may elucidate how traditional measures of clinical knowledge relate to what clinicians actually do during patient care, thereby providing a critical methodological foundation for a novel approach to clinical evaluation and certification.
The first month of anesthesia training can be a time of great stress for new residents and their instructors. The study team undertook an additional study to document what anesthesiology educators and trainees actually do in the OR during the first three weeks of clinical training. They found that the new resident training initially emphasizes manual task performance, neglecting other tasks. The anesthesiologist instructor manages trainee workload and enhances learning of essential manual tasks by performing many cognitive tasks. Additional analysis will guide optimal clinical training in real cases and in simulations. The effects of sleep deprivation were studied on actual clinical performance in 15 pairs of anesthesia cases, each pair consisting of a case occurring during an on-call night shift and a daytime case matched as closely as possible based on type and duration of surgery and patient ASA status. The high correlation between video and real-time results suggests that there was little effect of observer fatigue on data obtained from night cases and that observer bias was minimal. This study provides strong evidence of task pattern, mood and workload differences in clinicians due to sleep deprivation, and emphasizes the need for optimization of anesthesia provider work environment.

Cognitive Task Analysis
The team used a cognitive task analysis technique based on the Critical Decision Method to elicit expert anesthesiologist's knowledge about the extubation decision. Concept Maps were constructed to depict the relevant knowledge (concepts) and relationships between associated concepts.
A major finding was that the post-anesthesia extubation decision ultimately depends almost exclusively on four factors: 1) the patient's current respiratory function (ventilation and oxygenation); 2) the provider's expected ability to mask ventilate the patient if extubation fails (i.e., an incorrect decision); 3) the provider's expected ability to reintubate if necessary; and 4) the impact on the patient or providers if extubation fails (e.g., greater risk of patient injury due to pre-existing disease). Psychosocial issues (e.g., surgeon preference, postoperative disposition, etc.) played a significant role in decision making while economic implications were less well appreciated. Researchers plan to continue to refine this important safety tool in the future. Once an expert model of the extubation decision is established, it will be possible to assess knowledge content among anesthesia providers, explore how residents acquire this knowledge, and investigate alternative teaching strategies.
As a result of NPSF funding, this team has received national and international recognition for their on-going patient safety research and education efforts. Investigators have been invited as visiting professors at the University of Pittsburgh, and at the Chang Gung Memorial Hospital (Taipei, Taiwan). In addition, they have also been invited to present at the European Society of Anaesthetists annual meeting (Nice, France) and at the inaugural Patient Safety Symposium of SAFER California Healthcare. This group plays a significant role in the San Diego Center for Patient Safety, a community research and educational resource that includes members from multiple San Diego hospitals, several local academic institutions, and community physicians.
In addition, this team of investigators was successful in obtaining the following federal patient safety grants:

• Department of Veterans Affairs, Health Services Research and Development Service. "Unexpected Clinical Events: Impact on patient safety." $738,976. 7/1/01-6/30/04.
• Agency for Healthcare Research and Quality (AHRQ). "Unexpected clinical events: Impact on patient safety." (Co-funded supplement to VA HSR&D grant). $240,475. 9/30/01-8/31/04.
• Agency for Healthcare Research and Quality (AHRQ). "DCERPS: Standardized encounters to study patient safety" $442,865. 9/15/01-8/31/04.

Publications to date as a result of NPSF funded study:

1. Weinger, M.B., Vredenburgh, A.G., Schumann, C.M., Macario A., Williams, K.J., Kalsher, M.J, Smith, B., Troung, P.C., Kim, A.: Quantitative description of the workload associated with airway management procedures. J Clin Anesth 12: 273-282, 2000.
2. Weinger, M.B., Slagle, J.M.: Task and workload analysis of the clinical performance of anesthesia residents with different levels of experience (abstract). Anesthesiology 95: A1145, 2001.
3. Weinger, M.B., Slagle, J.M., Kim, R, Hughes, S., Gonzales, D.C.: A task analysis of anesthesia residents' initial clinical training (abstract). Anesthesiology 95: A1146, 2001.
4. Weinger, M.B., Larson, J.W., Slagle, J.M., Gonzales, D.C.: Elicitation of expert knowledge about the post-anesthesia extubation decision (abstract). Anesthesiology 95: All 87, 2001.
5. Ou, J.C., Weinger, M.B., Vora, S., Mazzei, W.J., Slagle, J.M.: Further evaluation of the effects of nighttime work on mood, task patterns, and workload during anesthesia care (abstract). Anesthesiology 95: Al 196,2001.
6. Weinger, M.B., Slagle, J., Kim, R., Gonzales, D.: A task analysis of the first weeks of training of novice anesthesiologists (proceedings paper). Proc HFES 2001 Congress 45: 404-408, 2001.
7. Weinger, M.B., Slagle, J.: Human factors research in anesthesia patient safety: Techniques to elucidate factors affecting clinical task performance and decision making (proceedings paper). Proc Annual Sympos AMIA 756-760, 2001 (will appear in a special 2002 Patient Safety supplement of Journal of the American Medical Informatics Association).
8. Slagle, J., Weinger, M. B., Dinh, M-T.T., Wertheim, V.V., Williams, K.: Intra- and inter-rater reliability of a clinical task analysis methodology. Anesthesiology 96: (in Press), 2002.
9. Fraind, D.B., Slagle, J.M., Tubbesing, V., Hughes, S., Weinger, M.B.: Re-engineering intravenous drug and fluid administration processes in the operating room: Step One - task analysis of existing processes. Anesthesiology (in Press), 2002.

Auditory Warning Signals in Critical Care Settings
University of Maryland and the University of Illinois
Investigators: Yan Xiao, PhD; F. Jacob Seagull, PhD, Christopher D. Wickens, PhD, Colin Mackenzie, MB
A team of researchers from the University of Maryland and the University of Illinois examined the fundamental question of the value of audio alarms as information-providing technologies. While audio alarms may seem to have inherent value, their actual utility was not previously well understood and they are known to be problematic in clinical settings. The team used cognitive engineering techniques and examined how providers use the information in audio alarms and developed new design principles re-evaluating the role of audio alarms in acute care settings.

The goal of this project was two-fold: to demonstrate the value of cognitive engineering in understanding safety enhancing technology, and to understand the roles of auditory alarms in patient monitoring. The project had three specific aims: (1) to determine patterns of information usage during real and simulated anesthesia, (2) to develop a general framework which depicts the role of and need for (i) auditory warning signals for alerting and (ii) auditory "updating" signals in patient monitoring, and (3) to produce a set of tentative design principles for such auditory display of monitoring signals in critical care.

Results:

1. It was feasible to deploy ambulatory eye-trackers in clinical environment. The ambulatory eye-tracking device was successfully used during the high-workload period of anesthesia induction and airway management. Differences in monitoring frequency by the anesthesia care providers were quantified for different stages of anesthesia care.
2. Presence of auditory alarms correlated with the length of airway management. When there were more alarms as the result of experimental manipulation, it took longer (54 s vs. 34 s) to accomplish laryngoscopy and intubation. The results support the hypothesis of the negative impact of false alarms. Combined with previous studies on auditory alarms, the results suggest that current auditory alarms could adversely affect care provision. Design of alarming devices needs to consider the negative impact of auditory alarms.
3. Care providers looked at different things in real operating rooms compared to during simulation. Using an eye-tracker, we compared anesthesia care providers visual scanning patterns during induction of anesthesia in real elective surgery with the patterns in simulated airway management. In simulation, care providers spent less time looking at the patient, and checked the monitoring equipment more frequently. The simulated patient seemed to be less informative and thus clinicians information from monitoring devices. This suggested potential limitations in evaluating care providers' monitoring behavior in a simulated environment.
4. Care providers varied their monitoring behaviors during different phases of their tasks. We attempted to compare how frequently the anesthesiologist looked at the vital signs monitors during the induction of anesthesia. Five time periods were selected for comparison: preparation for induction, induction, pre-oxygenation, intubation, and one minute post-intubation. The results did show statistically different sampling rates in the five selected periods. There was a large increase of sampling rate immediately after intubation, whereas during intubation the sampling rate was zero.

Application to Patient Safety
Is the health care environment in hospitals and clinics friendly to the health care providers? Are devices designed with considerations of how they should improve patient safety? If answers to these two questions are negative, how should we make the health care environment more user friendly? This project examined these questions using auditory alarms as a case in point, using methodologies of cognitive engineering.
Designers have long adopted the practice of attaching alarms of various types to displays and devices. In many senses, such practice is well-intentioned: it aims at well-known limitations in human performance, such as attention spans and difficulties in tracking many parameters: Control room operators can be warned of leaking valves; anesthesiologists can be warned of disconnected circuits; pilots can be warned of low fuel levels; train operators can be warned of over-heated engines. On the flip side, however, in almost every work setting in which alarms are deployed, there are reports of inappropriate alarms.
During patient care an anesthesiologist gathers information from a variety of sources to ensure patients safety. Much of the information is through the eyes. Knowing where an anesthesiologist looks could thus provide a basis for estimating what information is needed and determining how best present the needed information to the anesthesiologist. Measuring eye-gazes while an anesthesiologist cares for a patient is a direct, objective way of collecting data of anesthesiologists visual scanning patterns. However, technical difficulties abound for deploying such methods. We tested a commercial ambulatory eye-tracking device during the induction of anesthesia for elective surgery. The collected eye-gaze data were used to analyze how frequently anesthesiologists looked at various patient monitors.
The study established the feasibility of using an ambulatory eye-tracking device in the operating rooms. Whereas the findings of the analysis were not unexpected, they provided an objective basis of quantifying information gathering patterns of the anesthesiologist. During induction of anesthesia, the anesthesiologist has a high workload. Effective means are needed to provide patient status information to the anesthesiologist. Currently the vital signs monitors are scattered around the patient, making it difficult to gather information. For example, during intubation during which the sampling rate was zero in this study, well-positioned patient monitors could make it possible for the anesthesiologist to detect problems early.
There are potentially many uses of eye-gaze data. For example, further analysis of the recorded data from the current study will establish the feasibility of discerning which parameter the anesthesiologist looks at and when. Ambulatory eye-tracking devices could also be used in comparing different designs of patient monitors, evaluating different monitoring strategies that may be used by the anesthesiologist, and best allocation of patient monitors. Interestingly, data from eye-tracking devices may be analyzed to guide the design of innovative auditory "displays": presenting patient status information through sound. Eye-gaze data could then demonstrate whether or not the anesthesiologist spends more time looking at the patient when auditory displays are used.
Patient safety can be improved substantially if cognitive engineering is used routinely in the design of critical safety devices. Groundbreaking advances can be made in information provision for patient monitoring and for safeguarding the patient.

Photo: Project team discussing scenario scripts for simulated airway management, in which alarms were manipulated to examine care providers information gathering patterns. From left to right: Darin Via, MD, Jacob Seagull, PhD, Yan Xiao, PhD, and Richard Kyle. The simulation took place in the patient simulation lab at Uniformed Services University of Health Sciences, Bethesda, Maryland.

For more information about the University of Maryland Human Factors Research Program laboratory, please visit: http://hfrp.umm.edu/alarms/

Publications and Presentations:
The research supported in part or in whole by the NPSF grant has produced a number of publications. One abstract presented at the 1998 annual meeting of American Society of Anesthesiologists (ASA) was selected as one of the 15 abstracts (out of more than 1400) to be featured as press releases. The abstract presented at a meeting sponsored by Society of Technology in Anesthesia was awarded as the best research abstract.

1. Seagull, F.J., Xiao, Y., Mackenzie, C.F., Jaberi, M., Dutton, R. Monitoring behavior: a pilot study using an ambulatory eye-tracker in surgical operating rooms. Proceedings of Human Factors and Ergonomics 43rd Annual Meeting, pp. 850-854, 1999.
2. Xiao, Y., & Seagull, F.J. An analysis of problems with auditory alarms: defining the roles of alarms in process monitoring tasks. Proceedings of Human Factors and Ergonomics 43rd Annual Meeting, pp. 256-260, 1999.
3. Seagull, F.J., Xiao, Y., Mackenzie, C.F., & Wickens, C.D. Auditory alarms: From alerting to informing. Proceedings of IEA 2000/HFES 2000 Congress, Vol 1, pp. 223-226, 2000.
4. Xiao, Y., & Seagull, F.J. Auditory Warning Signals: New Concepts and Approaches. Proceedings of IEA 2000/HFES 2000 Congress, Vol 1, p. 210, 2000.
5. Xiao, Y. Mackenzie, C.F., Seagull, F.J., & Jaberi, M. Managing the Monitors: An Analysis of Alarm Silencing Activities During An Anesthetic Procedure. Proceedings of IEA 2000/HFES 2000 Congress, Vol. 4, pp. 250-253, 2000.
6. Seagull, F.J., & Xiao, Y. Patients Monitoring Technology: Friend or Foe? Proceedings of IEA 2000/HFES 2000 Congress, Vol 4, p. 249, 2000.
7. Seagull, F.J., Xiao, Y., Dutton, R., Downey, D., & Jaberi, M. Diagnosis In Real and Simulated Environments: Do We Look At Different Things. Anesthesiology, 93(34): A-572, Sept. 2000.
8. Xiao, Y., Via, D., Kyle, R., Mackenzie, C.F., & Burton, P. Stress with Simulated Trauma Management Measured Salivary Amylase. Anesthesiology, 93(34): A-1226, Sept. 2000.
9. Xiao, Y., Seagull, F.J., Jaberi, M., & Downey, D. Experimental use of an ambulatory eye-tracking device in operating rooms (OR). Anesthesiology, 89(A3):A1137, 1999.
10. Darin K. Via, Richard Kyle, Yan Xiao, Jacob Seagull, Colin MacKenzie, Eye Tracking System Improves Evaluation of Performance during Simulated Anesthesia Events, 2001 Society for Technology in Anesthesia International Meeting on Medical Simulation.
11. Seagull, F.J., Wickens, C.D. and Loeb, R.G. (In press). Attention and workload in auditory, visual, and redundant patient-monitoring conditions. Proceedings of the Human Factors and Ergonomics Society annual meeting, 2001.
12. Seagull, F.J., Xiao, Y. (In press). Using eye-tracking video data to augment knowledge elicitation in cognitive task analysis. Proceedings of the Human Factors and Ergonomics Society annual meeting, 2001.

Theory and Methods for Minimizing Name Confusion Errors
University of Illinois at Chicago, IL
Investigators: Bruce L. Lambert, Ph.D., Michael Cohen, RPh, MS, Prahlad Gupta, PhD, Gordon Schiff, MD, Clement Yu, PhD
This project focused on what are called Look-Alike-Sound-Alike (LASA) drug names. One of the primary objectives was to create an improved computer algorithm to identify drug names that look and sound alike when new drug names are being assigned. Using psycholinguistic theory and practice, this project produced a more systematic approach in identifying LASA names, preventing new similar names from being introduced, and helping to prevent the errors associated with drugs that have similar names.

The project was designed to refine theory and develop methods for predicting and preventing drug name confusion errors, especially those involving visual perception of handwritten drug names. Specifically, the experiment was designed to examine the effect of prescribing frequency, neighborhood frequency and neighborhood density on the probability of a pharmacist making an error in a visual perceptual identification task. Participants viewed a series of noise-masked, handwritten drug names as they were briefly presented on a computer monitor. The task was to correctly identify the presented name by typing it into a provided text box. In addition, the project was to build a computerized drug name search-and-retrieval system based on what we learned from the psychological experiments.

Thirty-seven licensed, practicing pharmacists participated in this study. Participants were recruited from among the attendees at the 2000 annual meeting of the National Community Pharmacy Association in San Antonio, TX. Individuals were not paid for their participation.

One hundred sixty (160) three-syllable drug names were selected to fill the cells of a 2 x 2 x 2 design, where the factors were stimulus frequency (high/low), neighborhood frequency (high/low), and neighborhood density (high/low). Names and prescribing frequencies were obtained from the drug databases contained within the U. S. National Ambulatory Medical Care Survey (NAMCS) and the U. S. National Hospital Ambulatory Medical Care Survey (NHAMCS) for the years 1992-1996. In cell-to-cell comparisons, sets of names at the same level of a factor (e.g., all names at high levels of stimulus frequency) did not differ statistically from one another on their scores for that factor. Conversely, names from cells at different levels of an experimental factor did differ significantly from one another. All cell-to-cell comparisons were based on Tukeys Honestly Significant Difference. The names were handwritten by 5 practicing physicians from the University of Illinois Hospital and Clinics. The physicians included one fourth-year psychiatry resident, one senior attending physician specializing in occupational medicine, and three internal medicine residents.

Physicians were paid US$20 to write each of the 160 names on a multi-page list. Although they were told that the names were to be used in a study of drug name confusion, they were explicitly instructed to write in their normal manner-not to make any extra effort to make their writing neat or legible. The names were scanned on a Hewlett-Packard ScanJet ADF at 600 dpi and saved as TIF files. Scanned images were then imported into Adobe Photoshop, saved in JPEG format, and degraded with Gaussian noise as well as vertical and horizontal graining. Due to a computer programming error, 4 names appeared twice in the experiment (Mellaril, Pancrease, Restoril, and Zefazone) and 4 names never appeared (Nolvadex, Stelazine, Acular, and Brethancer). Data from the second appearance of the repeated names were deleted, and all subsequent analyses were based on 156 names.

Thirty-seven participants each responded to 156 names, producing 5772 total responses. The error rate was 45.7% (2637/5772). On average, each participant incorrectly identified 71 of 156 names (std. dev.=17.76, range, 44 to 131, median = 67). Detailed analysis of errors revealed that there were 202 (7.7%) omission errors and 2435 (92.3) substitution errors. Of the 2435 substitution errors, 1148 (47.1%) involved spelling errors or other non-drug names, and 1287 (52.9%) involved other drug names. Tests of the hypotheses revealed significant main effects of stimulus frequency, neighborhood frequency, and neighborhood density.

The team has also implemented a software application for drug name search and retrieval. The system can search for drug products based on similarity in spelling, pronunciation or non-name attributes of a drug (e.g., strength, dosage form, color). The evaluation of the recall and precision of the retrieval system is not yet complete.

Software designed and implemented by the Dr. Lambert has already been used by major pharmaceutical companies to screen proposed drug names prior to FDA approval. This service was offered by the University of Illinois at Chicago through the Med-Errs subsidiary of the non-profit Institute for Safe Medication Practices. Name searching through Med-Errs was offered for three years. A total of more than 150 distinct searches was performed during that period (where each search involves the comparative evaluation of up to 10 names). The same software system has also been used by the United States Adopted Names Council (USANC) to screen new non-proprietary (generic) names. The USANC used the system for one year, for a total of more than 150 searches. In both cases, the client submits a list of names. Each name is submitted as a query against a database of brand and generic names. Orthographic similarity or distance scores are computed for each name in the database, and the top n most similar/least distant names are displayed to the user (n = 50 names by default). The investigators of this study have now developed a prototype version of this system that searches for drugs based on multi-attribute product similarity, not just name similarity (i.e., the system also examines similarity in dosage form, strength, route of administration, etc.).

Photo: Lin, Lambert, Chang

Publications to date that resulted from this study:
Lambert, B. L., Chang, K.-Y., Lin, S.-J. (2001). Descriptive analysis of the drug name lexicon. Drug Information Journal, 35, 163-172.
Lambert, B. L. Chang, K.-Y., & Lin, S.-J. (2001). Effect of orthographic and phonological similarity on false recognition of drug names. Social Science & Medicine, 52, 1843-1857.
Lambert, B. L., Chang, K.-Y., Lin, S.-J., Gupta, P. (2000, October). Effect of prescribing frequency, neighborhood frequency, and neighborhood density on pharmacists visual perception of [typewritten] drug names. Paper presented at the Annual Meeting of the American Association of Pharmaceutical Scientists, Indianapolis, IN.

Looking for Trouble in All the Right Places: Electronic Decision Support for Error Reduction in a Large HMO
Kaiser Permanente, Oakland, CA and Cornell University
Investigators: Gabriel J. Escobar, MD, Mary Anne Armstrong, MA, Larry I. Palmer, JD, Linda J. Nozick, PhD, Andrea Kabcenell, MPH
The original hypothesis of this study was that it is possible to use commonly available hospital information systems to identify situations that have a high probability of being associated with human error. The objectives were to develop and validate a set of computer algorithms that would detect specific patterns of clinical, laboratory, and/or administrative data associated with extremely high risk events in obstetrics.

The investigators have found that electronic scanning techniques can be highly effective in detecting high risk events in perinatal care. The results have also confirmed that these techniques are superior to existing methods (e.g., voluntary incident reporting) used by hospital quality assurance departments.

However, the reactions encountered from the parent organizations of investigators were mixed. Some quality and risk managers were extremely enthusiastic about transplanting these techniques outside the research environment into the operational arena. Others were uncomfortable with that possibility because, given limited resources and competing priorities, it would be difficult to incorporate these techniques into existing operations.

While some senior leaders were enthusiastic about conducting research on electronic signatures for purely internal purposes (i.e., within the protected environment of quality management bodies), considerable discomfort was expressed with the notion of conducting research and publishing in the public domain. "Our initial reaction to this was one of dismay, but, upon reflection and consultation with NPSF Research Committee, we realized that fears about publishing in the public domain have a rational basis. We suspect that these fears and concerns are shared by other organizations and are acting as strong disincentives to conduct human error research throughout medicine."

The researchers anticipate that the use of electronic signatures will become more widespread in both patient safety research and in actual operational error reduction programs. The use of such signatures is more rapid, sensitive, and ultimately less expensive than current methods such as incident reports or manual paper chart review. Also, as more electronic systems come on line, the sophistication of electronic signatures will increase.

Publications:
Dissemination of the results of this study results has been severely hampered by legal implications. However, after numerous re-visions, consultations, and negotiations two manuscripts will appear in Quality and Safety in Health Care in the near future.

This team of investigators was successful in obtaining the following federal patient safety grants:
New Approaches to Improving Quality of Kaiser Permanente Obstetric Care
Internal Kaiser Permanente project being conducted under peer-review protection. Project based on this NPSF project to transfer the use of electronic signatures to the operational arena.
The Perinatal Patient Safety Project
Internal Kaiser Permanente project being conducted under peer-review protection. Project is providing human factors training to 4 labor and delivery services.
The CERTs Prescribing Safety Program
Federally funded project involving 10 HMOs. Project is pooling electronic scanning results of potential medication errors. Confidentiality strategy employed in this project was based on the experience acquired in our NPSF-funded project.
Golden Needles in the Hay: Use of 'Electronic Signatures' to Support Quality Improvement in Kaiser Permanente.
This internal Kaiser Permanente project will develop new electronic signatures in areas other than perinatal care.