Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically Based Use Cases for Validating Safety-Enhanced Usability and Guidelines for Standardization

Technical Evaluation, Testing, and Validation of the

Usability of Electronic Health Records:

Empirically Based Use Cases for Validating Safety-

Enhanced Usability and Guidelines for Standardization

Svetlana Z. Lowry

Mala Ramaiah

Sheryl Taylor

Emily S. Patterson

Sandra Spickard Prettyman

Debora Simmons

David Brick

Paul Latkany

Michael C. Gibbons

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.IR.7804-1

NISTIR 7804-1

Technical Evaluation, Testing, and Validation of the Usability

of Electronic Health Records:

Empirically Based Use Cases for Validating Safety-Enhanced

Usability and Guidelines for Standardization

Svetlana Z. Lowry

Mala Ramaiah

Information Access Division

Information Technology Laboratory

heryl Taylor

Software and Systems Division

Information Technology Laboratory

mily S. Patterson

The Ohio State University

Columbus, OH

andra Spickard Prettyman

Akron University

Akron, OH

ebora Simmons

St. Luke’s Health System

Houston, TX

avid Brick

NYU Langone Medical Center

New York, NY

aul Latkany

New York Eye and Ear Infirmary

New York, NY

ichael C. Gibbons

The Johns Hopkins University

Baltimore, MD

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.IR.7804-1

U.S. Department of Commerce

Penny Pritzker, Secretary

National Institute of Standards and Technology

Willie May, Acting Under Secretary of Commerce for Standards and Technology and Acting Director

NISTIR 7804-1

October 07, 2015

Acknowledgments

The authors gratefully acknowledge the intellectual contributions from the following:

• Mary Theofanos, M.S., Human Factors Scientist, National Institute of Standards and

Technology

• Brian Stanton, M.S., Cognitive Scientist, National Institute of Standards and Technology

• Kristen Greene, Ph.D., Cognitive Scientist, National Institute of Standards and Technology

• David R. Hunt, M.D., Medical Director, Patient Safety & Health IT Adoption at Office of the

National Coordinator, Office of the Chief Medical Officer

• Teresa Zayas-Caban, Ph.D., Special Assistant to the Deputy National Coordinator , Office of

the National Coordinator

• Ellen V Makar, M.S.N., R.N.-B.C., C.C.M., C.P.H.I.M.S., C.E.N.P., US Dept. of Health and

Human Services, Agency for Healthcare Research and Quality, Center for Evidence and

Practice Improvement

• Michael L. Hodgkins, M.D., M.P.H., Chief Medical Information Officer, American Medical

Association

• Tejal Gandhi, M.D., M.P.H., CPPS, President, National Patient Safety Foundation

• Limor Hochberg, M.S., Human Factors Specialist, Life & Health, UL LLC, Wiklund R&D

• Kelly Cochran, M.S., R.N., Policy Advisor, Health Information Technology, Department of

Health Policy, American Nurses Association

• Mary D. Patterson, M.D., M.Ed., Associate Vice Chair, Medical Education Research in

Simulation, Director of Children's Academy of Pediatric Educators, Children's National

Medical Center

• Janey Barnes, Ph.D., Human Factors Specialist, User-View, Inc.

• Susan Harkness Regli, Ph.D., Human Factors Scientist, University of Pennsylvania Health

System

• Ross Teague, Ph.D., Senior Manager of User Experience, Allscripts Healthcare Solutions,

Inc.

• Hardeep Singh, M.D., MPH, Chief, Health Policy, Quality and Informatics Program,

Houston Veterans Affairs Health Services Research Center for Innovations

• John Ritter, HL7 EHR Work Group co-chair; PHR Work Group co-facilitator; EHR System

Usability Work Group co-facilitator HL7 International Mentoring Committee co-chair ISO

TC/215 U.S. Technical Advisory Group member Western Pennsylvania HIMSS Board of

Director member

• Thomas Elmquist, M.D., Cardiologist, New York Cardiovascular Associates

• Todd C. Hughes, Ph.D., Chief Technology Officer, Next Century Corporation

• Theresa Cullen, M.D., Chief Medical Information Officer, Veterans Health Administration

(VHA)

• Jorge A. Ferrer M.D., M.B.A., Biomedical Informatician, VHA OIA Health Informatics,

Adjunct Assistant Professor, UTHealth School of Biomedical Informatics

• Jan Horsky, Ph.D., Senior Research Scientist, Brigham and Women’s Hospital, Division of

General Internal Medicine and Primary care; Harvard Medical School

iii

In addition, the authors appreciate the access to the data collected under NIST contract number

SB134110CN0107 (UL-Wiklund) and the many instructive contributions from a broad spectrum of

stakeholders on the draft document during an invited stakeholder meeting held at the National

Institute of Standards and Technology on September 28, 2015, as well as peer reviewers for their

constructive feedback on prior drafts of the document.

DISCLAIMER

Certain commercial entities, equipment, or material may be identified in this document in

order to describe a concept adequately. Such identification is not intended to imply

recommendation or endorsement by the National Institute of Standards and Technology, nor

is it intended to im

ply that these entities, materials, or equipment are necessarily the best

available for the purpose.

Table of Contents

Acknowledgments........................................................................................................................... ii

Table of Contents .......................................................................................................................... iv

Executive Summary ........................................................................................................................ v

1 Introduction: Enhancing Safety with Empirically Derived Guidelines for Standardization .. 1

2 Methods for Identifying Critical Use Risk Areas ....................................................................... 2

3 Major Findings from Data Analysis .......................................................................................... 9

4 Methods for Generating Guidelines and Validation Use Cases .............................................. 15

5 Use Cases for Validation Testing to Ensure Safety-Enhanced Design .... Error! Bookmark not

defined.

6 Empirically Based Human Factors Guidance for Safety-Enhanced Design of Health

Information Technology ......................................................................................................... 36

7 Conclusions ............................................................................................................................... 41

8 References ................................................................................................................................. 43

Executive Summary

This document provides the empirical rationale for critical patient safety-related usability guidelines

for standardization. As well as requirements for validation testing to ensure safety-enhanced design.

These standardization guidelines are targeted at eliminating ‘never events’ and associated patient

harm by proactively addressing and mitigating the root causes of use errors from EHR design and

implementation elements, as characterized in our framework on the relationship between usability

and patient safety (NISTIR 7804).

Requirements for validation testing are instantiated through realistic use cases that can be applied

during design and evaluation of Electronic Health Record (EHR) systems and for user performance

testing. The ultimate goal is to drive and empower effective and safe human performance in the use

of EHRs. The objective of this research is to enhance safety-related usability with empirically

derived guidance in order to improve the effectiveness and efficiency of EHRs by eliminating or

reducing the most critical and likely causes of patient harm from mistakes and errors in interaction.

This research drew upon five different methods of empirical human performance data collection,

using crosscutting analytic methods, with a diverse set of analysts from different disciplines,

backgrounds, and perspectives. Researchers applied this mixed method approach in order to capture

user expectations, knowledge, and outcomes regarding EHRs. Multiple forms of data were collected

from a variety of user types, allowing for a comprehensive view of EHRs. Two large, multi-hospital

healthcare systems in the U.S. served as sites for most of the data collection, including observations

and interviews. Data collection included: 1) an online survey; 2) site observations; 3) follow-up

interviews with users; 4) usability testing of five different EHRs; and 5) expert reviews of the same

EHRs. Research results demonstrate strong congruence among the data, methods, and the analysts.

Human factors guidelines for standardization, which were explicitly derived from the empirical

evidence obtained through field data collection, are provided to improve the safety-related usability

of EHRs in each of the following three critical use risk areas:

1) Consistently display information critical to patient identification in a reserved area to

avoid wrong patient errors,

2) Provide cues to reduce the risk of entering information and writing orders in the wrong

patient’s chart, and

3) Support efficient and easy identification of inaccurate, outdated, or inappropriate items in

lists of grouped information by having information presented clearly and in a well-organized

manner.

Two use cases, one for the inpatient setting and one for the outpatient setting, are provided for

validation with summative usability testing. Employing use cases oriented to empirically- derived

usability challenges is intended to validate that potential patient safety risks are proactively mitigated

in the tested EHRs. These use cases may also prove useful for organizations for adapting aspects of

the design during implementation and optimization processes to protect against inadvertently

introducing new risks to patients.

Ultimately, this research demonstrates that patient safety is negatively affected when critical safety

tasks are performed with the support of poorly designed EHRs. As a result, mistakes and errors

frequently occur; with end users becoming frustrated and unwilling to trust the systems they are

given and therefore are more likely to rely on potentially unsafe workarounds.

1 Introduction: Enhancing Safety with Empirically Derived

Guidelines for Standardization

This document outlines the empirical rationale for critical patient safety requirements instantiated in

use cases that can be used during the design, evaluation, and user performance testing of EHR

systems. Electronic Health Records (EHRs) offer great promise for improving healthcare processes

and outcomes, including increased patient safety. Emerging evidence suggests that the use of health

information technology (HIT) may help address significant challenges related to healthcare delivery

and patient outcomes.

For example, three reports suggest that the use of HIT may improve

healthcare outcomes

and reduce patient mortality.

In addition, the use of HIT is a key component

of a national strategy to improve healthcare quality and patient safety.

Given the estimate that one

in three patients will potentially be harmed during a hospitalization,

the role for EHRs to improve

patient safety may be significant.

On the other hand, studies found that patient mortality unexpectedly increased following the

introduction of an EHR in a pediatric hospital

and that an order-entry system contributed to a severe

medication overdose.

Therefore, empirically-derived guidelines for standardization are needed to

foster enhanced ability to protect patients from inadvertent harm from EHRs. As with any HIT, EHR

usability problems that can adversely impact patient safety can be assessed, understood and

managed. Data collected through multiple methods can increase the validity of findings regarding

the magnitude of risk to patients.

The ultimate objective of enhancing safety with empirically derived guidelines for standardization is

to improve the effectiveness and efficiency of EHRs, and thus reduce the opportunity for patient

harm from mistakes and errors. Failing to have effective and efficient EHRs could result in clinical

users ‘extensive workarounds that can directly and negatively impact patient safety.

2 Methods for Identifying Critical Use Risk Areas

Researchers applied a multiple research methods approach in order to triangulate user interactions,

experience and dispositions of common encounters with EHRs. Multiple forms of data were

collected from a variety of user types allowing for a comprehensive view of EHRs. Data include: 1)

an online survey; 2) site observations at two geographical locations; 3) interviews with users at these

two locations; 4) usability tests with five different EHRs; 5) and expert reviews of the same EHRs.

A team of researchers with expertise and extensive experience in research methods executed the

effort, aided by three additional researchers, including clinical experts and human factors experts.

Data Collection

Data for this study were collected via NIST Contract Number SB134110CN0107 (UL-Wiklund).

Data collection commenced in September 2010 and concluded in May 2015. Two large, multi-

hospital healthcare systems in the U.S. served as sites for much of the data collection, including the

observations and interviews. Two weeks of observations and 40 interviews occurred in a variety of

care settings and with different types of user groups who routinely used EHRs. There was extensive

effort of unobtrusive ‘in situ’ observation and follow-up interviews in these locations. A total of 86

EHR users were interviewed and observed, including 49 medical assistants and administrative staff

members. Verbatim transcriptions of interviews and researcher field notes provided accurate and

reliable data for the analysis. There were over 300 pages of interview transcripts and 336

observation notations.

In addition, five EHR developers volunteered their EHRs for usability testing. There were a total of

15 two-hour test sessions per EHR, resulting in a total of 63 participants, including 31 providers

(physicians, nurse practitioners, and physician assistants), 21 nurses, and 11 medical assistants.

Several two-person teams, each including a test administrator and a data analyst (i.e., data logger),

conducted the usability tests. All test sessions took place in usability test facilities that included a test

room and adjacent observation room. All test sessions were video recorded. Test data included use

errors, “close calls”, difficulties, anecdotal comments about EHR interactions and quantitative

ratings of the EHR according to a scoring system focused on safety-related usability which was

developed as part of NIST contract SB134110CN0107. This scoring system, using responses as a 5-

point Likert scale from 1 = strongly disagree to 5 = strongly agree, was comprised of these

statements:

• The EHR uses familiar terms, symbols, and units.

• The EHR makes important information stand out.

• The EHR’s workflows match my expectations.

• The EHR makes it easy to detect and correct mistakes.

• The EHR gives me a comprehensive view of a patient’s health.

• The EHR will prevent harmful mistakes.

• The EHR makes it easy to share information with other healthcare professionals.

• I can use the EHR effectively while working under pressure.

• The EHR is easy to learn to use.

• I am confident that I can use the EHR to perform tasks correctly.

At the same time, six usability, informatics, and human factors experts independently reviewed the

same five EHRs in order to identify interface strengths and weaknesses of each. Of 665 strengths

and weaknesses identified overall, 60 weaknesses were identified by a human factors expert to

potentially have patient safety implications. Examples include having the patient name displayed on

the lower right corner of the screen, unlabeled graph axes, values displayed without accompanying

units (e.g., 225 without lbs), a value of 11 years and 11 months plotted as 11 years on the X axis of a

graph, not allowing physician users to edit or cancel an order after submission, and requiring one tab

to be open when writing an order and a different tab to be open when submitting a medication order.

Finally, a single academic medical center participated in an online survey, with approximately 2,500

surveys sent to clinicians and administrators from 19 different departments/units. There were 559

responses, of which 339 responses were complete (13.5% response rate). Nurses and physician

providers were the largest categories of user types who responded. There were 21 questions on the

survey, including two open-ended questions: 1) What EHR characteristics make it particularly easy

to use? and 2) What EHR characteristics make it particularly difficult to use?

Data Analysis

Quantitative and quantitative data set/analysis are described in NIST GCR 15-996 “Technical Basis

for User Interface Design of Health IT”.

Consecutive qualitative analysis of the interview and

observation data began with multiple readings of the full data set by the lead researcher and one of

the research analysts. Each of these two researchers coded a subset of the qualitative data in order to

begin a conscious and collaborative process of peer debriefing related to coding agreement. The

goal was to insure that the researchers’ use of codes and their application to segments of text was

consistent; “the more coders (using the same codebook) agree on the coding of a text, the more we

can consider the codebook a reliable instrument”

(page 310). This process is recommended in

qualitative research in order to establish the “credibility” or “truth value” of the findings.

9,10

During

peer debriefing sessions, researchers operationalized all codes and reviewed the data linked to

various codes. This process occurred regularly during the multiple coding phases, allowing for

continual reflection on and refinement of the codes and their application to the data.

Both researchers continued to read through the interview and observation data, creating tentative

labels for chunks of data that summarized what was emerging from the data. This was the open

coding process

or first cycle coding

. During this coding cycle, approximately 60 different codes

were used, with 30 of those occurring consistently across the data. While many of the remaining 30

codes only occurred once or twice, several of these represented critical cases where the data seemed

important enough (for example, those related specifically to patient safety). These critical case

codes became part of the code list that was used in the next coding cycle.

Once saturation occurred in the open coding process, the place where no new properties or

dimensions emerged from the coding process, we moved on to axial coding. Axial, or second

cycle

, coding provided a means to identify the relationships and connections amongst the open

codes. This allowed for the development of conceptual categories that link the codes and associated

data. Richards and Morse

argue that coding “leads you from the data to the idea, and from the idea

Available at http://nvlpubs.nist.gov/nistpubs/gcr/2015/NIST.GCR.15-996.pdf. Accessed September 28, 2015.

to all the data pertaining to that idea” (page 137). During this phase of the analysis, two top-level

categories of codes: 1) issues related to input and/or handling of data and information and 2) issues

related to understanding and use of the EHRs were identified.

Finally, selective coding identified core variables and emergent themes that occurred across the data

set.

Each researcher developed and shared memos and models about ongoing interpretations of the

data as a means to identify conceptual categories and move toward the development of emergent

themes for critical use risk areas that allowed researchers to predict patterns “of what may be

observed and what may happen in similar present and future contexts.”

Here, three major critical

use risk areas emerged as our ultimate findings: Identification of information, Consistency of

information, and Integrity of information.

At the same time, the lead researcher and the data analyst coded the usability test data. The same

conceptual categories and emergent themes developed in this data as in the interview and

observation data. Similarly, the lead researcher reviewed the survey data for overlap between this

and other data sources. Again, similar ideas surfaced across the data sets. Finally, the human factors

researcher examined expert reviews to discern patterns across reviews related to the issues under

consideration, specifically the relationship between the EHRs and patient safety. The lead

researcher used the findings to map back to the other data forms. The conceptual categories and

emergent themes are represented across the data sets, demonstrating the strength of the findings.

Examples of Codes, Categories, and Critical Use Risk Areas

In qualitative research, a code “symbolically assigns a summative, salient, essence-capturing,

and/or evocative attribute for a portion of language-based or visual data.”

Coding is a cyclical act

that serves as an intermediary between data collection and data analysis. Saldaña

argues that

qualitative codes capture essential elements of the data, and when those elements form clusters they

can “facilitate the development of categories and thus analysis of their connections” (p. 8). Codes

are rooted in the data, but as coding and analysis continue, codes and categories become more

refined, and often more conceptual and abstract. A further shift is evidenced “when the major

categories are compared with each other and consolidated in various ways, you begin to transcend

the “reality” of your data and progress toward the thematic, conceptual, and theoretical” (p. 12).

Below are excerpts of particular pieces of interview data and the codes applied to them, followed by

how these led to broader categories of analysis and later to emergent themes that ran across data sets.

It should be noted that the excerpts below are exemplars to demonstrate how data were coded,

however many more data segments were coded with the same codes. Many codes resurfaced

multiple times in the same interview and most occurred across all interviews.

Table 1. Examples of individual data excerpts and codes

DATA CODE(S)

“Sometimes, I might issue a rapid strep test and then save the note as a draft so that I have to go back in at end of day to save as

final based on strep test results. I might need to edit [the] plan, and then I also give patient both plans of action. [The] system will

send me a message reminding me to save the drafts, but you have to be paying attention, and there are a lot of things to pay

attention to”.

DRAFT VS. FINAL;

RELIANCE ON

MEMORY

“Running through a list of med[ication]s can take quite a bit of time. When a patient gets admitted to the hospital and discharged,

the med[ication] list changes. To me, there is a right way to write a prescription, so that we can communicate it. When [a] patient

gets discharged, sometimes pertinent medic[ations] can get deleted, and the format changes. When they get discharged, the units

might be off, the units might say mg, but the mom tells me she gives mL. The mom won’t know what mg means. If [the] patient

come[s] from [the] hospital, it might take me a half hour to update it”.

MEDICATION LISTS

CHANGE;

MEDICATIONS

DELETED;

UNITS OF MEASURE

CHANGE

“I think you could send a message on the wrong patient. Let’s say that the mom called and the child has breathing difficulty and

you are accidentally in [the] wrong record. It’s not the information that gets into the wrong patient, it’s knowing what patient the

information belonged to. Sometimes, [the] wrong vaccine gets recorded. I notice something might be wrong, the age is wrong, I

can get someone to go in and delete [the] vaccine, but then we are missing the information on [the] proper patient’s record. We

may be able to identify later when that patient comes in, but if we can’t find it, the patient gets an extra vaccine”.

WRONG

PATIENT/CHART;

VACCINE ERRORS

“My patient list is incorrect. It is not a complete patient list and I don’t know why. Sometime over the past 6 to 8 months 2/3 of

my list got purged. I have to remember the patients’ names and search through [the current EHR] to find…It is very frustrating

because I have no idea why it happened. Suddenly, one day I had 3,200 patients on my list and the next day I had 700. It doesn’t

make any sense. Every day the system does not feel firm or complete”.

WRONG PATIENT

LISTS;

FRUSTRATION

“A patient going from outpatient to inpatient or vice versa. Often times the medications are not addressed appropriately at those

transition periods. Often what’s in the discharge summary is not what’s in the medication list in the patient summary and so often

times it’s confusing to figure out. The person discharging the person is not diligent in updating the medications and I don’t know

if this happens, but it seems to be that somehow the inpatient medications and discharge summary medications should all auto-

populate”.

WRONG/DISCREPANT

MEDICATION LIST;

NEED FOR AUTO-

POPULATION

“No checks and balances for diagnoses in patient summary. Say a patient is seen by one provider and provider goes in patient

summary, sees that they have [a] scar on [the] chest so must have had CAD [Coronary Artery Disease] and surgery. If I see the

patient and change the summary all around and correct it, say they never had a heart attack, say I update that, if that patient goes to

see someone else… there’s no accountability. Anyone can change anyone else’s patient summary. Patient could come back to me

after seeing 3 other people, no longer looks like what I spent an hour updating. I can go back and look at my patient summary and

try to paste it forward. But [there] are oftentimes disease-specific information that other services might delete, and they delete it,

but it’s actually the record. Can be frustrating and lead to errors”.

CHANGES MADE TO

NOTES;

ACCOUNTABILITY;

DELETED

INFORMATION;

FRUSTRATION;

ERRORS

Table 1. Examples of individual data excerpts and codes (Cont.)

DATA CODE(S)

“Patients can receive 2 doses of medication because of how read-only scripts are handled in the system, e.g., [for example] read-

only means that there is no physical order, the pharmacy can view and dispense the medication, but with a read-only order there is

no way to scan or manually document that the patient received the medication. Order could be filled and given twice. EHR user

has to complete 8-9 clicks to get to a more detailed view to see comments if the script has been given to the patient”.

WRONG

MEDICATION OR

DOSE;

DIFFICULTY OF USE

(8—9 CLICKS)

“I’ve seen this EHR [at this hospital] go from a reporting mechanism to a data entry mechanism, to the point where it becomes a

file cabinet that you shove paper into. I guess if you control it [data entry], you know where things are. But, if my assistant or my

colleagues begin to use my file cabinet, pretty soon I’m not going to know what’s in there. I might be able to find a paper I filed 6

months ago, but it’s a needle in a haystack. Indexing is not standardized. It’s difficult to figure out where to start. [This is] what

am I worried about”.

“FILE CABINET”;

OUT OF CONTROL;

“NEEDLE IN

HAYSTACK”;

NEED FOR

STANDARDIZATION

“The list in the EHR is populated by multiple individuals. It’s a mish mash. Stuffing information in drawer, I may own a section

of that, not [the] entire piece”.

“MISH MASH” OF

INFORMATION

USED BY DIFFERENT

INDIVIDUALS;

POSSBILITY FOR

CHANGES/ERRORS

“Let’s say the patient is on Coumadin and then an internist gives an antibiotic. If [I] get prompted to refill the Coumadin, [then] I

get prompted that [the] patient is on an antibiotic and it tells me to consider decreasing [the] dose. That’s a robust decision support

piece that only works with providers in this EHR system [at this hospital]. It only works if I’m here and I see that you prescribed

that. It [EHR] doesn’t reflect the antibiotic received at [the] walk-in clinic”.

WRONG

MEDICATION OR

DOSE;

MEDICATION

INTERACTIONS

“Omitted information. It would be through omission, or you chart on the wrong kid. That would be human error. Safety thing is

when patient was omitted and all information was left out. The log in/log out thing is a huge pain in the butt. If I’m working at the

desk and someone says “Hey,” I run. If User Y comes back and was working at that spot, he might sit down and chart on me [the

chart I was working on before leaving]”.

OMITTED

INFORMATION;

WRONG

PATIENT/CHART

As coding continued, codes seemed to cluster into two major conceptual categories: 1) input and

handling issues and 2) understanding and use issues. For example, “Draft vs. Final” seemed to

be an input issue, as did “Changes Made to Notes” and “File Cabinet.” These codes were

connected by their relationship to the ways in which data made its way (or not) into the system.

Likewise, data related to “Medications Change” or “Wrong Patient” clustered around the ways in

which participants understood and used the EHRs. Sometimes chunks of data related to both

categories, as in the case of “Wrong Patient/Chart.” For example, P43 above noted that someone

might be working on a chart, need to leave for another task, and then someone else would begin

to work on it, thinking it was a different chart. This was an input issue (in that data was input

into the wrong chart), but also a use issue in that it was often difficult to discern which chart one

was in at a given time. The second cycle of coding allowed analysts to revisit previously coded

data and cluster it around these two conceptual categories. During this process, analysts noted

that both categories produced issues related to Identification of Information, Consistency of

Information, and Integrity of Information. These critical use risk areas pulled together a variety

of related codes that were united conceptually. For example, Identification of Information

represented those instances in the data where a piece of information was unable to be identified,

was misidentified, or was difficult to identify (this could be a patient name, a medication, a chart

number, or whether something was a draft versus a final entry). Consistency of Information

related to where and how information was presented, and often referred to a lack of

standardization in where and how information was presented as in where and how the patient

name was presented or a medication was entered. Finally, Integrity of Information encompassed

those codes where the accuracy of information was difficult to discern, sometimes due to

information being changed, being deleted, or not being entered.

Once coding of observation and interview data was complete, researchers looked across data sets

for congruence of analysis. Many of the codes generated in the analysis of observation and

interview data also emerged in the usability tests and expert reviews. For example, location of

patient name, unlabeled graph axes, values displayed without accompanying units, not allowing

physician users to edit or cancel an order after submission, and requiring one tab to be open

when writing a medication order and a different tab to be open when submitting a medication

order. Survey data also corroborated the coding structure and the subsequent development of

conceptual categories and themes. Identification of information, Consistency of information, and

Integrity of information cut across the data as major areas of use risk.

Trustworthiness

According to Lincoln and Guba

, trustworthiness involves establishing credibility,

transferability, dependability, and confirmability. Triangulation is one way to insure a

comprehensive, well developed, and systematic analysis that provides for credibility in the

process and the findings.

19,20

Triangulation can take many forms, including triangulation of

methods (using multiple methods for both data collection and analysis), triangulation of sources

(from within the same method—for example having Medical Assistants, Nurses, and Providers

as participants in the interviews and usability tests), and triangulation of analysts (to provide for

differing analytic lenses and perspectives). Researchers in this project used all three types of

triangulation to provide a more robust process and insure a more holistic and comprehensive

picture of the findings. Mays and Pope

suggest additional mechanisms for enhancing the

trustworthiness of a study, including respondent validation, where study participants are asked to

respond to and assess the researchers’ interpretations. While traditional respondent validation

(or member checking as it is also known) was not utilized here, researchers held a debriefing

session with people from a range of roles and positions, including nurses, doctors, and human

factors experts in order to obtain feedback regarding the analysis, interpretation, and findings.

Peer debriefing and respondent validation provided additional credibility for the study.

The use of “thick description”

and the voices of participants allow for greater transferability, or

the ability of readers to determine for themselves whether or not the results generalize to their

own situations. The presentation of extensive quotes and the detailed description of the coding

and analysis process in this report allow for such transferability. Dependability refers to how

likely it is that the findings would be consistent if the study were done again with a similar set of

participants in a similar context. Dependability can be achieved through the use of an external

audit where researchers who were not involved in the project review the process and products of

the research. In this study, the debriefing session with a variety of different experts offered a

space for presentation and discussion of the research process and the findings. Confirmability

provides a mechanism to determine if the conclusions, recommendations, and/or interpretations

can be traced to their sources, insuring that the findings are a product of the study and not of

researcher bias. An audit trail of the data and its processing allows for such confirmability.

Mays and Pope

also argue that a clear presentation of the data collection and analysis methods

provide for greater validity of the study.

3 Major Findings from Data Analysis

This research examined the use of EHRs in different healthcare settings and with different types

of users in order to gain a comprehensive view of the EHR impact. The objective is to improve

their effectiveness and efficiency and thus reduce the likelihood of contributors to avoidable

patient harm. We found three overarching themes representing critical areas of use risk, based

upon a convergence of triangulated evidence from all of the data sources: identification of

information, consistency of information and integrity of information. Within each overarching

risk area are subcategories, characterized by prototypical statements and questions, and possible

consequences, all rooted in the data. These are detailed in Table 2 below. Across all three of the

risk areas and their related subcategories there arose several major issues related to patient

safety: 1) the occurrence of unintended actions; 2) the likelihood of use errors; and 3) the high

level of user frustration. The three areas and their related issues highlight a lack of effectiveness

and efficiency related to EHRs. Ultimately, both the objective (actual observation and analysis

of user performance) and subjective data demonstrate that these issues negatively affected patient

safety during safety-critical tasks and times. For example, one participant discussed transition

periods and how these often led to discrepancies in medication lists:

“So the patient going from outpatient to inpatient, or inpatient to outpatient. Often times

the medications are not addressed appropriately at those transition periods, and so often

times what's in the discharge summary is not what's in the medication list in the patient

summary...And so often times it's confusing to figure out…The patient can end up being

confused and taking either too much or too little, or the wrong medication combination.

I, as the provider, on the other hand am unclear on what the patient is actually taking,

and it leads to confusion when the patient calls and says, "I'm not responding to the

treatment in the hospital. Then I have to go back and try to figure out, okay, are they not

responding because they're not actually taking the right medicine, or something like that?

And it also leads to confusion because the-- I had one patient who called our office from

their hospital bed and said, "They're sending me home. I'm confused about my

medication and nobody will talk to me." So not only was the documentation fraught with

errors in instruction to the patient, they were just as confused as I was trying to figure out

what they were going home with.”

The data excerpt above demonstrates how the integrity of the information is compromised

leading to incorrect dosages or medication combinations. Effectiveness is diminished as the

likelihood for critical use errors increases and efficiency is affected since it will now take

additional time to identify accurate information. This relationship was seen many times across

the data set. Also related to integrity of information, a different participant noted how difficult it

was to create an accurate patient summary:

“Actually, I was in with a patient. Gosh, was it Monday? I was trying to update the

patient summary because that's essentially what I spend a lot of my time doing now with

all new patients is I'm building all the patient summaries. Let's see if I can get to this

patient summary. What ended up happening is, as I'm updating, I can hit update here,

and when I'm updating, I can't see labs, I can't see anything else. I'm asking them,

"When's your last tetanus? Have you had your diabetes screen checked?" And I can go

up here to Actions, which I've now learned, and I can open up any one of these things in

a different-- let's see if we can get the labs. I can open up labs, but what happened to me

last time was that I was trying to open these up and it overwrote whatever I was doing

here. It wouldn't let me save anything. It wouldn't let me get back to it to save it.”

Not being able to see lab results and other pertinent information, as well as not being able to save

information as needed also reduce effectiveness and efficiency and create a situation where

frustration and unintended actions are likely to occur.

In addition to the integrity of information, the consistency of information presentation

and where to find it was a common issue. Two other participants articulated examples that many

others corroborated related to this pattern:

“I’ve seen this EHR here at [hospital] go from a reporting mechanism to a data entry

mechanism, to the point where it becomes a file cabinet that you shove paper into. I guess

if you control it, you know where things are. But, if my assistant or my colleagues begin

to use my file cabinet, pretty soon I’m not going to know what’s in there. I might be able

to find a paper I filed 6 months ago, but it’s a needle in a haystack. Indexing is not

standardized. It’s difficult to figure out where to start…

Because if you read through it and you don't change it to make it up to date, and you

don't bring forward the information that you've learned - they've got a new allergy, or

they used a medicine that you gave last time and they have an adverse affect, or anything

that they've told you that happened since you last used that note. If you don't bring it in to

that note, and you don't record it, then not only have you created a safety concern for

that patient, you've created a liability for yourself and for the institution.”

Finally, the identification of information arose as a major issue, with many participants

noting how difficult it was to easily locate and identify information and how easy it was to be in

the wrong place. The data excerpt from a different participant below clearly illustrates the

potential consequences of this issue:

“An example today: When I went in today to see a patient, there were two notes saved,

one was a 3 year old draft that my nurse started, the other note was for a different

patient. That’s easily solved. I went in the room, noticed an incorrect note. I sent a

message to the nurses. Asked them to delete it from record. But, in the meantime, one of

my staff members went into room of 12 month old and couldn’t find the record. I think

you could send a message on the wrong patient. Let’s say that the mom called and the

child had breathing difficulty and you are accidentally in wrong record. It’s not the

information that gets into the wrong patient, it’s knowing what patient the information

belonged to. Sometimes, [the] wrong vaccine gets recorded. I notice something might be

wrong, the age is wrong, I can get someone to go in and delete the vaccine, but then we

are missing the information on the proper patient’s record. We may be able to identify

later when that patient comes in, but if we can’t find it, the patient gets an extra vaccine.”

It is important to note that the examples presented above serve as exemplars for a broad range of

data found across the different methods that support the use risk areas and subcategories. These

are by no means unique statements, but representative of the data as a whole.

Table 2. Summary of Analytic Findings on Major Safety-Related Risk Areas and Possible Consequences

Areas of Critical Use Risk

Subcategories

Possible Consequences

Identification of Information

Am I in the right place and doing the right

thing?

• For patient

• For patient lists

Incorrect patient list

Who are these patients?

• Wrong treatment

• Wrong billing

•

Wrong charting of information

• Missed, omitted, delayed care

• Care or billing activity conducted on the wrong patient

• Have to pull up every patient chart

• Wrong medication ordered

• For records

• For medication/order

Passing/sharing information

What happens in the handoff?

Multiple EHRs used

What happens when EHRs don’t coordinate?

• Data/information are not recorded in EHR

• Misrecording or recording in wrong patient chart

Fragmented information

Data are often fragmented and found in multiple

places

• Often no context for displayed information

Consistency of Information

Why are things not listed and displayed in

standardized ways?

Misidentified patient/chart Where am I?

Record number is incorrect/Patient name is

misidentified.

• Documentation/orders in wrong chart (often without

knowing it)

• For information

• For organization

• For format

• For different systems

• For draft vs. final versions

• For omissions and/or changes

Supplements used

Did I remember to transfer data to the EHR?

Paper, whiteboards frequently used (reliance on

memory)

Multiple EHRs used

Where do I find X on this EHR?

Functions and screens shift Where is my

information?

• Cannot find information when needed in the EHR

• Reliance on memory for transfer of information/data

Cannot find information

Where is my information? Am I in the right

place?

• On screen

• In file

• In EHR

• Functions and screens shift

• Information found in different places (including record

number, patient name, medications prescribed, etc.)

Standardization of where things are and what

they are

• Location and format of date

• Location of record number

• Location and format of name

• Format of amount (e.g., metric vs. US)

• Running list of current medications

• Notation in wrong record

• Incorrect diagnosis or prescription

• Incorrect medication or order OR double vaccine

Table 2 Summary of Analytic Findings on Major Safety-Related Risk Areas and Possible Consequences (Cont.)

Areas of Critical Use Risk

Subcategories

Possible Consequences

Integrity of Information

Why and how are things changed, deleted,

or omitted?

• Lack of control over changes in dates,

notes, units of measure

• Inability to know what information is

Cannot figure out EHRs

How do I do this?

• Navigation is difficult

• Adding/deleting data is difficult

• Scrolling through long notes is time-

consuming

• More likely to just use whiteboard/paper and not put

data in EHR, resulting in incomplete files/charts

valid, relevant, and up to date

Draft vs. final version

Is this a draft or final version? Often forget to

finish a final version

• Omissions of data/information; notation of incorrect

data/information

•

What happens if change in patient condition in interim?

Changes to note/chart

You changed what?!!

• Different user can change someone else’s

note/input

• Information lost

• Inaccurate and/or incomplete data and charts

Common references not there

Why are height and weight not here? (and other

common data like vital signs)

• Having to do things (input) multiple times or search

multiple places

Included in the two use cases are the following aspects, categorized by the areas of use risk that

emerged from the data analysis and related to supporting evidence from the data analysis:

1) Critical Use Risk Area 1: Identification of Information

A) Accidentally transposing a patient’s first and last name is easy to do when the

names are both commonly used first names as well as possible last names

(e.g., William John, John William). This increases the potential for making

clinical orders or documenting in the wrong patient’s chart due to the ease of

inverting the names when searching for the chart, flipping the open chart to

another person and mistakes can more easily occur when quickly scanning the

name to confirm that the right chart is open. In the use cases for eligible

providers and nurses, both William John and John William need to be

included in the patient list.

B) Getting confused as to which chart is actively open for ordering medications

and viewing information is challenging when there is an interruption in the

midst of providing care for one patient in order to a high priority activity like

look up the current medications for another patient in response to a phone call

from the Emergency Department nurse for a patient who is not able to provide

the information reliably.

C) Identifying an allergy to the common sugar substitute that is contained in the

hospital’s standard diabetic diet is challenging when relying upon clinical

decision support alerts that are not designed to cover this scenario and/or have

high false alarm rates.

D) Being interrupted while providing care to one patient can increase the risks of

performing clinical actions or documenting information in the wrong patient

chart.

2) Critical Use Risk Area 2: Consistency of Information

A) Having a primary care provider in the outpatient setting identify that the

results of a diagnostic (MRI imaging) test are not available because they were

not ordered as planned during a prior hospital stay can be challenging when

results from within an organization and from other organizations are not

displayed and sorted in a consistent fashion.

B) Managing and documenting an allergy to a preservative in intramuscular

vaccines, thimerosal, is challenging because it differs from how the typical

allergy to a medication is identified and alerted with clinical decision support

and visualization paradigms Clinical decision support recommends ordering a

vaccination for influenza despite the patient having an allergy to thimerosol,

which is a preservative in the vaccination in the traditional injection delivery

mechanism. A different form of the vaccine (that does not contain thimerosol)

needs to be ordered to avoid an allergic reaction.

C) Treating an infectious eye disease requires coordinating care with other

specialized personnel, including infectious disease specialists and eye care

specialists. It can be particularly challenging to coordinate care using “read-

only” documentation.

D) As soon as the nurse notices that the patient had a markedly pink irritated right

eye on morning rounds, this information is critical to share quickly with other

care providers in order to protect healthcare workers and other patients from

infection by initiating and maintaining contact isolation precaution

procedures. Sharing this information across transitions of care is similarly

important.

3) Critical Use Risk Area 3: Integrity of Information

A) Managing medications requires reviewing medication lists which have

discrepancies.

B) Reviewing the laboratory results of a blood test cannot be done during the

primary care provider outpatient clinic visit with the patient because the

results are not available for review.

C) An imaging test that was not completed in the acute care setting is not readily

discoverable during the follow-up visit with the primary care provider as it

was not completed in the primary care setting.

Supporting evidence from the interview data: “Patients can receive 2 doses of medication because of how read-only scripts are

handled in the system (e.g., read-only means that there is no physical order, the pharmacy can view and dispense the medication,

but with a read-only order there is no way to scan or manually document that the patient received the medication. Order could

be filled and given twice. EHR [Electronic Health Record] user has to complete 8-9 clicks to get to a more detailed view to see

comments if the script has been given to the patient.”

Supporting evidence from the interview data: “I guess part of the biggest thing that we noticed, or that I see, is the transition

periods. So the patient going from outpatient to inpatient, or inpatient to outpatient. Often times the medications are not

addressed appropriately at those transition periods, and so often times what's in the discharge summary is not what's in the

medication list in the patient summary... And so often times it's confusing to figure out. The person discharging the patient is not

diligent in updating the medications. And I don't know if this happens but it seems to me somehow the inpatient medications and

discharge summary medications should all populate the updated patient summary, but I don't know if that happens. And so if

there's a discrepancy between what's in the patient summary med-list versus what's in the discharge summary, what can happen

as a result of that? The patient can end up being confused and taking either too much or too little, or the wrong medication

combination.”

4 Methods for Generating Guidelines and Validation Use Cases

Group of the human factors experts collaboratively mapped the findings from the data analysis to

human factors terminology in order to relate the insights to what is known in the human factors

field about contributors and barriers to successful, high reliability, and high reliance performance

by experts in domains with high consequences for failure. This translation resulted in

transforming the major findings into human factors issues and deficiencies, which enabled a

more targeted search for existing solutions and standards on which to base the recommendations.

Subsequently, a team composed of a methodologist, clinical physicians, clinical nurses and

human factors experts provided guidance on the generation of the recommendations and use case

during a focused two-day working meeting. As a result of this meeting, the team generated

safety-related usability technical guidance that draws upon existing human factors knowledge

and is empirically grounded and prioritized on the basis of being repeatedly raised as important

across five data collection methodologies.

5 Use Cases for Validation Testing to Ensure Safety-Enhanced

Design

The research findings provide empirical evidence about critically important usability issues that

could potentially have negative impacts on patient safety for providing clinical care with the

support of EHRs in both the inpatient and outpatient care settings. These issues provide the

foundation for the creation of two challenging use cases intended for summative usability testing

validation.

Healthcare is delivered within a system encompassing both ambulatory and inpatient settings. As

such, designing support for clinical care in either or both of these settings is challenging even for

an ordinary evolution of the care of a typical patient over a select time period in his or her life

that requires continuity of care across multiple providers with dedicated roles.

In this section, we provide for evaluation purposes one use case for inpatient care and one use

case for outpatient care. These two use cases are contextualized within an overarching narrative

for a single patient with multiple care activities conducted by multiple types of care providers.

When recruiting study participants for the validation testing, there are two categories of care

providers, 1) eligible professionals (medical doctors, nurse practitioners, or physician assistants)

and 2) nurses (registered nurses, licensed practical nurses). By employing the same patient for

the validation testing, it is possible to reuse demographic information and test data.

Overview of both use cases

The two use cases are situated within an integrated series of care encounters, consisting of:

1. An acute care hospital admission through the emergency department of a 45-bed community

hospital for a number of concerning symptoms, including chest pain and a productive cough.

2. An outpatient visit with a primary care provider two years later, after a recent hospital

admission for a possible stroke

The use cases for evaluation cover selected aspects of the interactions dependent on the

electronic health record. Initially, an overall scenario of the patient visit is provided. Next, the

detailed steps involved by each user interacting with the EHR are enlisted. This is followed by a

short description of the critical usability issues in the tasks of each of the user roles.

Use Case 1: First Hospital Stay

We discuss the use case in several parts. The complex scenario is likewise broken into relevant

parts for better understanding and application. The scenario description is followed by the user

tasks based on roles and the workflow per the scenario. Hence, the user may appear repeatedly as

called by the events mentioned in the scenario. We associate the critical usability issues

encountered in the performance of the tasks under each role as a subcategory.

Scenario 1, Part I

Mr. William John is a 65-year-old Hispanic male who presents to the Community General

Hospital Emergency Department with complaints of severe chest pain, difficulty breathing and

productive cough. Mr. John has a known history of Type 2 diabetes mellitus and

hypercholesterolemia. His allergies include Thiomerisol and “pink sweetener”. He is triaged and

taken into the Emergency Department (ED). The ED nurse records the vital signs, which are

within the normal limits except for lower oxygen saturation of 90% and an increased heart rate

of 110 beats per minute, and informs the Emergency Physician. The Emergency Physician orders

oxygen therapy per nasal cannula immediately, 12 lead EKG, serial troponin levels, chest X-ray,

sputum gram stain now, CBC, continuous telemetry monitoring, finger stick blood glucose

following the hospitals s diabetic protocol and continuation on his routine medications of a long

acting insulin, regular insulin on a sliding scale, atorvastatin, vitamins and aspirin. He also starts

the patient on azithromycin. He orders an 1800 calorie diabetic diet. The X-ray is confirmatory

for pneumonia. The first troponin is negative. An initial 12 lead EKG reveals abnormal but non-

significant ST segment changes. There is no cardiologist available and the internist will be the

admitting physician.

Scenario 1, Part II

Mr. John is admitted to a Medical-Surgical (Med-Surg) unit under the care of the internist

(admitting physician) for observation and treatment. Vital signs including O

saturation had

returned to normal. The resident physician informs the attending physician about the condition

of the patient. The nurse collects the specimens and sends them to the lab in a labeled container.

The attending physician, during rounds, changes the O

therapy to “as needed” to maintain

oxygen saturation above 96%.

Scenario 1 Part III

Day 2: The nurse, when assessing the patient in the beginning of the shift, notices that the patient

had a markedly pink itchy irritated right eye with a tearing sensation. The patient reported that

he was on some eye drops in the distant past, but did not recall the name of the medication. The

doctor who was then providing eye care moved out of town, and he does not remember the

clinic’s number.

The nurse had a high suspicion of viral conjunctivitis and puts the patient in contact isolation.

She asked the physician if he wanted an Ophthalmology consult. Per the physician order, she

requested the consultant Ophthalmologist. The ophthalmologist, during the consult, was able to

elicit a history of being in contact with a family member that recently had “pink eyes”. The

patient was an occasional contact lens wearer but had none recently. The ophthalmologist

established that the visual acuity, intraocular pressure, and ophthalmic exam were normal except

for signs and symptoms consistent with viral conjunctivitis. The ophthalmologist ordered topical

tetracaine 1% to be applied by the ophthalmologist to both eyes (and to have the bottles available

in the patient’s medication tray for when she rounded on the patient). He ordered that the patient

be able to self-administer preservative free artificial tears in single-use vials prn (as needed).

Because of excessive discharge, she also ordered that an assay for adenovirus be available on the

patient’s floor. The ophthalmologist planned to return the following day or ask the patient to

come to follow up in the office in 24 hours should the patient be discharged.

Scenario 1, Part IV

Day 3: The troponin levels remained negative, and there was no change in the EKG. Mr. John

had decreased chest pain and cough and other vital signs are normal. The patient stabilizes in 48

hours with the lab report positive for growth of streptococcus pneumoniae and confirming

sensitivity to azithromycin. The blood cultures returned negative with no growth. He is

discharged home with a diagnosis of pneumonia and conjunctivitis to follow up as an outpatient

in seven days with his primary care physician. He is asked to see the ophthalmologist the next

day to follow up on his viral conjunctivitis since he left the hospital before seeing him. At

discharge, the nurse hands over to the patient the summary with the discharge instructions.

Figure 1. Sequence of events in first hospital stay included in validation testing

Use Case 2: Outpatient care after second hospitalization

Scenario 2, Part I

Mr. John is now a 67-year-old male who presents to the Community General Hospital

Emergency Department with complaints of headache, and a reported two short periods of

confusion. He has just recovered from influenza that kept him bedridden at home for 8 days with

vomiting and diarrhea. Mr. John is a known patient of Type 2 diabetes mellitus and

hypercholesterolemia. The patient is triaged and taken into the Emergency Department. The ER

nurse records the vital signs, which are within the normal limits. The Emergency physician

orders electrolytes, CBC, a chest x-ray, CT scan, EKG, echo cardiogram, carotid artery Doppler

and calls the internist who admits the patient to the ICU for evaluation of a possible stroke or

TIA and orders an MRI. The MRI in the hospital is out of order and the MRI is not done. The

echocardiogram results are scanned into the chart in the media section instead of cardiology

where they are usually located. All other tests are negative except the lab tests come back with a

low sodium and potassium and this is resolved within 24 hours by intravenous fluids. He

receives orders for an MRI outpatient, Coumadin therapy for anticoagulation and teaching by the

pharmacist.

Mr. John is sent home with a diagnosis of possible transient ischemic attack (TIA) and discharge

instructions to follow up as an outpatient in seven days with his primary care physician and in

three days with a neurologist. He receives five prescriptions to fill immediately.

Scenario 2, Part II

Upon coming into the office of his primary care physician on day two after discharge, Mr. John

reports he feels much better but still gets headaches. The primary care doctor looks for the tests

done at the hospital. He sees the MRI and echocardiogram were ordered but cannot find the

results. The physician gets several calls during the outpatient encounter from the emergency

department where another of his patients, Bill Bates, has arrived with chest pain so he stops and

takes the urgent call. After addressing the needs of Mr. Bates, lab tests are drawn and Mr. John is

sent home and will see the neurologist in the morning.

The urgent call is about Mr. Bill Bates. This interruption starts with the nurse in the primary care

physician’s office being called to the phone. The emergency department nurse says that Mr.

Bates has come to the ED with a severe nosebleed and headache. The emergency department

needs to know if Mr. Bates is on Coumadin. Mr. Bates cannot remember his medications and left

them at home.

Figure 2. Sequence of events from second hospital day which are reviewed during the outpatient

visit during validation testing

The findings from the data analysis have implications for the design of use cases for ensuring

safety-enhanced design in the context of these activities, in addition to others:

1. Ordering medications

2. Ordering laboratory tests

3. Ordering diagnostic imaging tests

4. Avoiding drug-drug, drug-allergy, and food-allergy interactions

5. Maintaining the problem list

6. Maintaining and reconciling the medication list

7. Maintaining the medication allergy list

8. Interpreting the recommendations from clinical decision support

9. Using the electronic medication administration record

10. Performing clinical information reconciliation and incorporation

11. Electronic prescribing of medications

When conducting summative usability testing evaluations, representative, appropriately licensed,

participants simulate the roles of user categories (e.g., eligible professionals and nurses). The

mapping of the listed activities above and the user category roles are:

1. Eligible professional (Medical Doctor, Nurse Practitioner, or Physician Assistant) role

a. Computerized provider order entry – medications

b. Computerized provider order entry – laboratory

c. Computerized provider order entry – diagnostic imaging

d. Clinical decision support

e. Drug-drug, drug-allergy interaction checks

f. Problem list

g. Medication allergy list

h. Medication list

i. Clinical information reconciliation and incorporation

2. Nurse role

a. Electronic medication administration record

b. Vital signs documentation

c. Clinical decision support

d. Drug-drug, drug-allergy interaction checks

e. Medication allergy list

f. Clinical information reconciliation and incorporation

g. Retrieve imaging results

Table 3: Usability Test Tasks for Safety-Enhanced Design (Representative Use Cases)

Use Case 1: First Hospital Stay

Scenario 1, Part I: Triage Nurse

Participant is now logged in as the Triage Nurse

Critical Task

Scenario Information

Usability Test Task

Patient chart has been

“initiated”

Nurse documents patient’s

demographics

Patient = Mr. William John

Age = 65 years

Ethnicity = Hispanic

Sex = Male

• Sign on to the System

• Open the patient record for Mr. William John

• Enter the provided demographic information

for this patient into the System

Nurse enters patient’s

complaints, current

illnesses/conditions (history),

and allergies

Patient complaints = severe

chest pain, difficulty

breathing, and productive

cough

Enter the provided clinical information for this

patient into his patient record

Current illnesses/conditions

(history) = Type 2 diabetes

mellitus,

Hypercholesterolemia

Allergies = thimerosal and

“pink sweetener”

Nurse documents triage

information with disposition

to Emergency Department

(ED)

Patient is triaged and taken to

the ED

Enter information into the patient’s record about

triaging and transporting him to the ED

Scenario 1, Part I: Emergency Department Nurse

Participant is now logged in as the ED Nurse

Critical Task

Scenario Information

Usability Test Task

ED nurse verifies

information documented

by the Triage nurse

Patient = Mr. William John

Age = 65 years

Ethnicity = Hispanic

Sex = Male

• Sign on to the System

• Open the record for Mr. William John

• Review the provided demographic and clinical

information that were entered into his record

by the Triage nurse

Patient complaints = severe

chest pain, difficulty

breathing, and productive

cough

Current illnesses/conditions

(history) = Type 2 diabetes

mellitus,

Hypercholesterolemia

Allergies = thimerosal and

“pink sweetener”

ED nurse records the

patient’s vital signs, and

notifies the ED physician

Temperature, Blood Pressure,

Respiratory Rate = within

normal limits for an adult

male

• Enter the provided clinical information for this

patient into his record

• Enter a note stating that the ED physician was

notified about the increased heart rate and low

Saturation

Heart rate = 110 BPM

Saturation = 90%

Scenario 1, Part I: Emergency Physician

Participant is now logged in as the ED Physician

Critical Task

Scenario Information

Usability Test Task

ED physician verifies

information documented

by the Triage and ED

nurses

Patient = Mr. William John

Age = 65 years

Ethnicity = Hispanic

Sex = Male

• Sign on to the System

• Open the record for Mr. William John

• Review the provided demographic and clinical

information that were entered into the record

for this patient by the Triage and ED nurses

Patient complaints = severe

chest pain, difficulty

breathing, and productive

cough

Current illnesses/conditions

(history) = Type 2 diabetes

mellitus,

hypercholesterolemia

Allergies = thimerosal and

“pink sweetener”

Temperature, Blood Pressure,

Respiratory Rate = within

normal limits for an adult

male

Heart rate = 110 BPM

Saturation = 90%

ED physician orders O

therapy, 12-lead EKG, lab

tests, a radiology test,

telemetry, medications,

and diabetic diet

Orders =

• O

therapy per nasal

cannula STAT

• 12-lead EKG

• Serial troponin levels

• Sputum for gram stain, now

• Complete Blood Count

(CBC)

• Finger-stick glucose

• Chest X-ray

• Continuous telemetry

monitoring

• Long-acting insulin

• Regular insulin on sliding-

scale

• Atorvastatin

Enter the provided orders for this patient into his

record

Critical Task

Scenario Information

Usability Test Task

• Vitamins

• Aspirin

• Azithromycin

• 1800-calorie diabetic diet

Scenario 1, Part I: Emergency Department Nurse

Participant is now logged in as the ED Nurse

Critical Task

Scenario Information

Usability Test Task

ED nurse documents

medication administration,

therapy, EKG

performed, lab specimens

obtained, glucose test

performed, and telemetry

monitoring

Treatments & tests

performed =

• Continuous O

therapy

per nasal cannula initiated

• Continuous telemetry

monitoring initiated

• 12-lead EKG completed

• Finger-stick glucose

completed

Lab specimens collected =

• Serial troponin level

• Sputum for gram stain

• Complete Blood Count

(CBC)

Medications administered =

• Regular insulin on

sliding-scale

• Atorvastatin

• Vitamins

• Aspirin

•

Azithromycin

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided information for medication

administration, O

therapy initiated, EKG

completed, finger-stick glucose completed (and

result), and specimens collected for this patient

into his record

Scenario 1, Part I: Emergency Physician

Participant is now logged in as the ED Physician

Critical Task

Scenario Information

Usability Test Task

ED physician reviews

results of diagnostic tests

Diagnostic Test Results =

• Chest X-ray confirms

pneumonia

• Troponin is negative

• EKG shows abnormal,

but non-significant, ST

segment changes

• Sign on to the System

• Open the record for Mr. William John

• Find and review the provided diagnostic test

results

ED physician orders the

patient to be admitted as an

inpatient

Admit patient to Medical-

Surgical unit today for

observation and treatment

Enter admission order into the patient’s record

Scenario 1, Part II: Inpatient Nurse

Participant is now logged in as the Inpatient Nurse

Critical Task

Scenario Information

Usability Test Task

Inpatient nurse receives

the patient on the

Medical-Surgical unit and

verifies information

documented by the Triage

nurse, ED nurse, and ED

physician, and reviews

diagnostic test results and

the order for inpatient

admit

Patient = Mr. William John

Age = 65 years

Ethnicity = Hispanic

Sex = Male

• Sign on to the System

• Open the record for Mr. William John

• Find and verify the documented demographic

information

• Find and verify the documented patient

complaints, current illnesses/conditions,

allergy, vital sign, and O

saturation

information

• Find and review the documented medication

administration information

• Find and review the information documented

for the treatments and tests performed

• Find and review the diagnostic test results

• Find and verify the inpatient admit orders

Patient complaints = severe

chest pain, difficulty

breathing, and productive

cough

Current illnesses/conditions

(history) = Type 2 diabetes

mellitus,

hypercholesterolemia

Allergies = thimerosal and

“pink sweetener”

Temperature, Blood Pressure,

Respiratory Rate = within

normal limits for an adult

male

Critical Task

Scenario Information

Usability Test Task

Heart rate = 110 BPM

Saturation = 90%

Treatments & tests performed

• Continuous O

therapy per

nasal cannula initiated

• Continuous telemetry

monitoring initiated

• 12-lead EKG completed

• Finger-stick glucose

completed

Medications administered =

• Regular insulin on sliding-

scale

• Atorvastatin

• Vitamins

• Aspirin

• Azithromycin

Diagnostic Test Results =

• Chest X-ray confirms

pneumonia

• Troponin is negative

• EKG shows abnormal, but

non-significant, ST

segment changes

Admission order = Admit

patient to Medical-Surgical

unit today for observation and

treatment

Inpatient nurse documents

vital signs information

and O

Saturation

information

Saturation = 98%

Enter the provided clinical information for this

patient into his record

Temperature, Heart Rate,

Blood Pressure, Respiratory

Rate information = within

normal limits for an adult

male

Scenario 1, Part II: Attending Physician

Participant is now logged in as the Attending Physician

Critical Task

Scenario Information

Usability Test Task

Attending physician makes

rounds and orders the O

Therapy changed to as

needed

Order = change O

therapy

per nasal cannula from

“continuous” to “prn, to

maintain O

Saturation >

96%”

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided O

therapy change order into

this patient’s record

Scenario 1, Part III: Inpatient Nurse

Participant is now logged in as the Inpatient Nurse

Critical Task

Scenario Information

Usability Test Task

On Day 2, the inpatient

nurse assesses the patient at

the beginning of the shift,

and observes that his right

eye is markedly pink, itchy,

irritated, and tearing; the

nurse documents these

findings

Suspecting viral

conjunctivitis, the nurse

puts the patient in contact

isolation and documents this

action

The nurse reports the

findings to the attending

physician and asks if she

wants an Ophthalmology

consult; based on the

physician’s verbal order, the

nurse orders the consult

Patient’s right eye is

markedly pink, itchy,

irritated, and tearing

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided information about the

patient’s eye into his record

• Enter the provided information about the patient

being placed on contact isolation into his record

• Enter into the patient’s record the provided

information about the condition of the patient’s

eye being reported to the physician

• Enter into the patient’s record the provided

information for verbal order on behalf of the

attending physician for an Ophthalmology

consult

Patient has been placed on

contact isolation

Condition of patient’s eye

reported to attending

physician

Critical Task

Scenario Information

Usability Test Task

Ophthalmology consult for

possible viral conjunctivitis

ASAP

Scenario 1, Part III: Ophthalmologist

Participant is now logged in as the Ophthalmologist

Critical Task

Scenario Information

Usability Test Task

The ophthalmologist

performs a consult and

documents her findings and

plan

• Patient has a history of

being in contact with a

family member who

recently had “pink eye”

• Patient was an occasional

contact lens wearer, but

has not been wearing

them recently

• Visual acuity, intraocular

pressure, and ophthalmic

exam are normal except

for signs and symptoms

consistent with viral

conjunctivitis

• Plan: will return

tomorrow or ask patient to

schedule a follow-up visit

in 24 hours if patient is

discharged before rounds

tomorrow

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided findings and Plan into this

patient’s record

The ophthalmologist orders

an eye medication

• Topical tetracaine 1%, to

be administered in both

eyes by ophthalmologist

(have bottles available in

patient’s medication tray

for physician’s rounds

Enter the provided order for the eye medication

into the patient’s record

The ophthalmologist orders

eye drops for the patient to

administer to himself

• Preservative-free artificial

tears in single-use vials

for patient self-

administration

Enter the provided order for the artificial tears

into the patient’s record

The ophthalmologist orders

an assay for adenovirus

• Keep an assay for

adenovirus available on

the patient’s hospital unit

Enter the provided order for the adenovirus assay

into the patient’s record

Scenario 1, Part IV: Attending Physician

Participant is now logged in as the Attending Physician

Critical Task

Scenario Information

Usability Test Task

Attending physician reviews

diagnostic test results, vital

signs, clinical observations

• Troponin levels

remained negative

• No change in the EKG

• Decreased chest pain and

cough

• Vital signs are normal

• Patient is stable

• Sputum culture and

susceptibility report is

positive for growth of

streptococcus

pneumoniae with

sensitivity to

azithromycin

• Blood cultures were

negative/no growth

• Sign on to the System

• Open the record for Mr. William John

• Find and review the provided clinical

information that is documented in the patient

record, including diagnostic test results, vital

signs, and other assessments

Attending physician

documents a discharge

summary

The troponin levels have

remained negative, and there

has been no change in the

EKG. Mr. John has

decreased chest pain and

cough, and other vital signs

are normal. The patient has

stabilized over the past 48

hours. The lab report for

sputum specimen is positive

for growth of streptococcus

pneumonia, with

confirmation of sensitivity to

azithromycin. The blood

cultures returned negative

with no growth. Patient to be

discharged home with a

diagnosis of pneumonia and

conjunctivitis.

Enter the provided discharge summary

information into the patient’s record

Attending physician orders

the patient to be discharged

from the hospital to home

with a final diagnosis of

pneumonia and

conjunctivitis, and with

instructions to schedule a

visit with his primary care

physician and a

Discharge patient to home

Final diagnosis pneumonia

and conjunctivitis

Patient to schedule a visit

with his primary care

physician in seven days

Enter the provided discharge order information

into the patient’s record

Critical Task

Scenario Information

Usability Test Task

recommendation to schedule

a visit with the

ophthalmologist the next

day for follow up on his

viral conjunctivitis

Recommended that patient

schedule a visit with the

ophthalmologist for

tomorrow for follow up on

the viral conjunctivitis

Use Case 2: Outpatient Care Testing (inpatient data is only to prepopulate hospital chart not

to test)

Scenario 2, Part I: Triage Nurse

Participant is now logged in as the Triage Nurse,

Critical Task

Scenario Information

Usability Test Task

Patient chart has been

“initiated”

Nurse documents patient’s

demographics

Patient = Mr. William John

Age = 67 years

Ethnicity = Hispanic

Sex = Male

• Sign on to the System

• Open the patient record for Mr. William John

• Enter/verify the provided demographic

information for this patient into the System

Nurse enters patient’s

complaints, current

illnesses/conditions (history),

and allergies

Patient complaints =

headache and reported two

short periods of confusion

Enter the provided clinical information for this

patient into his patient record

Current illnesses/conditions

(history) = Type 2 diabetes

mellitus,

hypercholesterolemia

Nurse documents triage

information with disposition

to Emergency Department

(ED)

Patient is triaged and taken to

the ED

Enter information into the patient’s record about

triaging and transporting him to the ED

Scenario 2, Part I: Emergency Department Nurse

Participant is now logged in as the ED Nurse

Critical Task

Scenario Information

Usability Test Task

ED nurse records the

patient’s vital signs

Temperature, Heart Rate,

Blood Pressure, Respiratory

Rate = within normal limits

for an adult male

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided clinical information for this

patient into his record

Scenario 2, Part I: Emergency Physician

Participant is now logged in as the ED Physician

Critical Task

Scenario Information

Usability Test Task

ED physician orders lab

tests, a radiology test,

EKG, and other diagnostics

tests

Orders =

• Electrolytes

• Complete Blood Count

(CBC)

• Chest X-ray

• CT Scan of head

• EKG

• ECHO cardiogram

• Carotid artery Doppler

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided orders for this patient into

his record

Scenario 2, Part I: Internist

Participant is now logged in as the Internist

Critical Task

Scenario Information

Usability Test Task

Internist orders admit to

ICU and additional

diagnostics testing

Orders =

• Admit to ICU for

evaluation of possible

stroke or TIA

• MRI

• Sign on to the System

• Open the record for Mr. William John

• Enter the provided orders for this patient into

his record

Internist reviews results of

diagnostic testing

• Electrolytes results = low

and K

• Complete Blood Count

(CBC) result = normal

• Chest X-ray = negative

• CT Scan of head =

negative

• EKG = negative

• ECHO cardiogram =

misfiled in media section

of patient record

• Carotid artery Doppler =

negative

Review the provided results in the patient record

Internist orders outpatient

diagnostic testing,

medication with teaching,

and discharge with follow-

Orders =

• MRI outpatient

• Coumadin with teaching

by pharmacist

• Discharge to home, final

diagnosis possible TIA

• Patient to schedule a visit

with his primary care

physician in seven days

Enter the provided discharge orders for this patient

into his record

Critical Task

Scenario Information

Usability Test Task

• Patient to schedule a visit

with neurologist in three

days

Scenario 2, Part II: Primary Care Physician

Participant is now logged in as the Primary Care Physician

Critical Task

Scenario Information

Usability Test Task

Primary care physician

looks for diagnostic results

from tests performed in the

hospital

• MRI and ECHO

cardiogram results not

available

• Electrolytes results = low

and K

• Complete Blood Count

(CBC) result = normal

• Chest X-ray = negative

• CT Scan of head =

negative

• EKG = negative

• ECHO cardiogram =

misfiled in media section

of patient record

• Carotid artery Doppler =

negative

• Sign on to the System

• Open the record for Mr. William John

• Search for MRI and ECHO cardiogram results

in the patient’s record

• Review the provided results that are available in

the patient’s record

Primary care physician is

interrupted by call from

Emergency department

regarding Mr. Bates who

has presented with a nose

bleed.

Physician moves to Mr.

Bates chart. Physician opens

Mr. Bates chart and reviews

his history and medication.

• Physician opens Mr. Bates chart for review

• Physician closes Mr. Bates chart and is able to

go back to Mr. William Johns chart.

6 Empirically Based Human Factors Guidance for Safety-

Enhanced Design of Health Information Technology

Human factors guidance for safe and usable design exists for a number of safety-critical

domains, but not currently for health information technology (HIT) design. For example,

the Human Factors Design Standard (HFDS) is a compilation of human factors principles

and guidance for the procurement, design, development, and testing of aviation systems,

facilities, and equipment.

Within this document, the principle of ‘simplicity’ is defined

as “Information should be presented simply and in a well-organized manner.” In order to

achieve this principle, the following guidance is provided, along with other

recommendations: "Information should be presented in consistent, predictable locations.”

Based upon our empirical findings, detailed in the previous section, as well as a

comprehensive review of existing standards under NIST Contract Number

SB134110CN0107, we provide a limited set of high-priority guidance below. These

guidelines for standardization are targeted uniquely at eliminating ‘never events’ and

associated patient harm by proactively addressing and mitigating the root causes of

critical use errors from EHR design and implementation elements, as distinguished in our

framework on the relationship between usability and patient safety, as described in Figure

3 of NISTIR 7804 Technical Evaluation, Testing and Validation of the Usability of

Electronic Health Records.

The empirically based human factors guidance for safety-enhanced design is:

1. Consistently display information critical to patient identification in a reserved area (specified

below) to avoid wrong patient errors

1.1 Patient identification information shall be displayed in the upper left hand corner of

all screens/windows in a consistent order; so that users can efficiently and accurately

find and verify patient identity

1.2 The information shall continue to be displayed in the same location regardless of

scrolling or other navigational mechanisms to move within the screen/window

1.3 The order shall be to first display the patient’s name with the last (family) name

capitalized, followed by a comma and then first (given) name, middle name, and

modifier, followed by date of birth using e.g., Nov 9, 1961 format and age and

gender, and then followed by MRN number.

1.4 For mobile devices or tablets with smaller screen sizes, it may be preferable to display

the information horizontally using the same ordering convention and white space

between the three elements. The information should be demarcated on the bottom

and/or the side, such as by employing white space, shading, or a line, from additional

optional identifiers

1.5 An example of this reserved area is:

SMITH, Walter Joseph III

Nov 9, 1961 (53 yo M)

MRN1348887

a. NAME: The last (family) name should be first and capitalized followed by a ‘,’ and

space prior to a capitalized first (given) name with the rest of the name in lower

case. The capitalization is used to distinguish the last name in cases of ambiguity

(e.g., Clark Kelly could be Clark KELLY or Kelly CLARK). It also reduces variation for

names with multiple capitalizations, such as McDonald.

b. NAME MODIFIER: In the absence of a modifier (e.g., Jr, Sr, III), nothing shall be

displayed in that location

c. DATE: The month represented as the first three letters of the month (or four in

languages other than English such as Italian where this is needed to

disambiguate months) shall be represented with a capitalized first letter with the

rest in lower case in order to make the capitalized last name more distinguishable

quickly on the display. The full year shall be displayed as four numeric digits

d. AGE: Displaying the age reduces the cognitive work required by the user to

convert date of birth into age. For years old, the display convention is ‘yo’ with a

space after the number, rounded down to the nearest digit. Similarly, months old

is displayed as ‘mo’, weeks old as ‘wo’, and days old as ‘do’. In Neonatal Intensive

Care Units, DOL 1 is often used for the first day of life, which corresponds to 0

days old. Similarly, DOL 2 is the second day of life. Decisions on when to display

yo, mo, wo, do, and DOL are expected to vary by institution. For example, a

hospital may display DOL for the first five days of life, followed by do until 30

days old, then wo until 24 weeks old, then mo until 24 months old, and finally yo

after 24 months of age. For the purposes of tracking accuracy of information, it

should be possible to display ‘on demand’ the value of the age in the original

format in which it was stored or transferred with interoperable systems. Age for

patients should not be displayed in values of less than 1 unit (e.g., 0.0001 yo)

e. GENDER: For gender, the display options should be M or Male for Male, F or

Female for Female, and Other. Additional details specifying subcategories under

Other, as necessary, shall be viewable on demand, such as transgender, or

reasons for a gender change

f. MRN: The allocation of digits to the Medical Record Number (MRN) should be

able to be modified in the future to accommodate future changes. Additional

identifiers such as care episode can be included on this line after the MRN. The

font size for MRN and other numeric identifiers can be smaller than the other

information displayed in the reserved area or placed to the right of the name and

date of birth information, but should still be viewable by older users.

MRN

information may be displayed in the reserved area only in response to an explicit

user action and/or when a barcoded wristband is scanned. Other identifiers, such

as encounter numbers, shall not be displayed in the reserved area in order to

reduce the likelihood of confusing the identifiers

g. ADDITIONAL IDENTIFIERS: Optional additional identifiers shall not be included in

the reserved area, as defined by being below a clearly demarcated horizontal line

or to the right of the area above the demarcation line. The display of optional

identifiers should not cover task-critical information except for short periods ‘on

demand.’ Additional optional identifiers include:

1) Place of birth

2) Picture. Note: Recommended to be a color picture of an individual patient

taken within the last 5 years, with no other individuals in the picture, and as a

close-up of the head facing the camera.

3) Biometrics

4) Genome

5) Barcode

6) Episode/encounter code

7) Suspected, confirmed or ruled out to have a highly infectious disease (e.g.,

‘Confirmed Ebola’), etc.

Kochurova, Olga, Joan K. Portello, and Mark Rosenfield. "Is the 3× reading rule appropriate

for computer users?." Displays 38 (2015): 38-43.

2. Provide visual cues to reduce risks of entering information and writing orders in the wrong

patient’s chart

2.1 Visually differentiate a chart

that enables a user to have unrestricted access to input

information (i.e., input mode) from a chart, which restricts the user’s ability to input

information (i.e., view-only mode)

2.2 Enable user to enter information on only one patient’s chart at one time

2.3 Enable user to have a chart in view-only mode in parallel with a chart with unrestricted

access to input information in order to support specialty-specific care needs (e.g.,

coordinated mother and child care following a birth, coordinated care of multiple birth

patients)

2.4 Enable user to easily transition from the current chart with unrestricted access to input

information to another chart by a deliberate action (i.e., identification/activation of the

patient chart), by the user.

2.4.1 Categories of charts that are likely to be needed by clinical providers are

1) charts for patients that are scheduled to be seen in the near future (e.g.,

24 hours), 2) charts for patients that have recently had information input

into them, 3) charts that have ordered laboratory tests or imaging tests

that are pending results, and 4) charts that have planned actions such as

documenting progress notes which have not yet been completed.

2.4.2 Easily transition implies that context has to be preserved in a way that is

clear to the user when the user transitions to another chart and back to a

previous chart; and that it is easy to find and identify a desired patient's

chart for any relevant patient in the system. Context should be preserved

in these transitions such that unsaved work in progress text should be

preserved by the system until saved (or deleted) by the user

2.5 Visually distinguish the mechanism for moving within a single patient’s chart and

transitioning from one chart with unrestricted access to input information to another

In the 2002 article "Maintaining a Legally Sound Health Record." Journal of AHIMA 73(2), a

chart is defined as "generated at or for a healthcare organization as its business record and is the

record that would be released upon request. It does not affect the discoverability of other

information held by the organization. The custodian of the legal health record is the health

information manager in collaboration with information technology personnel. HIM professionals

oversee the operational functions related to collecting, protecting, and archiving the legal health

record, while information technology staff manage the technical infrastructure of the electronic

health record."

3. Support efficient and easy identification of inaccurate, outdated, or inappropriate items in lists

of grouped information by having information presented simply and in a well-organized

manner.

Ways to achieve this include the following:

3.1 Lists of patients assigned to a particular clinician user should be presented in consistent,

predictable locations within and across displays and print-outs and the content should

not vary based on display location.

3.2 The status of a note and order as draft as compared to final shall be clearly indicated on

appropriate displays.

3.3 Clearly indicate the method by which the system saves information, whether auto-save or

requiring deliberate action to save, or combinations thereof.

3.4 Inputted information should be automatically saved when a user transitions from one

chart to another.

3.5 The language used should be task-oriented and familiar to users, including being

consistent with expectations based upon clinical training.

3.6 Enable a user to easily order medications that have a high likelihood of being the

appropriate medication, dose, and route. The likelihood is increased when displays are

tailored to specialty-specific user requirements, comply with national evidence-based

recommendations, are in accordance with system, organizational, unit, or individual

provider preferences specified in advance, or are similar to orders made by the same

physician on similar patients, on the same patient in the past, or providers with similar

characteristics.

3.7 Support assessing relationships of displayed information and allowing users with

appropriate permissions to modify locations and relationships for inaccurately placed

information, including laboratory results, imaging results, pathology results, consult

notes, and progress notes. This includes information within a single patient’s chart as well

as information placed in the wrong patient’s chart. The information about the time and

person that made the change should be viewable on demand.

7 Conclusions

This report examines the use of EHRs and their potential to improve and hinder healthcare

quality and patient safety. We drew upon five different methods of data collection, utilized

cross-cutting analytic methods, and had analysts from different disciplines and perspectives. The

results demonstrate strong congruence among the data, across the methods, and amongst the

analysts. Three major critical use risk areas surfaced during analysis: 1) Identification of

information; 2) Consistency of information; and 3) Integrity of information. These three areas

are consistent with findings from other studies that examined the use of EHRs.

Ultimately, the data from this study demonstrate that during safety-critical tasks and times,

patient safety is negatively affected, in part because mistakes and critical use errors occur more

frequently and because users are highly frustrated, and thus more likely to employ workarounds,

such as relying upon supplemental artifacts, e.g., paper ‘shadow charts’ or whiteboards. Figure 3

below highlights the ways in which problems with identification, consistency, and integrity of

information can result in suboptimal and unsafe patient care.

Figure 3. Relationship between usability findings and safe and effective clinical care

In accordance with the empirical evidence, and in order to make care more optimal and safe, we

provided human factors guidance for improving the usability of EHRs. There are three areas for

this guidance:

1) Consistently displaying information critical to patient identification in a reserved area to

avoid wrong patient errors,

2) Providing cues to reduce risks of entering information and writing orders in the wrong

patient’s chart, and

3) Supporting efficient and easy identification of inaccurate, outdated, or inappropriate

items in lists of grouped information by having information presented simply and in a

well-organized manner.

Finally, we provided two use cases for use during summative usability testing to achieve the goal

of safety-enhanced design by validating that potential patient safety risks are proactively

addressed and/or mitigated, one for the inpatient setting and one for the outpatient setting. These

use cases may also prove useful for organizations that tailor aspects of the design during

implementation and optimization processes to protect against inadvertently introducing new risks

to patients.

8 References

Buntin, M., Burke, M., Hoaglin, M., & Blumenthal, D. (2011). The benefits of health

information technology: a review of the recent literature shows predominantly positive results.

Health Affairs, 30 (3), 464-471.

Gibbons, M.C., et al. (2009). Impact of consumer health informatics applications. Evidence

Report/Technology Assessment, Oct (188), 1-546; and Cebul, R., Love, T., Jain, A.K., & Hebert,

C.J. (2011). Electronic health records and quality of diabetes care. New England Journal of

Medicine, 365, 825-833.

Longhurst, C., Parast, L., Sandborg, C., et al. (2010). Decrease in hospital-wide mortality rate

after implementation of a commercially sold computerized physician order entry system.

Pediatrics, 126, 14-21.

Institute of Medicine. (2001). Crossing the quality chasm: a new health system for the twenty-

first century. Washington, DC: National Academy Press.

Classen, D.C., et al. (2011). Global trigger tool shows that adverse events in hospitals may be

ten times greater than previously measured. Health Affairs, 3(4), 581-589.

Han, Y.Y., Carcillo, J., Venkataraman, S., et al. (2005). Unexpected increased mortality after

implementation of a commercially sold computerized physician order entry system. Pediatrics,

116(6), 1506 –1512. Correction in Pediatrics, 117(2), 594.

Horsky J, Kuperman GJ, Patel VL. Comprehensive analysis of a medication dosing error

related to CPOE. J Am Med Inform Assoc 2005; 12(4):377-82.

Hruschka, D. J., Schwartz, D., Cobb St. John, D., Picone-Decaro, E., Jenkins, R. A., Carey, J.

W. (2004). Reliability in coding open-ended data: Lessons learned from HIV behavioral

research. Field Methods, 16, 307-331.

Erlandson, D. A., Harris, E. L., Skipper, B. L., & Allen, S. D. (1993). Doing naturalistic

inquiry. Thousand Oaks, CA: Sage.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage.

Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative

analysis. Thousand Oaks, CA: Sage.

Saldaña, J. (2013). The coding manual for qualitative researchers. Thousand Oaks, CA:

Sage.

Saldaña, J. (2013). The coding manual for qualitative researchers. Thousand Oaks, CA:

Sage.

Richards, L., & Morse, J. M. (2007). User’s guide for qualitative methods (2

ed). Thousand

Oaks, CA: Sage.

Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative

analysis. Thousand Oaks, CA: Sage.

Saldaña, J. (2013). The coding manual for qualitative researchers. Thousand Oaks, CA:

Sage.

Saldaña, J. (2013). The coding manual for qualitative researchers. Thousand Oaks, CA:

Sage.

Saldaña, J. (2013). The coding manual for qualitative researchers. Thousand Oaks, CA:

Sage.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage.

Patton, MQ. (1999). Enhancing the quality and credibility of qualitative analysis. HSR: Health

Services Research, 3, 1189-1208.

Mays, N. & Pope, C. (2000). Qualtative research in health care: Assessing quality in

qualitative research. BMJ: British Medical Journal, 320, 50-52.

Ahlstrom, V., & Longo, K. (2003). Human Factors Design Standard (HF-STD-001). Atlantic

City International Airport, NJ: Federal Aviation Administration William J. Hughes Technical

Center.

Svetlana, L.Z., Matthew, Q.T., Ramaiah, M., Schumacher, R.M., Patterson, E.S., North, R.,

Zhang, J., Gibbons, M.C., Abbott, P. “NISTIR 7804 Technical Evaluation, Testing and

Validation of the Usability of Electronic Health Records.” NIST Interagency/Internal Report

(NISTIR) 7804, 2012. Available at: http://www.nist.gov/customcf/get_pdf.cfm?pub_id=909701.