IMAT5209 - Human Factors: Bedside Nursing CDS System Evaluation

This report details a usability evaluation of a clinical decision support system (CDSS) designed for bedside nurses to manage critical symptom changes in hospitalized patients, aiming to reduce preventable failure-to-rescue cases. The evaluation employs the UFuRT framework (user, function, representation, and task analysis) to assess the system's user interface and workflow. The methodology includes heuristic evaluations by domain experts and end-user performance evaluations with nurses using simulated use cases and the NASA Task Load Index. The report highlights the importance of usability in CDSS implementation, focusing on early symptom recognition and response, the role of frontline nurses, and cognitive design principles for the user interface. The goal is to create a system tailored to nurses' cognitive levels and adaptable to the fast-paced rescue workflow, ultimately improving patient safety and outcomes. Desklib provides access to this and other solved assignments for students.

Running head: HUMAN FACTORS IN SYSTEM DESIGN
Human Factors in System Design: Evaluation of Bedside Nursing CDS
System
Name of Student-
Name of University-
Author’s Note-

1Human Factors in System Design
Part ONE: The interactive system and its users
Clinical decision support systems (CDSS) are important tools to improve health care
outcomes and reduce preventable medical adverse events. However, the effectiveness and
success of CDSS depend on their implementation context and usability in complex health care
settings. As a result, usability design and validation, especially in real world clinical settings, are
crucial aspects of successful CDSS implementations (Yuan et al. 2013). A novel CDSS to help
the frontline nurses for managing the critical symptom change in hospitalized patients is
developed in this interactive system, hence reducing preventable failure to rescue cases. A robust
user interface and implementation strategy that fit into existing workflows was key for the
success of the CDSS.
Guided by a formal usability evaluation framework, UFuRT (user, function,
representation, and task analysis), a high-level specification of the product that captures key
usability requirements is developed and is flexible to be implemented. The users of the proposed
CDSS to identify requirements, listed functions, and operations the system must perform. A
visual and workflow representations was designed for the product to perform the operations. The
user interface and workflow design were evaluated via heuristic and end user performance
evaluation. The heuristic evaluation was done after the first prototype, and its results were
incorporated into the product before the end user evaluation was conducted. First, 4 evaluators
with strong domain expertise was recruited to study the initial prototype. Heuristic violations
were coded and rated for severity. Second, after development of the system, a panel of nurses
was assembled, consisting of 3 licensed vocational nurses and 7 registered nurses, to evaluate the
user interface and workflow via simulated use cases. Each session was recorded successfully
completed and its completion time. Each nurse was asked to use the National Aeronautics and

2Human Factors in System Design
Space Administration (NASA) Task Load Index to self-evaluate the amount of cognitive and
physical burden associated with using the device.
Users of the proposed CDSS were members of the clinician team responsible for rescuing
patients in the hospital. They included floor nurses, RRT nurses, and physicians. The user roles
described in this section were based on interviews with hospital clinicians.
Part TWO: The use cases
Figure 1: Use case for the System

3Human Factors in System Design
(Source: Created by Author)
Part THREE: The usability requirements
Clinical decision support systems (CDSS) are important tools to improve health care
outcomes and reduce preventable medical adverse events. In the US, CDSS is one of the key
requirements for the government mandated meaningful use of electronic medical record (EMR)
adoption. It was suggested that smart, portable, point-of-care, and interoperable technology
solutions could help reduce inefficiencies and improve patient safety and outcomes for nurses.
The effectiveness and success of CDSS depend on their implementation context and
usability in complex health care settings. Studies have shown that different CDSS
implementations often yield very different clinical outcomes-. A study found that a home grown
CDSS designed specifically for a hospital out-performed 31 other similar CDSS deployments
included in the study. A multi-site study indicated that nurses routinely over-ride CDSS
recommendations that do not fit their local practice, leading to a potential increase of errors.
In particular, CDSS implementations often suffer from poor usability, which directly
impacts their adoption and effectiveness. For instance, user interface (UI) workarounds have
been shown to greatly diminish the effectiveness of widely used CDSSs. While many CDSSs
rely on alert/reminder-based user interactions to prompt the clinician correct potential guideline
violations, alert fatigue was a common issue for those systems. A study showed that physicians
who receive CDSS alerts were only slightly more likely to take appropriate actions than those
who do not. In the area of diagnostic decision support, it has been demonstrated that the accuracy
of diagnostic aid tools depends on their UI. Tools that require simple copying and pasting from
free text medical records yield more accurate results than tools that require the physician to

4Human Factors in System Design
extract and categorize information from the medical records. As a result, usability design and
validation, especially in real world clinical settings, are crucial aspects of successful CDSS
implementation.
In this study, a novel CDSS for the CHRISTUS St. Michael health system (a 350 bed
acute care hospital) is developed to help frontline nurses to manage critical symptom changes in
hospitalized patients in a better way. The CDSS is currently undergoing clinical pilots inside the
hospital. The goal of the CDSS was to reduce preventable failure to rescue (FTR) cases in the
hospital. Since the nursing work environment is subject to constant interruptions and is error
prone, a robust UI and implementation strategy that fit into the existing workflow was crucial to
the success of the system.
Part FOUR: The evaluation methodology
Early Symptom Recognition and Response:
The FTR is a leading patient safety indicator with the highest incident rates among all
indicators according to a recent large-scale study. In 2010, FTR measure was included as one of
the Inpatient Prospective Payment System measures by the Center for Medicare and Medicaid
Services, which directly affects hospitals’ reimbursements.
FTRs are often considered preventable because the symptoms of a deteriorating patient
could present hours before the rescue starts. Examples of such critical symptom change include
patient complaint of a new pain, mental status change, and difficulty breathing etc. Studies have
indicated that many FTRs could have been averted if the critical symptoms in patients were
captured, evaluated, and communicated early.

5Human Factors in System Design
It was suggested that the nurses’ early recognition, evaluation, and decision making of
symptom signs could play an important role in FTR. A study conducted in a surgical oncology
population indicated that many complications are detectable by nurses and can be managed with
timely intervention. It was suggested that 23,000 in-hospital cardiac arrests in the UK could be
prevented every year if early signs of symptoms were detected and acted upon. A 2009 study
demonstrated that an early symptom recognition and response system could help improve
outcome of sepsis and septic shock, which have hard-to-detect symptoms.
Simply detecting and evaluating the critical symptom changes is not enough. The
potential complication must be communicated to the rest of the clinical team, and be escalated to
the right team members in order to organize effective interventions. It was argued that FTRs are
often caused by the failure to communicate. Interventions such as the rapid response team (RRT)
have demonstrated effectiveness in reducing FTRs when the issues are escalated on time. In fact,
the national deployment of RRT has the explicit purpose of supporting nurses in managing
critical changes before coding arrest. It was also suggested that escalating to surgical residents
could improve rescue success rates, indicating that the optimal path of escalation needs to be
selected by the nurses as part of the decision-making process.
Role of Frontline Nurses in Symptom Evaluations and Rapid Response Interventions:
Frontline nurses are often the first to notice critical symptom changes. Their decisions at
the point-of-care are crucial factors determining whether FTR events can be reduced. However,
at the same time, nurses are ill equipped to manage critical symptom changes in hospitals.
The frontline nursing staff in most hospitals have very high workloads, need to manage
extensive multitasking, and are fatigued. The fatigue has been demonstrated to negatively impact

6Human Factors in System Design
nurses’ cognitive performance, including symptom evaluations. In fact, studies have shown a
strong anti-correlation between nursing staffing levels and medical error rates.
The average skill and training levels of nurses do not adequately prepare them to evaluate
potentially complex symptom changes. A study found that a 10% increase in the proportion of
nurses holding a bachelor’s degree was associated with a 5% decrease in the odds of FTR.
Furthermore, most diagnostic aid CDSSs, such as differential diagnostic tools and diagnostic
reminder tools, were designed for physicians to use in office settings, as opposed to nurses at the
bedside.
While the RRT is a proven effective intervention for FTR, RRT resources can be under-
utilized because the nurses do not feel comfortable activating the RRT. Better communication
has been shown to improve RRT utilization. It has been suggested that mandatory RRT
activation helps reduce cardiorespiratory arrests outside of critical care areas in a hospital.
The hieratical structure in hospitals is known to impede nurse decision-making process.
Nurses are often discouraged from communicating and escalating problems (Zheng et al. 2018).
While hospitals across the nation have implemented teamwork frameworks, such as the
TeamSTEPPS, the emergency communication between nurses and physicians is still often error
prone and require standardization.
Design of CDSS
A specially designed CDSS could potentially help the nurse address the above issues
related to critical symptom changes and FTRs. Such CDSS requires special design
considerations for two reasons.

7Human Factors in System Design
First, the system must be tailored to the nurses’ training and cognitive levels, and
generate action items that are appropriate for the nurse. Most floor nurses have gone through less
than 4 years of medical training after high school, and they do not have independent authority to
treat the patient without the physician’s prescription.
Second, the system must be adapted to the fast paced workflow during a rescue operation.
The tool must be ubiquitous, instant on, and provides useful feedback in merely minutes. The
application should enhance real-time communication across team members, as opposed to
bringing in another computer that impedes face-to-face communication.
Both challenges highlight the need for a novel design, and formal evaluation of the
system UI and workflow.
Cognitive Design of UI:
Human-computer interaction and workflow designs are crucial for the success of clinical
informatics projects. A large body of research has been devoted to study methods and techniques
to evaluate usability of systems.
Early efforts focused on creating human models and breaking down tasks into small
pieces that could be directly measured and optimized for user performance. For instance, the
goals, operators, methods, and selection rules family of frameworks are widely used to model
human users as information processors. They break down user actions (eg, every key stroke), and
measure time consumed in each step to evaluate the overall effectiveness of the UI. However,
such frameworks do not take into account the intrinsic difficulty of the task and the functionality
of the UI. They are very good at evaluating systems that predominantly require movement
operations, but are less effective in evaluating systems with heavy cognitive tasks.

8Human Factors in System Design
For cognitive systems, analysis of the UI itself is a key aspect of usability design, because
UI design often has a deterministic effect on user performance. Research in cognitive theory has
indicated that different visual representation of the same underlying work problem could produce
dramatically different user performance in terms of ability to complete tasks correctly and
productivity. A well-known example is that Arabic numerals are much easier to add and multiply
than their equivalent Roman numerals.
Furthermore, complex work often requires collaboration of multiple users. It was
demonstrated that cognition can be distributed across multiple users working on the same
system. Hence, another important aspect of usability design is to evaluate each user’s goals and
functions, and then translate them into a cohesive UI.
A popular design approach that works with the above cognitive design principles is the
work-centered design (WCD). WCD treats the UI as an aid for the user to achieve a specific
work task. It conceptualizes steps for knowledge capture, requirement analysis, aiding design,
and evaluation, which is a process followed closely in modern software development.
A particularly interesting application of distributed cognition and WCD in the medical
informatics field is the UFuRT (user, function, representation, and task analysis) framework. For
this project, we decided to use the UFuRT framework as a guide for usability design. The
primary reason for us to choose UFuRT is its successful track record in design and evaluation of
medical information technology (IT) products. Its usability evaluation process consists of 4
major steps:

9Human Factors in System Design
1. User analysis is used to identify users and stakeholders of the work product, and
document their needs and objectives. The user requirements are translated into system
design requirements in this process.
2. Function analysis aims to generate an essential description of the work. The UFuRT
process calls for a 4-step analysis to detail the dimensions, constraints, relations, and
finally operations.
3. Representational analysis is the design process to identify and determine the
implementation representations of relations among the dimensions identified in the
functional analysis. The representation includes UIs and workflows for different types of
users of the system. Representational analysis is a crucial step of the design process since
it has been convincingly demonstrated that different representations of the same task can
have very different impacts on the user’s efficiency and productivity. The ease-of-use of
the UI is also one of the major factors driving adoption of any technology product.
4. Task analysis is to identify steps by a specific user on a specific representation in order to
carry out an operation.
In the context of our project, we used UFuRT framework to analyze software
requirements and inform the specification. Hence, we focused on user analysis and UI design
aspects of representation analysis. We performed a high-level functional analysis and did not
perform task analysis in the design stage. The reason was that complete functional and task
analysis require full knowledge of every detail of the product, which would not provide enough
flexibility for our iterative software development process.
Part FIVE: The evaluation

10Human Factors in System Design
Design Goals and System Requirements: The overall objective of the system was to help
prevent patient safety events during critical changes. Through interviews with hospital-based
clinicians, we have specifically identified symptom evaluation and escalation as the 2 main
functional goals of the CDSS.
Improve Symptom Recognition and Evaluation:
Existing Procedures: While nurses do not make diagnoses, they are the first to recognize
and evaluate the patient symptom changes. Based on their evaluation, they would decide how to
(or whether to) coordinate further care, and their evaluation results are often accepted by the
team as the basis of a formal diagnosis.
Existing diagnostic CDSS tools provide a proven framework to help reduce errors in
diagnostic evaluation, and improve documentation of the clinical findings that lead to diagnoses.
Specially, the CDSS needs to provide 2 core functionalities.
Provide Just-in-Time Medical Content to the Nurse: For many critical symptom changes,
there are multiple possible diagnoses. An example is that a hospitalized patient suddenly feels
chest pain. The chest pain could be an indicator of heart attack, which needs to be attended to by
a cardiologist or surgery team immediately; or the chest pain could indicate reflux or indigestion,
which is a rather common condition that is simple to treat.
The frontline nurses typically do not have enough medical training and experience to
thoroughly evaluate those potential diagnostic outcomes. The CDSS should provide specific
instructions for the nurse to follow, and then make recommendations on what to do next
(Shneiderman et al. 2016). For instance, it should provide specific instructions on whom to call
and what to say during the call for each potential diagnosis. The system does not replace human

11Human Factors in System Design
decision-making or training, but it provides support to help nurses deal with complicated
emergent situations to the best of their capabilities.
Reduce Common Cognitive Errors: Common cognitive errors that lead to diagnostic
errors include premature closure, anchoring, confirmatory bias, and framing. Those errors
happen because the clinicians ignore certain findings or give certain other findings too much
weight. Studies have indicated that cognitive errors such as premature closure are the most
common cause of diagnostic errors made by clinicians. A key design goal of the CDSS was to
help reduce those common cognitive errors.
To reduce framing and premature closure, the CDSS should encourage and prompt the
clinicians to check all possible diagnostic outcomes, especially severe outcomes that lead to
FTRs. The CDSS should also prompt the clinicians to verify all important symptoms and
findings related to major diagnostic outcomes to minimize missed diagnoses.
To reduce anchoring or confirmatory bias, the CDSS should present an objective estimate
of likely diagnoses and suggested clinical actions based on the current findings. The objective
probability estimate could reduce the user’s reliance on reconceived decision biases.
Facilitate Team Communication:
Teamwork: Teamwork is one of the few proven approaches to improve patient safety and
care quality in hospitals. Particularly, our system should be designed to increase the utilization of
the RRT, and improve communication between nurses and physicians.
RRT Utilization: RRT is an effective approach to help reduce FTR when it is deployed
correctly. Our CDSS aimed to improve the effectiveness of the RRT by activating RRT early and
making RRT mandatory when the nurse detects certain warning signs.