IMAT5209: Human Factor in System Design - Heuristic Evaluation Report

This report presents a heuristic evaluation of a mobile health system designed to support diabetes self-management. The study utilized a modified heuristic evaluation technique involving dual-domain experts (healthcare and usability professionals), validated scenarios, and in-depth severity ratings. The evaluation identified 129 usability problems with 274 heuristic violations, with the categories of Consistency and Standards, and Match between System and Real World dominating the issues. Severity ratings indicated that a significant number of violations were major or catastrophic, highlighting the need for redesign. The report details the evaluation methodology, including the use of specific scenarios and tasks to simulate patient interactions. The findings emphasize the importance of usability in mobile health applications and the potential for expert evaluations to identify crucial issues related to self-management and patient safety. The report concludes with a discussion of the implications of these findings for the design and development of more effective and user-friendly mobile health systems.

Running head: HUMAN FACTOR IN SYSTEM DESIGN
Human Factor in System Design: Heuristic Evaluation of Mobile Health
System for Self-management Support of Diabetes
Name of Student-
Name of University-
Author’s Note-

1HUMAN FACTOR IN SYSTEM DESIGN
Table of Contents
Part One: Interactive System and its Users......................................................................................2
Part Two: Use Cases........................................................................................................................3
Part Three: The Usability Requirement...........................................................................................4
Part Four: The Evaluation Methodology.........................................................................................6
Part Five: Evaluation.......................................................................................................................8
Part Six: Findings of Evaluation....................................................................................................11
Conclusion.....................................................................................................................................18
References......................................................................................................................................19

2HUMAN FACTOR IN SYSTEM DESIGN
Part One: Interactive System and its Users
Mobile health platforms offer significant opportunities for improving diabetic self-care,
but only if adequate usability exists. Expert evaluations such as heuristic evaluation can provide
distinct usability information about systems. The purpose of this study was to complete a
usability evaluation of a mobile health system for diabetes patients using a modified heuristic
evaluation technique of (1) dual-domain experts (healthcare professionals, usability experts), (2)
validated scenarios and user tasks related to patients’ self-care, and (3) in-depth severity factor
ratings. Experts identified 129 usability problems with 274 heuristic violations for the system.
The categories Consistency and Standards dominated at 24.1% (n = 66), followed by Match
between System and Real World at 22.3% (n = 61). Average severity ratings across system
views were 2.8 (of 4), with 9.3% (n = 12) rated as catastrophic and 53.5% (n = 69) as major. The
large volume of violations with severe ratings indicated clear priorities for redesign. The
modified heuristic approach allowed evaluators to identify unique and important issues,
including ones related to self-management and patient safety.
Recent figures from the World Health Organization show that 347 million people in the
world are affected by diabetes; this chronic disease is predicted to be the seventh leading cause
of death in the world by the year 2030 (Quiñones and Rusu, 2017). Data from the Department of
Health and Human Services and the Centers for Disease Control and Prevention show that in the
United States alone, the number of people living with the disease is 29.1 million. This number
continues to grow rapidly. Of the people diagnosed with this disease, 90% have type 2 diabetes.
Factors such as poorly regulated glycemic levels have a large influence on patients’ conditions
and are therefore vital to monitor for controlling the disease. Type 2 diabetes is also lifestyle
related to a large degree and can be self-managed to a certain extent in addition to using more

3HUMAN FACTOR IN SYSTEM DESIGN
conventional treatment procedures. Self-management is becoming increasingly important in
diabetes care; researchers found that self-management support should be integrated into patients’
everyday lives to achieve desired, improved patient outcomes. As an adjunct to diabetes
management, researchers highlighted the use of information and communication technology
(ICT) and the development of applications for day-to-day self-care and disease management. In
support of that goal, the mobile health (mHealth) system for this study was developed as an
individually based mobile and web support system for type 2 diabetes patients’ self-
management.
Part Two: Use Cases
To facilitate their adoption, mobile healthcare applications and systems for chronic
disease management must be usable. Despite the increasing availability of self-management
tools, many of the patient-operated ICT applications are still deficient in terms of usability.
Completing an evaluation process in a self-management context for these kinds of systems
requires an understanding of users and their needs when performing tasks in such a system.
Having this focus can help ensure that mobile applications are safe for clinical and patient use
and possibly prevent user errors. Authors thus argue that it is vital for user interfaces to be
designed in a way that does not contribute to errors as this can also be a factor negatively
affecting users’ experiences with the system. Usability evaluations can therefore help to
appropriately determine how well the application or system meets the clinical need and patients’
expectations and in safeguarding both the quality of care and patient outcomes. These
evaluations can be expert based, such as heuristic evaluation (HE) or cognitive walk-through, or
empirical and user based evaluations, such as think-aloud methods involving user tests with
actual system users.

4HUMAN FACTOR IN SYSTEM DESIGN
Figure 1: Use Case of the Mobile System
(Source: Created by Author)
Part Three: The Usability Requirement
System Description The mobile system we evaluated was designed as a low-cost,
convenient, personalized self-care management and tracking tool for use by a large number of
patients with diabetes. It was also meant to function as support for conversations between
patients and their healthcare providers. The system combined a web service and mobile phone
solution for patients to send in self-management values, that is, glucose and blood pressure, via
text messaging. The web user interface is divided into sections consisting of a Dashboard,
Glucose Diary, Blood Pressure, Medication Adherence, an Exercise and Weight progress, and

5HUMAN FACTOR IN SYSTEM DESIGN
Appointment reminder view. Each has graphical representations of the different measurements
and goals with progress indicators in red, yellow, and green. For example, the sections include a
meter to visualize glucose readings, a blood pressure bar with systolic and diastolic values, a
medication adherence section indicating how much of the prescribed medication was taken, and
an exercise and weight progress section to show exercise and weight measures. Using their cell
phones, patients can retrieve, enter, and edit their values and goals. Scenarios and tasks were
developed based on these kinds of patient interactions and uses.
Scenarios and tasks outlined specific steps that evaluators used to interact with the
diabetes self-management system in the HE. Tasks were based on real case scenarios to simulate
how patients would use the system in a self-management process in a clinic or at home. To
ensure that these were as realistic as possible, a panel also evaluated both scenarios and tasks.
The panel included a physician with a diabetes specialty, a diabetes RN, a public health
professional with chronic patient intervention systems expertise, and a diabetes patient. The
panel verified and validated tasks for content validity and accuracy (content validity index of
0.91 of 1.0). The eight tasks and scenarios were disease specific and had varying levels of
difficulty. For example, tasks consisted of viewing and locating glucose values on graphs,
identifying and correcting collected glucose values, setting weight and exercise goals and
medicine and appointment reminders, and viewing summary statements about medical
measurements. Table 1 includes an example of a scenario and task.
Scenario:
During your follow-up appointment with your provider, you agreed that a stronger
commitment regarding weight loss and exercise would improve your diabetes condition. You
now would like to activate the system's support service for exercise tracking and weight

6HUMAN FACTOR IN SYSTEM DESIGN
tracking and put in your tracking goals regarding your exercise and weight.
Please complete the following tasks.
1. Select and activate the service that you would like to use to set tracking goals for exercise
and weight.
2. Set your exercise goal to 3 times per week.
3. Set your weight goal to 180 pounds.
4. When you consider yourself done with the task, finish and return to “Participant Home.”
Table 1: Scenario and Task used
(Source: Georgsson, Staggers and Weir 2015)
Part Four: The Evaluation Methodology
Heuristic Evaluation Heuristic evaluation is one of the most common usability inspection
methods completed by usability experts. Users are explicitly not part of this kind of method.
Instead, experts apply the knowledge they have about usability principles, processes, and
standards to evaluate systems (Väätäjä et al. 2016).19 Heuristic evaluation was first defined by
Nielsen and Molich.16 In this technique, usability experts evaluate an application to find
usability problems, assign them to a specific category of heuristic and ascribe a severity rating.
Nielsen20 originally defined 10 heuristic categories and recommended assigning severity scores
to a master list of usability violations. Authors have attempted to modify and extend Nielsen’s
techniques in different ways to achieve better results in various contexts. These include Zhang et
al,21 who came up with 14 heuristics by combining Nielsen’s 10 heuristics with Shneiderman’s
eight “golden rules” to evaluate infusion pumps. Allen and colleagues9 employed a more
simplified use of the HE inspection method by having evaluators select only those heuristics they

7HUMAN FACTOR IN SYSTEM DESIGN
deemed appropriate for their assessment and assigned severity ratings for the usability problems
on the fly instead of first creating a master list of all usability problems. Chattrati chart and
Brodie22 extended the method to a technique they called HE-Plus, a directed approach using
usability problem profiles to help evaluators focus their evaluations on specific types of problem
areas to provide more consistent and reliable evaluation results. Heuristic evaluation or expert
usability evaluation can be useful because it provides a unique perspective and distinct
information19 and because it is a discount usability technique, meaning it is relatively quick, cost
effective, and resource efficient.16,23 However, as other authors have shown, the original
method by Nielsen can be improved upon for better results. Critics of the technique, for example,
indicate that many problems found with HE can be minor interface design problems or of a more
general nature. User tests, in comparison, involve actual users and identify problems of a critical,
qualitative nature. On the other hand, these are also more costly and time-consuming. In sum,
current expert techniques require improvements to be able to find more severe usability problems
of a critical nature for users. In this article, we addressed this gap. Our approach to accomplish
this is by using a modified HE technique using its beneficial aspects and also focusing on the
patient user and their needs in disease management and system information and interaction
requirements to provide enhanced evaluation results. Our modifications involve (1) employing
dual-domain experts (healthcare professionals and usability experts combined) as evaluators, (2)
using realistic, validated user tasks with appropriate scenarios related to patients’ diabetes self-
care, and (3) making severity ratings specific and in-depth across three severity rating factors by
predicting each problem’s influence on patients with factors of impact, persistence, and
frequency. Our intent was to explore whether the technique would be able to detect both crucial

8HUMAN FACTOR IN SYSTEM DESIGN
and context-related problems in patient self-management in addition to the more common, minor
usability issues.
Expert Evaluators The HE was performed by three expert evaluators who identified
heuristic violations listed in Nielsen’s taxonomy.20 According to Nielsen, three to five single-
domain usability expert evaluators find, on average, between 74% and 87% of usability
problems.27 The number of usability problems found by dual-domain experts is even higher at
81% to 90%. Only two to three dual-domain evaluators are then deemed necessary.27 These
types of experts are seen as especially suitable in evaluating complex systems, such as those in
the healthcare area, because they have usability expertise and extensive knowledge in the
specific domain of application.28,29 Each expert for this study was thus carefully selected based
on dual-domain competency consisting of (1) extensive usability experience in health
informatics, (2) being healthcare professionals (registered nurses [RNs]), and experience with (3)
the patient group and their task requirements, and (4) diabetes self-management. As this was a
HE evaluation, it involved only expert evaluators and no patients. Therefore, institutional review
board approval was not required for this study.
Part Five: Evaluation
Evaluators had identical instructional materials to learn the system and to ensure
consistency across evaluators. Information materials consisted of a digital video on system
modules, how to navigate the portal, a study design manual detailing each specific scenario and
tasks to be performed, an application user manual, and an evaluation guide sheet. The study
design manual also included materials on how to conduct the evaluation, the scenarios, and a
usability task manual outlining how to navigate tasks. Providing specific scenarios and tasks to

9HUMAN FACTOR IN SYSTEM DESIGN
simulate the diabetes patient care process ensured that all experts had the same knowledge level
about the functionality and user tasks (Rossi, Nagano and Neto 2016). The procedure itself was a
two-part process. The evaluators first familiarized themselves with the system and its usage
using the materials and training described above. Then, they performed the modified HE as
visualized in Figure 1. Each dual-domain expert evaluator performed the eight scenarios and
tasks independently. After the evaluators detected a usability problem, they assigned each
problem to a heuristic violation/s from the categories in Table 2. A master list was compiled,
duplicate problems were removed, and the list was verified across the evaluators for accuracy.
Then, each evaluator individually assigned severity scores to each problem by using the severity
rating factors of frequency, impact, and persistence. These were also averaged by factor and
combined into one severity rating for each usability problem as described above. Descriptive
statistics were used to summarize heuristic violations and associated severity scores.
Visibility of system status:
The system should always keep users informed about what is going on, through
appropriate feedback within reasonable time.
Follow real-world conventions, making information appear in a natural and logical order.
User control and freedom:
Users often choose system functions by mistake and will need a clearly marked
“emergency exit” to leave the unwanted state without having to go through an extended
dialogue. Support undo and redo.
Consistency and standards:
Users should not have to wonder whether different words, situations, or actions mean the
same thing. Follow platform conventions.

10HUMAN FACTOR IN SYSTEM DESIGN
Error prevention:
Even better than good error messages is a careful design which prevents a problem from
occurring in the first place. Either eliminate error-prone conditions or check for them and
present users with a confirmation option before they commit to the action.
Recognition rather than recall:
Minimize the user's memory load by making objects, actions, and options visible. The
user should not have to remember information from one part of the dialogue to another.
Instructions for use of the system should be visible or easily retrievable whenever
appropriate.
Flexibility and efficiency of use:
Accelerators—unseen by the novice user—may often speed up the interaction for the
expert user such that the system can cater to both inexperienced and experienced users.
Allow users to tailor frequent actions.
Aesthetic and minimalist design:
Dialogues should not contain information which is irrelevant or rarely needed. Every
extra unit of information in a dialogue competes with the relevant units of information
and diminishes their relative visibility.
Help users recognize, diagnose, and recover from errors:
Error messages should be expressed in plain language (no codes), precisely indicate the
problem, and constructively suggest a solution.
Help and documentation:
Even though it is better if the system can be used without documentation, it may be
necessary to provide help and documentation. Any such information should be easy to

11HUMAN FACTOR IN SYSTEM DESIGN
search, focused on the user's task, list concrete steps to be carried out, and not be too
large.
Table 2: Usability Evaluation as per Nielson
(Source: Georgsson, Staggers and Weir 2015)
Part Six: Findings of Evaluation
The HE resulted in a total of 129 usability problems and 274 heuristic violations. The
usability problems by place of occurrence (view), number of heuristic violations, and mean
severity ratings are summarized in Figure 2. The number of usability problems ranged from a
low of 12 to a high of 34 across application views. The Dashboard view generated the most
usability problems (34), followed by the Glucose Diary view (21), the Blood pressure view (20),
and the Medication adherence view (15). Heuristic evaluation violations ranged from 25 to 69.
The largest number of heuristic violations were was on the Dashboard view (69), the Glucose
Diary view (49), the Blood pressure view (44), the Medication adherence view (31), and the
Appointment reminder view (29). The average severity ratings ranged from 2.7 to 3 on a scale of
0 to 4, with the Glucose Diary view and Medication adherence view having the highest at 2.9
and 3.0 respectively.
Heuristic Violations across System Views
Of the 10 types of HE violations depicted in Figure 3, the categories of Consistency and
Standards and Match Between the System and the Real World dominated at 24.1% (n = 66) and
22.3% (n = 61) respectively, followed by Aesthetic and Minimalist Design at 16.8% (n = 46) and
Recognition Rather Than Recall at 11.7% (n = 32). The heuristic categories Recover 1.4% (n =
4) and Help 1.03% (n = 3) had the fewest violations across all views.

12HUMAN FACTOR IN SYSTEM DESIGN
Severity Ratings across System Views
Most severity ratings across views (Figure 4) consisted of major and catastrophic severity
ratings. The most severely rated problems were located in the Dashboard view (n = 16, n = 4),
Glucose Diary view (n = 12, n = 2), and the Blood pressure view (n = 12, n = 2). Most of the
minor usability problems were similarly located in the Dashboard view (n =14) and the Glucose
Diary view ( n= 7). The Appointment reminder view, however, came next with six issues. There
were no cosmetic violations.
Nature of Usability Problems and Prioritization
The modified HE evaluation revealed that most catastrophic ratings concerned disease-
related task deficiencies and specific system-related shortcomings in displaying necessary
information for patients (Joyce et al. 2016). Some examples of these types of usability problems
and comments provided by the evaluators are as follows:
Dashboard:
Each entry should, at minimum, have the time (not just the date) since many people with
diabetes will do multiple glucose tests in one day.
(The total severity rating was 3.8 and factor rating 4 for all evaluators for frequency, and
persistence.)
Glucose Diary view:
It is very difficult to read the time line on the graph because the numbers are too crowded
which makes it difficult to distinguish and read specific dates. This is especially cumbersome for
diabetes patients who often have visual concerns.

13HUMAN FACTOR IN SYSTEM DESIGN
(The total severity rating was 4.0 and factor rating 4 on all factors of frequency, impact,
and persistence.)
Blood Pressure view:
It is disadvantageous for the patient to not see the diastolic blood pressure reading in the
graph to compare against; only the systolic value is shown.
When I hover I only can see the systolic value. If the whole BP is displayed and rated,
what happens if only one value is abnormal?
(The total severity rating was 3.9 and factor rating 4 by all evaluators on frequency and
persistence.)
Medication Adherence view:
With the indications in percentages, it is difficult for a patient or health care provider to
determine what medication was taken or not, which day and what time.
Tallying up totals to say 100% of medications is an odd way to think about medication
from a patient perspective. I wouldn’t say that I have taken 75% of my meds for the last month
for instance. I need to know specifics and insulin or Metformin and that these are jointly
displayed and tracked.
I found that this display did not match my mental model of medication compliance, need
individual information regarding medication.
(Total severity rating of 3.7 factor rating of 4 for two evaluators on each factor of across
frequency, impact, and persistence.)

14HUMAN FACTOR IN SYSTEM DESIGN
Figure 2: Heuristic Evaluation Process
(Source: Georgsson, Staggers and Weir 2015)
Figure 3: Severity and frequency of severity

15HUMAN FACTOR IN SYSTEM DESIGN
(Source: Georgsson, Staggers and Weir 2015)
In this study, most usability problems were categorized as major issues. The largest
volume of problems clustered in the categories of Consistency and Standards and Match between
the System and the Real World. Both of these categories indicate that the system requires better
design to support effective decision making and action control for relevant patient user tasks.31
This evaluation uncovered specific concerns related to the disease-related information
deficiencies of the system. These included, for example, that the Dashboard view allows only
one daily entry and lacks a time for glucose readings on its current meter. These are fundamental
issues for a diabetes application. Since patients with diabetes often have multiple readings in one
day, this is a major usability problem related to essential information needed or patients’ self-
management tasks. In the Glucose diary, the numbers on the timeline were too small, crowded
together, and difficult to read. This is especially problematic for individuals with diabetes as they
often have visual acuity issues. The likelihood of users performing errors is increased for both of
these issues.
Using the in-depth severity factor ratings, the use of dualdomain evaluators and validated
user tasks related to the care process enabled the evaluators to find a large volume of severe
deficiencies in the system (Hermawati and Lawson 2016). Finding major issues with HE is in
contrast to other authors’ work both within and outside the health domain who mainly found
minor interface issues.23–25 The identified major and severe usability problems found here
require immediate attention for redesign to fulfill patients’ self-management needs. The
methodological changes to HE may have enhanced evaluators’ abilities in finding important
usability problems.

16HUMAN FACTOR IN SYSTEM DESIGN
Usability testing with users is very helpful, but as was seen here, dual-domain experts add
important, additional dimensions to any usability evaluation. In particular, they were able to
understand tasks as part of the context of use for a chronic disease self-management system and
were able to identify specific system design concerns and uncover distinct information needs
related to these self-management tasks. Dual-domain experts may also be able to assist in
identifying possible patient safety issues that patients may not identify. A case in point was that
the medicine reminder was too nonspecific and provided insufficient information, making it
possible for patients to commit errors in insulin dosages.
We recommend that dual-domain experts be employed in future HEs for chronic disease
management systems whenever possible. These kinds of experts can identify unique and critical
usability problems as well as deeper cognitive support issues and specific disease-related
concerns. Dual-domain experts provide added value in uncovering pertinent issues.28,29,32 This
information can be used both during the iterative design process, during formative evaluation,
summative testing, and for comparing different versions and applications.
Other methodological modifications can assist future usability evaluations. A
standardized evaluation process and use of specific scenarios and tasks allowed for efficient
evaluations across the experts and can also aid in reproducibility and generalizability. The
modified severity factor rating method in this study also proved important as it allowed the
evaluators to think about the specific impact of each individual usability problem and provided a
more in-depth analysis of the specific usability problems. This modification provided an
objective method to determine the importance of the usability problem in relation to others of a
similar nature and could aid in problem prioritization.
Limitations

17HUMAN FACTOR IN SYSTEM DESIGN
Although the modified HE process uncovered many major usability problems, other
problems might have been detected had the number of dual-domain evaluators been expanded
over the recommended two to three evaluators.27 Both HE and user tests could also be combined
to detect an optimal number of usability problems,33 although it would entail higher costs. We
wanted to provide an efficient, cost-effective method of modifying HE to also be able to identify
more serious usability issues that could have an impact on patients in their disease management.
CONCLUSION Consumer health systems and applications in mHealth should be evaluated for
usability as well as medical adequacy. This article describes useful modifications to HE by
modifying and deepening Nielsen’s techniques. Specifically, modifications were using dual-
domain experts; employing validated, patient-centered self-care tasks and realistic care process
scenarios; and using separate in-depth severity factor ratings. In particular, dual-domain experts
can provide unique information related to the salient tasks for patient self-care and identify
potential patient safety issues as well as determine how an application adheres to known usability
guidelines.
Conclusion
Consumer health systems and applications in mHealth should be evaluated for usability
as well as medical adequacy. This article describes useful modifications to HE by modifying and
deepening Nielsen’s techniques. Specifically, modifications were using dual-domain experts;
employing validated, patient-centered self-care tasks and realistic care process scenarios; and
using separate in-depth severity factor ratings. In particular, dual-domain experts can provide
unique information related to the salient tasks for patient self-care and identify potential patient
safety issues as well as determine how an application adheres to known usability guidelines.

18HUMAN FACTOR IN SYSTEM DESIGN
This modified heuristic method can be used by other informaticists in healthcare who
need to conduct a fast and resource-efficient heuristic process related to patient self-
management. The results of the study show that a modified HE can uncover unique, critical
issues in this context. This kind of evaluation may be done at any point in the system lifecycle.
As an expert evaluation technique, HE should be included in any usability evaluation and is quite
suitable for mHealth applications designed for chronic disease patients. Thus, it has an important
place in usability evaluations. With the modifications provided here, Nielsen’s original
techniques can be improved to achieve improved results. Techniques like the ones described here
can be an important addition to any informaticts toolbox when determining chronic disease
systems’ adequacy for patient self-management needs.

19HUMAN FACTOR IN SYSTEM DESIGN
References
Georgsson, M., Staggers, N. and Weir, C., 2016. A modified user-oriented heuristic evaluation of
a mobile health system for diabetes self-management support. Computers, Informatics,
Nursing, 34(2), p.77.
Hermawati, S. and Lawson, G., 2016. Establishing usability heuristics for heuristics evaluation in
a specific domain: Is there a consensus?. Applied ergonomics, 56, pp.34-51.
Joyce, G., Lilley, M., Barker, T. and Jefferies, A., 2016. Mobile application usability: heuristic
evaluation and evaluation of heuristics. In Advances in Human Factors, Software, and Systems
Engineering (pp. 77-86). Springer, Cham.
Quiñones, D. and Rusu, C., 2017. How to develop usability heuristics: A systematic literature
review. Computer Standards & Interfaces, 53, pp.89-122.
Rossi, F.L., Nagano, M.S. and Neto, R.F.T., 2016. Evaluation of high performance constructive
heuristics for the flow shop with makespan minimization. The International Journal of
Advanced Manufacturing Technology, 87(1-4), pp.125-136.
Väätäjä, H., Varsaluoma, J., Heimonen, T., Tiitinen, K., Hakulinen, J., Turunen, M., Nieminen,
H. and Ihantola, P., 2016, October. Information visualization heuristics in practical expert
evaluation. In Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel
Evaluation Methods for Visualization (pp. 36-43). ACM.