Graduate Engineering Project: CRehab - AR & Cloud Rehab System

This graduate engineering project introduces CRehab, a novel cloud-based web platform designed to advance the rehabilitation of patients, particularly after hip replacement surgery. The system utilizes augmented reality (AR) games and cloud computing to provide remote physiotherapy, offering real-time assessment of patient performance during exercises. A probabilistic approach, employing Hidden Markov Models (HMMs), is used to analyze and extract key issues in the execution of therapeutic activities. The project aims to leverage affordable technology to provide efficient and direct interaction, ensuring patients perform exercises correctly. Experimental testing, involving healthy participants and patients with hip problems, demonstrates CRehab's effectiveness in facilitating remote rehabilitation at a low cost, with improved patient outcomes. The system's architecture includes a cloud-based web platform, user interfaces for patients and physiotherapists, and an assessment module incorporating feature representation and trained HMMs. The project also includes a literature review of related work, detailing the evolution of tele-rehabilitation systems and their applications. The conclusions emphasize the potential of CRehab to enhance patient care and suggest recommendations for future research, especially regarding the AR AI algorithm.

GRADUATE ENGINEERING PROJECT: MONITORING REHABILITATION PROGRESS ON
PATIENTS BY USING AUGMENTED REALITY GAMES AND CLOUD COMPUTING
(CREHAB)

Table of Contents
Abstract.......................................................................................................................................................4
Introduction.................................................................................................................................................5
Literature Review of Related Work.............................................................................................................5
Methodology................................................................................................................................................7
Cloud Based Web Platform for the System.................................................................................................7
General Architecture...............................................................................................................................7
Website server........................................................................................................................................8
Client Side..............................................................................................................................................8
Web Socket.............................................................................................................................................9
User Interface.........................................................................................................................................9
Patient Interface....................................................................................................................................10
Physiotherapist User Interface..............................................................................................................11
Database...............................................................................................................................................11
Assessment Module..............................................................................................................................11
Representation of Features....................................................................................................................12
Trained HMMs.....................................................................................................................................13
Experimentation.........................................................................................................................................14
Setup.....................................................................................................................................................14
Training Models...................................................................................................................................15
Results.......................................................................................................................................................15
Conditioning.........................................................................................................................................19
Discussions................................................................................................................................................19
Conclusions...............................................................................................................................................20
Recommendations................................................................................................................................20
References.................................................................................................................................................21

Abstract
Advances in technology and computing provide new approaches on how physical rehabilitation can be
achieved. In that sense, the present investigation proposes a savvy, online stage to advance the correction
of patients after hip substitution medical procedure. This paper proposes a novel method for remote
physiotherapy rehabilitation and develops the CRehab cloud based application. A probabilistic
methodology dependent on the advancement and preparing of ten Hidden Markov Models (HMMs) is
utilized to separate progressively the primary issues in the execution of the remedial activities. The goal
was to use affordable technology and offer patients real time assessment of their performance of the
exercise. An experimental method was used to test the system, first using healthy participants then using
two patients with hip problems and one healthy one and the results show the CRehab system is effective
in ensuring the right exercises are performed and can help provide remote rehabilitation at low costs,
efficiently, and with direct interaction.

Introduction
With rising medical costs, the need to ensure patients have the best outcomes, and the need to judiciously
allocate increasingly limited resources, new trends have emerged in medical care, such as using Internet
of Things and remote patient monitoring. Among the new medical trends is the use of remote therapy in
rehabilitation medicine: this concept entails enabling patients to undertake a part of their rehabilitation at
their premises (such as at home) and communicate with their care givers on the evolution of their process
of recovery through the Internet. The use of this technology approach is justified based on medical
reasoning; it will result in improved recovery as more physical activities can be undertaken by the patient
frequently. It is also justified by economic reasons as the number of visits to the therapist and time spent
at the medical facility is reduced; ethics is another justification for the use of the system because it results
in greater democratization of health as the patient is more empowered and can make choices. Mobility is
another justification because the patient travels less to the therapist. However, despite its theoretical
benefits, the use of the remote therapy system for rehabilitation medicine raises another pertinent issue;
the patient will undertake exercises without being supervised and guided by a qualified therapist and this
raises concerns about how correct the movements used by the patient would be.
This graduate engineering project explores available options and proposes the use of a web based
platform to be utilized for remote/ Tele-physical rehabilitation of patients incorporating the use of cloud
computing and augmented reality. This is a novel approach that will ensure the therapist can monitor the
patient as well as ensures they use the correct physical movements to ensure fast and complete recovery
and enjoy the envisaged benefits of such a system (to be called CRehab). The proposed system will be
developed and tested using patients with mobility problems as a result of dislocation or problems with
their hip bone and require a functional and effective program of rehabilitation to recover their full
mobility, stability, and strength. The CRehab system will operate based on two fundamental conditions to
achieve the objectives of remote rehabilitation: first, it uses low cost devices for capturing motion, and
secondly, it uses a cloud platform with integrated artificial intelligent module for augmented reality. This
is aimed at evaluating how correct the motions used by the patient during physical activity are and give
real time feedback to a [patient as well as the physiotherapist. The [paper is organized into the following
sections, following the abstract and this brief introduction; a related works section in which a detailed
literature review of tele-rehabilitation systems is undertaken. This is followed by a section describes the
general cloud based front and back-end platform of the system. This is followed by a section that
describes the module for assessing the CRehab system. The following section is about tests done to
evaluate and prove the functionality and reliability of the CRehab system. The final section discusses the
results of the tests done, conclusions drawn, and recommendations for further studies and how the system
can be improved, especially the AR (augmented reality) AI algorithm.
Literature Review of Related Work
Tele-rehabilitation refers to rehabilitation services delivery through telecommunications networks,
including the Internet and most types generally fall into two categories; clinical therapy (the focus of this
study) and clinical assessment. Tele-rehabilitation and its use in medical/ clinical care have been explored
in diverse fields including audiology, neuropsychology, physical therapy, speech-language pathology, and
occupational therapy. People unable to travel to a medical center can benefit greatly from Tele-
rehabilitation and enables professionals in the field to consult with patients remotely (Kumar and Cohn,
2013). Advances in new computer science technologies as well as advanced telemedicine devices have
spawned an increase in the use of Telemedicine applications with various rehabilitation treatments using
this method being reported. Tele-rehabilitation, despite being relatively new, has grown tremendously
especially in the developed world and its benefits include reduced medical costs and access to quality
rehabilitation care for patients in remote regions. However, it still has its demerits and drawbacks,
including skepticism by patients because of interacting remotely with their physicians and this should not

be underestimated (Peretti et al., 2017). Tele-rehabilitation systems supporting physical therapy anywhere
are attractive since they can help lower health costs and ensure patients enjoy a better quality of life. One
of the Tele-rehabilitation systems that have been put forth is the Kinect-based Tele-rehabilitation system,
christened KiReS (Simonsen et al., 2014). Apart from providing remote Tele-rehabilitation, the KiReS
has two additional functionalities; the knowledge extraction functionality and the Tele-immersion
functionality. The extraction functionality deals with knowledge on patients’ physical therapy records as
well as treatment protocols to select adequate patient rehabilitation exercises. The Tele-immersion
functionality offers an effective, convenient, and user friendly experience for its users during
rehabilitation sessions via a real-time two-way multimedia communication. The KiReS was tested and
showed success for patients with total hip replacement and shoulder problems (Anton et al., 2018).
Tele-rehabilitation is a promising application in assisting survivors of strokes to reco0ver and regain fully
function of their organs. It can be used remotely for supervised administration of occupational, physical,
speech, and other forms of therapy to improve the cognitive, neuropsychiatric, and motor deficits suffered
by stroke patients. According to Sarfo et al. (2018), tele-rehabilitation has been shown, based on
evidence, to have better or equal salutary effects on the higher cortical, mood, and motor disorders when
compared to conventional one-on-one therapy. As the global population continues aging with older
people, chronic diseases also increase and their management continues to place tremendous pressure on
healthcare delivery. A further review of existing literature by Chen et al. (2015) shows that there is
moderate, albeit limited evidence that tele-rehabilitation using various approaches has effects that are
equivalent to conventional rehabilitation programs in improving stroke survivors’ abilities to engage in
daily living activities and it is recommended that more studies and approaches be used in this field to
improve outcomes. Technological advancements provide promising alternatives to augment health care
delivery for such populations and among the newest methods is the use of Tele-rehabilitation Robotics.
This approach combines Tele-healthcare and robot assisted rehabilitation features to offer rehabilitation
services remotely. The effectiveness of this approach has been reported in various studies with
significant statistical and clinical improvements in upper and lower limb function in patients. Further,
feasibility studies have documented significant improvements in patients’ high fidelity kinematics;
however, barriers exist even as Tele-rehabilitation robotics continues to show promise and these will need
addressing to ensure greater acceptance of Tele-rehabilitation (Housley, Fitzgerald and Butler, 2018).
Across most of Northern Europe, the population is aging and coupled with social and economic
challenges, the health care systems in these regions are at risk of being overwhelmed by high costs as
older people disproportionately require and use rehabilitation services. One solution, especially for those
in rural areas is to change the delivery mode to incorporate new technologies and approaches that have
been implemented include the SENDoc (Smart Sensor Devices for Rehab and Connected Health) which
was implemented within the Northern Periphery and Arctic Program. The SENDoc program entails using
wearable sensors and sensor systems in communities with elderly people. The benefits of the approach
include reduced costs of rehabilitation, enhanced outcomes, and better management of patients. However,
the approach is not without its problems that include complexity of the system, design and safety issues.
The program has been noted for its key advantage which is the possibility of changing patient behavior,
enabling patients live independently and empowering patient active participation in their wellbeing. The
system also implies less travel as there is very little face-to-face interactions with care givers (Esquivel et
al., 2018). This article demonstrates that emote rehabilitation is a promising technology for improving
rehabilitation services remotely with significant advantages and some drawbacks as well.
According to Banerji et al. (2016), there is a shortage of technology and manpower in rehabilitation on a
global scale to meet the rehabilitation needs of some five million new patients every year left disabled
with stroke. One approach to solving this problem is the use of Neuroplasticity as the main method of
attaining significant motor recovery. However, most devices used in rehabilitation are either limited to
mechanical repetitive practice movements at the limb level or focus on cognitive tasks only. This can lead

to improvements in impaired persons but rarely translates into effective hand and limb use for daily
activities. The authors then present a novel, easy to use system termed SynPhNe that trains muscle and
the brain as a single system through the use of Neuroplasticity principles. This solution can be used at
home through a wireless configuration for home based guided therapy sessions. Tey, An & Chung (2017)
propose a novel method for remote rehabilitation of patients suffering from chronic obstructive
pulmonary disease that results in difficulty in breathing and which worsens with time. This means
extensive hours are required for pulmonary rehabilitation exercises as well as management of the patient.
The proposed novel system is a remote rehabilitation system based on a system of multi modal sensors’
application for patients with breathing difficulties. The system works by capturing motion data through a
stereo camera and using a wearable PPG sensor to capture photoplethysmogram signals- these form the
input variables for the framework of detection and evaluation. This system was used for pulmonary
rehabilitation of patients with breathing problems with significant improvements along with cost savings,
efficiency, and ease of use.
The preceding review of literature and related works shows that it is generally acknowledged and proven
that remote rehabilitation can help solve problems of manpower deficiency in the field, given an
increasing aged population, especially in the Developed World. Technologies involving sensors and using
the Internet have been proposed, but these, despite having advantages such as cost reductions and
efficiency, face certain barriers such as complex technology and challenges with controlling the
movements and exercises so that patients are able to get it right when doing supervised or self-regulated
rehabilitation exercises. To this end, this paper proposes another novel approach using relatively simple
technology and leveraging the power of the cloud (cloud computing) and AI to develop the CRehab
system.
Methodology
This research is a novel proposal and is undertaken using an experimental setup in which the system is
first designed and described, taking into consideration functionality features. The described system is then
tested experimentally using scientific conventions with variables set and measured. As such, the research
is undertaken using a quantitative research framework. The experimentation was done randomly using
simple random sampling to test the accuracy of movements using healthy subjects. The simple
randomization was done to give every possible person an equal opportunity for participation in the
experiment and therefore attain reliability and validity of the research.
Cloud Based Web Platform for the System
General Architecture
One of the main objectives of developing the CRehab is to have an easy to use system for both patients
and rehab professionals as well as the general medical fraternity (data capture and integration with
Electronic Health Records). The CRehab platform is based on a client-server cloud based architecture
with a cloud based server containing the remote rehab application and its associated server with a cloud
native Node JS. The user interface is accessible through an API and enables the therapists and the patients
to use and manage the display and the Kinect data. The client-server cloud connection is established
through a web socket to enable full duplex communication channels via a single TCP connection. This
arrangement is shown in the image below;

Website server
The cloud website server is developed based on Django, a Python web framework language which is
developed using the Django CMS (content management system) to simplify web application
implementation.
Client Side
This is the side which the patient interacts with and the proposed system will have a Kinect camera which
is what is used for capturing the movements of the patient. From the patient’s computer, a cloud-based
application is launched to enable data to be sent from the camera to the website. To use the Kinect data
and enable exercises to be performed, there is a gateway between the cloud based server and the Kinect
camera for managing the flow of data. The application is launched in the systems’ cloud back end and
retrieves data sent by the Kinect camera and creates a link to server to enable network communication.
This is developed using a JavaScript run time environment (Node.js) that allows creating of the HTML
and JavaScript coded CRehab application. Data from the Kinect camera is managed by the back-end that
generates a HTML web page for browser. Node.js is chosen for use because it has a wide range is creative

modules especially the Kinect 2 library that makes communication with the Kinect simple. The http
library is used in starting the application and once started; JavaScript is generated, displaying an avatar of
the patient in the web browser as illustrated in the image below;
Web Socket
Node.js has a library known as Socket.IO which makes it possible to deploy a real time bi-directional
communication (event-based). To achieve this, the data destination address must be declared within the
Node.js application as well as the senders’ IP address (this is declared in the Django based website). This
way, the Socket.IO library for Python and the Node.js one are able to communicate via a function call
(Rangpariya, 2018).
User Interface
Different users are accommodated on the application web interface, including therapists and patients who
have singular functionalities. The rehabilitation has several parts, each part being a stage. Every stage
corresponds to different levels in the context of the intensity of;
Range of Movement – ROM
Force

Stretching
Walking
Transitioning from one stage to the next is possible when the patient has correctly completed all exercises
in their present stage and gets a nod from the physiotherapist to proceed, as such, one problem with
remote rehab is solved as the physiotherapist is able to control and manage the process, unlocking the
next stage after being satisfied the patient has done the present stage properly.
Patient Interface
The patent interface captures patient movement and this is done using a work flow as captured in the
image below;

Before starting rehabilitation exercises, the patient is required to fill out a questionnaire to evaluate their
ability to engage in and complete the exercises. This self-assessment questionnaire must be passed before
the exercises can begin, and it will assess their levels of pain and discomfort and any other problems. Low
scores means the patients cannot begin the rehabilitation ad a high score means the patients can
commence the rehabilitation exercise. The available stage is open for the patient to begin exercises and
they can first view an explanation of the exercises and movements. Upon commencing the exercises, the
users’ avatar as well as their real time feedback information is displayed on the screen. The assessment
questionnaire details are saved into the cloud based database as well as the 3 D coordinates captured by
the Kinect camera. When the exercise session is completed, the patient is able to evaluate their
movements and their performance feedback. The questions asked before commencing are also asked at
the end of the session so the therapist can better understand the patients’ condition after the exercises.
Physiotherapist User Interface
The health professional has several functions available for them in the CRehab application and will be
required to create an account and be approved by the administrator. They can then add new patients to the
system or update their information, monitor patient performance as well as their movements and their
overall progress. The physiotherapist/ health professionals will also be able to send or get messages from
the patients or eve chat with them; they can update or change the rehabilitation program exercises
intensity.
Database
The Django website is database driven; the Django CMS has an integrated relational database
management system feature (the SQLite3) (Tee, 2015) that is sufficient for the CRehab design; the goal is
to have a functional, easy to use, and less complex application system. The database stores information
about the users, including the Bio, address, location, and other personal details such as contacts. It also
stores the patients exercise information including the type of exercise, the sets, stage, series, and the
assessments of the exercises in a relational manner (each person with their associated exercises).
Assessment Module
The assessment module is developed based on the Hidden Markov Model (HMM) Approach
The HMM is a statistical model (Markov) where a system that is being modeled is presumed to be a
Markov process that has observable (the hidden) states. Evaluation of the movement quality of the
patients in real time will use this approach and incorporates an AR feature based on AI that is integrated
into the CRehab application for the purposes of conducting assessments and is based on the HMM. The
HMM is a probabilistic model applicable in time series analyses, for instance, analyzing movements and
gestures. Time scale invariability can be provided using HMM in recognizing temporal patterns as it
models real world data into ‘hidden states’. HMM has been applied successfully in gesture recognition
domains (Kao & Fahn, 2011; Wang, Xia, Cai, Gao & Cattani, 2012). It (HMM) has also been applied in
language and speech processing (Yu, 2010; Kumar, Aggarwal & Jain, 2012), analyzing stock markets (De
Angelis & Paas, 2013), weather fore-casing (Dong & Lam, 2013), among others. HMM works based on
probabilities that include the initial, transitional, and emission. Initial probabilities refer to probabilities
distribution of ‘being in a state’ before the observation of a sequence. Transitional probabilities refers to
those represented by matrices where the probabilities show possible changes from a given state to the
other, Emission probabilities are utilized in modeling variances of the associated values for each state,
usually Gaussian Probability Density Functions obtained from continuous observed variables (Chebud &
Melesse, 2011).
The Baum–Welch EM (Expectation–Maximization) algorithm will be used for estimating model
parameters; the EM is a forward- backward algorithm utilized in the falsification Hidden Markov Chains

(Ghoniem & Shaalan, 2017). Probabilities are computed at point of sequence through inspection of past
values to establish how well the rest of the sequence is predicted by the model through an iterative
process aimed at finding the optimal HHM solution. The Viterbi algorithm is applied in inferring the
optimal sequence of states and these makes the approach suitable for classification (Mahé, Vercauteren,
Rosa & Dauguet, 2013). Characteristic of HMM is real world observations in the order (01…..0n) in which
(01…..0n) e X (Discrete, R); from these observations, ‘states’ can be inferred. Every state depends on the
preceding state although there is conditional independence of the observations given the state. The joint
probability p (o1, …, on, h1, …, hn) is arrived at through factorization (Sharma, Golubchik & Govindan,
2010) as shown below;
p(h1)p(o1|h1)∏i=2np(hi|hi−1)p(oi|hi) …………………………………….(1)
All possible probabilities of changing from a given hidden state into another make up the transition
probabilities. The forward algorithm is then used to compute the probability of a sequence generation
being trained HMM.
Representation of Features
The Kinect camera is used for capturing and extracting the 3-D (three dimensional) coordinates of a
person’s main body joints although other factors are required because joint coordinates are 4-D with yaw,
roll, and pitch, rather than 3-D. The personal coordinate system is created using depth data so motion can
be expressed in the form of relative movements in the sagittal, frontal, and horizontal planes.
Transformations to angle positions from joint positions are straightforward and can be easily done using
the cosine law (Reddivari et al., 2014);
Cos(A)=−a2+b2+c2/2bc…………………………………………………...(2)
Using a similar fashion, additional vectors are computed to represent the saggital plane (hip rotations)
while the PCS (personal coordinate system) can be used in constructing the floor plane and the vector
crossing all the points is translated to the reference point. Two vectors converge at the reference point,
enabling the extraction of the third direction by determining the perpendicular of the directions using the
cross-product rule;
A→× B→ =[a2b3−a3b2,a3b1−a1b3,a1b2−a2b1] ……………………….(3) (Alagu Priya & Lim,
2010).
The vectors are represented below;