Monitoring Rehabilitation Progress using Augmented Reality Games and Cloud Computing

This research proposes a novel method for remote physiotherapy rehabilitation using augmented reality games and cloud computing. The system aims to provide real-time assessment of patients' performance and offer remote rehabilitation at low costs. The research includes a literature review, methodology, and a cloud-based web platform for the system. The system is tested experimentally and the results show its effectiveness in ensuring the right exercises are performed.

GRADUATE ENGINEERING PROJECT: MONITORING REHABILITATION PROGRESS ON
PATIENTS BY USING AUGMENTED REALITY GAMES AND CLOUD COMPUTING
(CREHAB)

Table of Contents
Abstract.......................................................................................................................................................4
Introduction.................................................................................................................................................5
Literature Review of Related Work.............................................................................................................5
Methodology................................................................................................................................................7
Cloud Based Web Platform for the System.................................................................................................7
General Architecture...............................................................................................................................7
Website server........................................................................................................................................8
Client Side..............................................................................................................................................8
Web Socket.............................................................................................................................................9
User Interface.........................................................................................................................................9
Patient Interface....................................................................................................................................10
Physiotherapist User Interface..............................................................................................................11
Database...............................................................................................................................................11
Assessment Module..............................................................................................................................11
Representation of Features....................................................................................................................12
Trained HMMs.....................................................................................................................................13
Experimentation.........................................................................................................................................14
Setup.....................................................................................................................................................14
Training Models...................................................................................................................................15
Results.......................................................................................................................................................15
Conditioning.........................................................................................................................................19
Discussions................................................................................................................................................19
Conclusions...............................................................................................................................................20
Recommendations................................................................................................................................20
References.................................................................................................................................................21

Abstract
Advances in technology and computing provide new approaches on how physical rehabilitation can be
achieved. In that sense, the present investigation proposes a savvy, online stage to advance the correction
of patients after hip substitution medical procedure. This paper proposes a novel method for remote
physiotherapy rehabilitation and develops the CRehab cloud based application. A probabilistic
methodology dependent on the advancement and preparing of ten Hidden Markov Models (HMMs) is
utilized to separate progressively the primary issues in the execution of the remedial activities. The goal
was to use affordable technology and offer patients real time assessment of their performance of the
exercise. An experimental method was used to test the system, first using healthy participants then using
two patients with hip problems and one healthy one and the results show the CRehab system is effective
in ensuring the right exercises are performed and can help provide remote rehabilitation at low costs,
efficiently, and with direct interaction.

Introduction
With rising medical costs, the need to ensure patients have the best outcomes, and the need to judiciously
allocate increasingly limited resources, new trends have emerged in medical care, such as using Internet
of Things and remote patient monitoring. Among the new medical trends is the use of remote therapy in
rehabilitation medicine: this concept entails enabling patients to undertake a part of their rehabilitation at
their premises (such as at home) and communicate with their care givers on the evolution of their process
of recovery through the Internet. The use of this technology approach is justified based on medical
reasoning; it will result in improved recovery as more physical activities can be undertaken by the patient
frequently. It is also justified by economic reasons as the number of visits to the therapist and time spent
at the medical facility is reduced; ethics is another justification for the use of the system because it results
in greater democratization of health as the patient is more empowered and can make choices. Mobility is
another justification because the patient travels less to the therapist. However, despite its theoretical
benefits, the use of the remote therapy system for rehabilitation medicine raises another pertinent issue;
the patient will undertake exercises without being supervised and guided by a qualified therapist and this
raises concerns about how correct the movements used by the patient would be.
This graduate engineering project explores available options and proposes the use of a web based
platform to be utilized for remote/ Tele-physical rehabilitation of patients incorporating the use of cloud
computing and augmented reality. This is a novel approach that will ensure the therapist can monitor the
patient as well as ensures they use the correct physical movements to ensure fast and complete recovery
and enjoy the envisaged benefits of such a system (to be called CRehab). The proposed system will be
developed and tested using patients with mobility problems as a result of dislocation or problems with
their hip bone and require a functional and effective program of rehabilitation to recover their full
mobility, stability, and strength. The CRehab system will operate based on two fundamental conditions to
achieve the objectives of remote rehabilitation: first, it uses low cost devices for capturing motion, and
secondly, it uses a cloud platform with integrated artificial intelligent module for augmented reality. This
is aimed at evaluating how correct the motions used by the patient during physical activity are and give
real time feedback to a [patient as well as the physiotherapist. The [paper is organized into the following
sections, following the abstract and this brief introduction; a related works section in which a detailed
literature review of tele-rehabilitation systems is undertaken. This is followed by a section describes the
general cloud based front and back-end platform of the system. This is followed by a section that
describes the module for assessing the CRehab system. The following section is about tests done to
evaluate and prove the functionality and reliability of the CRehab system. The final section discusses the
results of the tests done, conclusions drawn, and recommendations for further studies and how the system
can be improved, especially the AR (augmented reality) AI algorithm.
Literature Review of Related Work
Tele-rehabilitation refers to rehabilitation services delivery through telecommunications networks,
including the Internet and most types generally fall into two categories; clinical therapy (the focus of this
study) and clinical assessment. Tele-rehabilitation and its use in medical/ clinical care have been explored
in diverse fields including audiology, neuropsychology, physical therapy, speech-language pathology, and
occupational therapy. People unable to travel to a medical center can benefit greatly from Tele-
rehabilitation and enables professionals in the field to consult with patients remotely (Kumar and Cohn,
2013). Advances in new computer science technologies as well as advanced telemedicine devices have
spawned an increase in the use of Telemedicine applications with various rehabilitation treatments using
this method being reported. Tele-rehabilitation, despite being relatively new, has grown tremendously
especially in the developed world and its benefits include reduced medical costs and access to quality
rehabilitation care for patients in remote regions. However, it still has its demerits and drawbacks,
including skepticism by patients because of interacting remotely with their physicians and this should not

be underestimated (Peretti et al., 2017). Tele-rehabilitation systems supporting physical therapy anywhere
are attractive since they can help lower health costs and ensure patients enjoy a better quality of life. One
of the Tele-rehabilitation systems that have been put forth is the Kinect-based Tele-rehabilitation system,
christened KiReS (Simonsen et al., 2014). Apart from providing remote Tele-rehabilitation, the KiReS
has two additional functionalities; the knowledge extraction functionality and the Tele-immersion
functionality. The extraction functionality deals with knowledge on patients’ physical therapy records as
well as treatment protocols to select adequate patient rehabilitation exercises. The Tele-immersion
functionality offers an effective, convenient, and user friendly experience for its users during
rehabilitation sessions via a real-time two-way multimedia communication. The KiReS was tested and
showed success for patients with total hip replacement and shoulder problems (Anton et al., 2018).
Tele-rehabilitation is a promising application in assisting survivors of strokes to reco0ver and regain fully
function of their organs. It can be used remotely for supervised administration of occupational, physical,
speech, and other forms of therapy to improve the cognitive, neuropsychiatric, and motor deficits suffered
by stroke patients. According to Sarfo et al. (2018), tele-rehabilitation has been shown, based on
evidence, to have better or equal salutary effects on the higher cortical, mood, and motor disorders when
compared to conventional one-on-one therapy. As the global population continues aging with older
people, chronic diseases also increase and their management continues to place tremendous pressure on
healthcare delivery. A further review of existing literature by Chen et al. (2015) shows that there is
moderate, albeit limited evidence that tele-rehabilitation using various approaches has effects that are
equivalent to conventional rehabilitation programs in improving stroke survivors’ abilities to engage in
daily living activities and it is recommended that more studies and approaches be used in this field to
improve outcomes. Technological advancements provide promising alternatives to augment health care
delivery for such populations and among the newest methods is the use of Tele-rehabilitation Robotics.
This approach combines Tele-healthcare and robot assisted rehabilitation features to offer rehabilitation
services remotely. The effectiveness of this approach has been reported in various studies with
significant statistical and clinical improvements in upper and lower limb function in patients. Further,
feasibility studies have documented significant improvements in patients’ high fidelity kinematics;
however, barriers exist even as Tele-rehabilitation robotics continues to show promise and these will need
addressing to ensure greater acceptance of Tele-rehabilitation (Housley, Fitzgerald and Butler, 2018).
Across most of Northern Europe, the population is aging and coupled with social and economic
challenges, the health care systems in these regions are at risk of being overwhelmed by high costs as
older people disproportionately require and use rehabilitation services. One solution, especially for those
in rural areas is to change the delivery mode to incorporate new technologies and approaches that have
been implemented include the SENDoc (Smart Sensor Devices for Rehab and Connected Health) which
was implemented within the Northern Periphery and Arctic Program. The SENDoc program entails using
wearable sensors and sensor systems in communities with elderly people. The benefits of the approach
include reduced costs of rehabilitation, enhanced outcomes, and better management of patients. However,
the approach is not without its problems that include complexity of the system, design and safety issues.
The program has been noted for its key advantage which is the possibility of changing patient behavior,
enabling patients live independently and empowering patient active participation in their wellbeing. The
system also implies less travel as there is very little face-to-face interactions with care givers (Esquivel et
al., 2018). This article demonstrates that emote rehabilitation is a promising technology for improving
rehabilitation services remotely with significant advantages and some drawbacks as well.
According to Banerji et al. (2016), there is a shortage of technology and manpower in rehabilitation on a
global scale to meet the rehabilitation needs of some five million new patients every year left disabled
with stroke. One approach to solving this problem is the use of Neuroplasticity as the main method of
attaining significant motor recovery. However, most devices used in rehabilitation are either limited to
mechanical repetitive practice movements at the limb level or focus on cognitive tasks only. This can lead

to improvements in impaired persons but rarely translates into effective hand and limb use for daily
activities. The authors then present a novel, easy to use system termed SynPhNe that trains muscle and
the brain as a single system through the use of Neuroplasticity principles. This solution can be used at
home through a wireless configuration for home based guided therapy sessions. Tey, An & Chung (2017)
propose a novel method for remote rehabilitation of patients suffering from chronic obstructive
pulmonary disease that results in difficulty in breathing and which worsens with time. This means
extensive hours are required for pulmonary rehabilitation exercises as well as management of the patient.
The proposed novel system is a remote rehabilitation system based on a system of multi modal sensors’
application for patients with breathing difficulties. The system works by capturing motion data through a
stereo camera and using a wearable PPG sensor to capture photoplethysmogram signals- these form the
input variables for the framework of detection and evaluation. This system was used for pulmonary
rehabilitation of patients with breathing problems with significant improvements along with cost savings,
efficiency, and ease of use.
The preceding review of literature and related works shows that it is generally acknowledged and proven
that remote rehabilitation can help solve problems of manpower deficiency in the field, given an
increasing aged population, especially in the Developed World. Technologies involving sensors and using
the Internet have been proposed, but these, despite having advantages such as cost reductions and
efficiency, face certain barriers such as complex technology and challenges with controlling the
movements and exercises so that patients are able to get it right when doing supervised or self-regulated
rehabilitation exercises. To this end, this paper proposes another novel approach using relatively simple
technology and leveraging the power of the cloud (cloud computing) and AI to develop the CRehab
system.
Methodology
This research is a novel proposal and is undertaken using an experimental setup in which the system is
first designed and described, taking into consideration functionality features. The described system is then
tested experimentally using scientific conventions with variables set and measured. As such, the research
is undertaken using a quantitative research framework. The experimentation was done randomly using
simple random sampling to test the accuracy of movements using healthy subjects. The simple
randomization was done to give every possible person an equal opportunity for participation in the
experiment and therefore attain reliability and validity of the research.
Cloud Based Web Platform for the System
General Architecture
One of the main objectives of developing the CRehab is to have an easy to use system for both patients
and rehab professionals as well as the general medical fraternity (data capture and integration with
Electronic Health Records). The CRehab platform is based on a client-server cloud based architecture
with a cloud based server containing the remote rehab application and its associated server with a cloud
native Node JS. The user interface is accessible through an API and enables the therapists and the patients
to use and manage the display and the Kinect data. The client-server cloud connection is established
through a web socket to enable full duplex communication channels via a single TCP connection. This
arrangement is shown in the image below;

Website server
The cloud website server is developed based on Django, a Python web framework language which is
developed using the Django CMS (content management system) to simplify web application
implementation.
Client Side
This is the side which the patient interacts with and the proposed system will have a Kinect camera which
is what is used for capturing the movements of the patient. From the patient’s computer, a cloud-based
application is launched to enable data to be sent from the camera to the website. To use the Kinect data
and enable exercises to be performed, there is a gateway between the cloud based server and the Kinect
camera for managing the flow of data. The application is launched in the systems’ cloud back end and
retrieves data sent by the Kinect camera and creates a link to server to enable network communication.
This is developed using a JavaScript run time environment (Node.js) that allows creating of the HTML
and JavaScript coded CRehab application. Data from the Kinect camera is managed by the back-end that
generates a HTML web page for browser. Node.js is chosen for use because it has a wide range is creative

modules especially the Kinect 2 library that makes communication with the Kinect simple. The http
library is used in starting the application and once started; JavaScript is generated, displaying an avatar of
the patient in the web browser as illustrated in the image below;
Web Socket
Node.js has a library known as Socket.IO which makes it possible to deploy a real time bi-directional
communication (event-based). To achieve this, the data destination address must be declared within the
Node.js application as well as the senders’ IP address (this is declared in the Django based website). This
way, the Socket.IO library for Python and the Node.js one are able to communicate via a function call
(Rangpariya, 2018).
User Interface
Different users are accommodated on the application web interface, including therapists and patients who
have singular functionalities. The rehabilitation has several parts, each part being a stage. Every stage
corresponds to different levels in the context of the intensity of;
Range of Movement – ROM
Force

Stretching
Walking
Transitioning from one stage to the next is possible when the patient has correctly completed all exercises
in their present stage and gets a nod from the physiotherapist to proceed, as such, one problem with
remote rehab is solved as the physiotherapist is able to control and manage the process, unlocking the
next stage after being satisfied the patient has done the present stage properly.
Patient Interface
The patent interface captures patient movement and this is done using a work flow as captured in the
image below;

Before starting rehabilitation exercises, the patient is required to fill out a questionnaire to evaluate their
ability to engage in and complete the exercises. This self-assessment questionnaire must be passed before
the exercises can begin, and it will assess their levels of pain and discomfort and any other problems. Low
scores means the patients cannot begin the rehabilitation ad a high score means the patients can
commence the rehabilitation exercise. The available stage is open for the patient to begin exercises and
they can first view an explanation of the exercises and movements. Upon commencing the exercises, the
users’ avatar as well as their real time feedback information is displayed on the screen. The assessment
questionnaire details are saved into the cloud based database as well as the 3 D coordinates captured by
the Kinect camera. When the exercise session is completed, the patient is able to evaluate their
movements and their performance feedback. The questions asked before commencing are also asked at
the end of the session so the therapist can better understand the patients’ condition after the exercises.
Physiotherapist User Interface
The health professional has several functions available for them in the CRehab application and will be
required to create an account and be approved by the administrator. They can then add new patients to the
system or update their information, monitor patient performance as well as their movements and their
overall progress. The physiotherapist/ health professionals will also be able to send or get messages from
the patients or eve chat with them; they can update or change the rehabilitation program exercises
intensity.
Database
The Django website is database driven; the Django CMS has an integrated relational database
management system feature (the SQLite3) (Tee, 2015) that is sufficient for the CRehab design; the goal is
to have a functional, easy to use, and less complex application system. The database stores information
about the users, including the Bio, address, location, and other personal details such as contacts. It also
stores the patients exercise information including the type of exercise, the sets, stage, series, and the
assessments of the exercises in a relational manner (each person with their associated exercises).
Assessment Module
The assessment module is developed based on the Hidden Markov Model (HMM) Approach
The HMM is a statistical model (Markov) where a system that is being modeled is presumed to be a
Markov process that has observable (the hidden) states. Evaluation of the movement quality of the
patients in real time will use this approach and incorporates an AR feature based on AI that is integrated
into the CRehab application for the purposes of conducting assessments and is based on the HMM. The
HMM is a probabilistic model applicable in time series analyses, for instance, analyzing movements and
gestures. Time scale invariability can be provided using HMM in recognizing temporal patterns as it
models real world data into ‘hidden states’. HMM has been applied successfully in gesture recognition
domains (Kao & Fahn, 2011; Wang, Xia, Cai, Gao & Cattani, 2012). It (HMM) has also been applied in
language and speech processing (Yu, 2010; Kumar, Aggarwal & Jain, 2012), analyzing stock markets (De
Angelis & Paas, 2013), weather fore-casing (Dong & Lam, 2013), among others. HMM works based on
probabilities that include the initial, transitional, and emission. Initial probabilities refer to probabilities
distribution of ‘being in a state’ before the observation of a sequence. Transitional probabilities refers to
those represented by matrices where the probabilities show possible changes from a given state to the
other, Emission probabilities are utilized in modeling variances of the associated values for each state,
usually Gaussian Probability Density Functions obtained from continuous observed variables (Chebud &
Melesse, 2011).
The Baum–Welch EM (Expectation–Maximization) algorithm will be used for estimating model
parameters; the EM is a forward- backward algorithm utilized in the falsification Hidden Markov Chains

(Ghoniem & Shaalan, 2017). Probabilities are computed at point of sequence through inspection of past
values to establish how well the rest of the sequence is predicted by the model through an iterative
process aimed at finding the optimal HHM solution. The Viterbi algorithm is applied in inferring the
optimal sequence of states and these makes the approach suitable for classification (Mahé, Vercauteren,
Rosa & Dauguet, 2013). Characteristic of HMM is real world observations in the order (01…..0n) in which
(01…..0n) e X (Discrete, R); from these observations, ‘states’ can be inferred. Every state depends on the
preceding state although there is conditional independence of the observations given the state. The joint
probability p (o1, …, on, h1, …, hn) is arrived at through factorization (Sharma, Golubchik & Govindan,
2010) as shown below;
p(h1)p(o1|h1)∏i=2np(hi|hi−1)p(oi|hi) …………………………………….(1)
All possible probabilities of changing from a given hidden state into another make up the transition
probabilities. The forward algorithm is then used to compute the probability of a sequence generation
being trained HMM.
Representation of Features
The Kinect camera is used for capturing and extracting the 3-D (three dimensional) coordinates of a
person’s main body joints although other factors are required because joint coordinates are 4-D with yaw,
roll, and pitch, rather than 3-D. The personal coordinate system is created using depth data so motion can
be expressed in the form of relative movements in the sagittal, frontal, and horizontal planes.
Transformations to angle positions from joint positions are straightforward and can be easily done using
the cosine law (Reddivari et al., 2014);
Cos(A)=−a2+b2+c2/2bc…………………………………………………...(2)
Using a similar fashion, additional vectors are computed to represent the saggital plane (hip rotations)
while the PCS (personal coordinate system) can be used in constructing the floor plane and the vector
crossing all the points is translated to the reference point. Two vectors converge at the reference point,
enabling the extraction of the third direction by determining the perpendicular of the directions using the
cross-product rule;
A→× B→ =[a2b3−a3b2,a3b1−a1b3,a1b2−a2b1] ……………………….(3) (Alagu Priya & Lim,
2010).
The vectors are represented below;

Vectors create the personal coordinate system for a patient with the new origin being he center floor; blue
lines are the lower limbs, additional vectors are represented by the dotted lines required for computing hip
movements in sagittal plane.
The absolute coordinates for each joint are converted into personal coordinates using the new orientations
such that the origin is floor crossing and movements can be expressed as left or right, front or back, and
up or down. The three directions speeds are represented by differences in orientation between successive
frames and the speed is the average sum for 15 frames. Rotations to be captured resemble the internal and
external rotation of the upper leg.
Trained HMMs
Learning must be achieved by the AR AI and this means the features are incorporated into the learning
parameters such that the semantics comply with the analysis of an exercise by the therapist. The HMM
representation incorporates most coarse movement types and restrictions posed by the Skeletonization
algorithm also considered. As such, the earlier representation now transforms into a tenfold semantic
manifestation as shown below;

Experimentation
Subjects were selected and asked to take part in the experiment as part of the testing of a new approach to
medical rehabilitation using the CRehab system. The quality of performed exercises is tested
automatically according to the HMM approach as the method gives insights into the rehabilitation
exercises ontological structure and represents a probabilistic inferences on how correct the physical
rehabilitation exercises are (Capecci et al., 2018; Reddivari et al., 2014). The different compensatory
movements ontology associated with right leg hip abduction exercise are analyzed experimentally using
healthy participants. Patients usually employ compensatory exercises in order to make physical exercises
easier (Han, Song & Kim, 2011). The compensatory movements, however, limit functional
improvements intended during rehabilitation for affected limbs and can result in pain and as seen from the
literature review, compensatory approaches sought to limit these movements. The proposed algorithm
will achieve higher accuracy of measuring movement as it can be compared to assessments
simultaneously created by the physiotherapists as suggested by Dixon, Zhang, Domalain, Flores & Lin
(2011), that electronic rehabilitation systems must be re-evaluated.
Setup
The study was undertaken with nine willing participants and it involved the participants executing hip
abduction exercises and the exercise repeated eight times for every participant. At the same time, the
movement of the participants on three different aspects was evaluated by therapists; the aspects include

ROM, compensation, and coordination. Coordination corresponds to synchronization of movement of
different body parts when an exercise is executed; ROM corresponds to movement amplitude, while
compensation is the undesired movements. The participants got feedback on how correct their
performance of the exercises was after the fifth set while subsequent exercises (three) were followed by
feedback being given by the therapists. The Kinects skeletal data is represented on the joints degrees of
freedom and the vectors created have four variables as shown below;
Feature vectors
F0 F1 F2 F3
Speed of hip center in
ms-1
Speed of shoulder center
in ms-1
Speed of right hip
Frontal plane in degrees
Frontal plane right hip
angel in degrees
15 frames buffer is used for calculating the speeds with the speed being the average of buffer values. The
data stream is available after the filling of the buffer while angles of upper legs in the frontal planes are
also integrated into the model and angles computed based on Cosine law and formula (2).
Training Models
HMM is an approach that is probabilistic aimed at modeling a given signal to hidden states; of which
states represent an arbitrary whole movement decomposition into successive phases. For determined
individuals, HMMs are trained using correct trials based on what the therapists determined of other eight
participants. Every model can predict the state of a given joint or group of joints and is important in
overcoming faults.
Results
The findings are labeled as follows;
G Good
F Fair
B Bad
Using movement symmetry, compensation is able to be estimated as it offers an insight into motor control
over exercise execution and must be characterized by even distribution of the antagonist and agonist
muscular loads in case of symmetry.

The synchronization between movements of the shoulder and shoulder movement symmetry were also
determined with values near 1 depicting perfect synchronicity/ symmetry. For synchronicity, values LESS
than 1 indicate that moved before the movement of the hip abduction movement and values GREATER
than 1 show the shoulder center moved following movement of hip abduction. For symmetry, scores
LESS than 1 imply the shoulder center achieved maximum amplitude faster than the time it took to return
to the starting pose and the opposite is true for scores GREATER than 1. The trials were conducted and
the average for each determined

The synchronicity between movement of the shoulder and symmetry of shoulder movement was also
conducted; results are shown below;
After getting the right movements, a second experiment was carried out with three participants, two with
hip problems and one a healthy participant. The second investigation is set up to examination the
pertinence of the HMM models in surveying the nature of the self-recovery developments continuously
and in clinical conditions. The quality is assessed as right or off base, with remuneration blames that
spread the most well-known mistakes during a normal restoration program. Such a trial is legitimized by
the way that sadly a vast lion's share of the models that intend to help physical recovery are not tried on
the end client
Conditioning
Due to the age of the participants in the second experiment, the exercises were demonstrated a couple of
times and the actions described explicitly. There was a three step game exercise which one of the older
participants cold not memorize clearly; showing that cognitive abilities also play a role in rehabilitation.
The participants clothing/ dressing had no impact on their exercises and the tracking process for skeletal
movements was done. A comparison of the movements of patients with hip problems and the healthy
ones showed significant compensation by the elderly patients. HMMs doe by the healthy patients
averaged 1451 for the healthy subjects and 1211 for the patients, showing that the healthy individuals had
a better execution of the exercises than the patients. The developed model was able to detect a serious
error during trial where there was a very low score and this demonstrates the model is as accurate as a
human therapist in automatically evaluating rehabilitation exercise performance.

Discussions
The findings from the study reinforce the belief that the use of remote rehabilitation can result in positive
outcomes for both patients and therapists, and the medical care system as a whole. The developed system
architecture ensures that data can be captured by the Kinect camera and be used to evaluate the
performance of a patient remotely. The model is also able to detect movements that are executed wrongly
and those executed correctly during self-rehabilitation and is able to capture and store data in real time for
assessment by a therapist, either I real time or after the exercises have been done and is able to capture
some compensation as perceived by the therapists and synchronicity when exercises are being performed.
The sequences for state transitions offer important clues on how compensations have occurred. The
trained models have ‘learned’ the states but more learning is needed on states distributions over time so
that shortened or elongated/ added or missing states are used directly as predictors of how correct the
movements are. Using the HMM probabilistic method, qualitative insights concerning the flexibility and
stability of the human body during the entire recovery process and the developed model is extensive. As
such, it can be applied to a larger scope of physiotherapy rehabilitations where games and specified
movements as set by the physiotherapist and are perfectly suitable for the upper limbs as well. The
developed method is based on HMMs that are joint specific and can modified and applied in various
scenarios in any clinical setting and other different fields of motion, including dancing and sports. The
only requirement is the collection of new measures of kinematics for joint-based features relevant to the
specific exercises for every therapeutic program. The collected data is then used for training the HMMs
that will ensure the system to identify possible errors in movement execution. The developed model is
also capable of detecting non-compliances that can hinder a smooth recovery process. The CRehab
system enabled interaction between the patient and the therapist remotely and both were able to occupy
the viewpoints of each other in an augmented reality environment. The therapist were able to remotely
deliver instructions to a patient an d be able to observe the performance of the patients motor skills with
the system able to detect any errors and compensation by patients based on movements recorded from
healthy participants. The system worked well in fulfilling the objectives as captured in the research
proposal, but improvements can still be made, especially in improvements to the algorithm.
Conclusions
Recognizing the challenges faced in rehabilitation care across the world, where more people are growing
older and making bigger demands on medical care and the high associated costs, this paper sought to
develop a novel approach to overcome these challenges. There is a shortage specialized therapists and
geographical dispersion as well as the need for physical movement of patients and with existing
technologies in computing, remote rehabilitation is possible. The developed system (CRehab) is a simple
but novel system involving the use of a Kinect camera installed at the patients location and connected to
the therapist through a cloud based server and a Node.js database, also hosted in the cloud. The Niode.js
was used because of its numerous applications, including extensive libraries for the website, the data
base, and data capture and storage. The system was set up in an experimental setup with healthy
participants with pre-determined movements/ exercises performed in order to obtain training data that
would be used in training the innovative algorithm. The Node.js platform was able to generate an avatar
of the patient and using the AI algorithm, was able to capture perfect movements that were repeated over
time and averages obtained. This data was then used to assess correct movements and levels of
compensation by patients when performing the same exercises. The CRehab system enabled data capture
and monitoring in real time, with therapists able to give detailed instructions and monitor their execution.
The CRehab was able to detect anomalies and compensation in executing movements as well as humans.
The findings show the system is functional and can be applied in various fields including in sport and
performance. It can capture movements’ data, and using the HMMs, determine if movements were done
correctly or not; this data is stored in a cloud database through a cloud Node.js website and can be
reviewed by a therapist at any time. The physiotherapist can then determine the level of progress of a

patient and ensure smooth and quick recovery of patients at a low cost. The findings are limited to the fact
that HMMs predictions can have ‘noise’ resulting in some inaccuracies.
Recommendations
While the developed system is novel, some improvements can be made especially the algorithm which
can further be adapted to HSMMs (Hidden Semi-Markov Models) where sequences are modeled in the
context of distribution of durations. This can enable higher flexibility of transition probabilities than when
the HMMs are used.

References
Alagu Priya, A., & Lim, H. (2010). Hierarchical packet classification using a Bloom filter and rule-
priority tries. Computer Communications, 33(10), 1215-1226. doi: 10.1016/j.comcom.2010.03.009
Anton, D., Berges, I., Bermúdez, J., Goñi, A. and Illarramendi, A. (2018). A Telerehabilitation System for
the Selection, Evaluation and Remote Management of Therapies. Sensors, [online] 18(5), p.1459.
Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5982396/ [Accessed 20 Jun. 2019].
Banerji, S., Heng, J., Banerjee, A., Ponvignesh, P., Menezes, D., & Kumar, R. (2016). Delivering Remote
Rehabilitation at Home: An Integrated Physio-Neuro Approach to Effective and User Friendly Wearable
Devices. Converging Clinical And Engineering Research On Neurorehabilitation II, 1, 1097-1101. doi:
10.1007/978-3-319-46669-9_178
Capecci, M., Ceravolo, M., Ferracuti, F., Iarlori, S., Kyrki, V., & Monteriù, A. et al. (2018). A Hidden
Semi-Markov Model based approach for rehabilitation exercise assessment. Journal Of Biomedical
Informatics, 78, 1-11. doi: 10.1016/j.jbi.2017.12.012
Chebud, Y., & Melesse, A. (2011). Operational Prediction of Groundwater Fluctuation in South Florida
using Sequence Based Markovian Stochastic Model. Water Resources Management, 25(9), 2279-2294.
doi: 10.1007/s11269-011-9808-z
Chen, J., Jin, W., Zhang, X., Xu, W., Liu, X. and Ren, C. (2015). Telerehabilitation Approaches for
Stroke Patients: Systematic Review and Meta-analysis of Randomized Controlled Trials. Journal of
Stroke and Cerebrovascular Diseases, 24(12), pp.2660-2668.
De Angelis, L., & Paas, L. (2013). A dynamic analysis of stock markets using a hidden Markov model.
Journal Of Applied Statistics, 40(8), 1682-1700. doi: 10.1080/02664763.2013.793302
Dixon, P., Zhang, X., Domalain, M., Flores, A., & Lin, V. (2011). Physical Medicine and Rehabilitation
after Face Transplantation. The Know-How Of Face Transplantation, 1, 151-172. doi: 10.1007/978-0-
85729-253-7_14
Dong, B., & Lam, K. (2013). A real-time model predictive control for building heating and cooling
systems based on the occupancy behavior pattern detection and local weather forecasting. Building
Simulation, 7(1), 89-106. doi: 10.1007/s12273-013-0142-7
Esquivel, K., Nevala, E., Alamäki, A., Condell, J., Kelly, D., & Davies, R. et al. (2018). Remote
Rehabilitation: A Solution to Overloaded & Scarce Health Care Systems. Trends In Telemedicine & E-
Health, 1(1). Retrieved from https://crimsonpublishers.com/tteh/fulltext/TTEH.000503.php
Ghoniem, R., & Shaalan, K. (2017). A Novel Arabic Text-independent Speaker Verification System
based on Fuzzy Hidden Markov Model. Procedia Computer Science, 117, 274-286. doi:
10.1016/j.procs.2017.10.119
Han, B., Song, H., & Kim, J. (2011). Vestibular Rehabilitation Therapy: Review of Indications,
Mechanisms, and Key Exercises. Journal Of Clinical Neurology, 7(4), 184. doi: 10.3988/jcn.2011.7.4.184
Housley, S., Fitzgerald, K. and Butler, A. (2018). Telerehabilitation Robotics: Overview of approaches
and clinical outcomes. Rehabilitation Robotics, pp.333-346.

Kao, C., & Fahn, C. (2011). A Human-Machine Interaction Technique: Hand Gesture Recognition Based
on Hidden Markov Models with Trajectory of Hand Motion. Procedia Engineering, 15, 3739-3743. doi:
10.1016/j.proeng.2011.08.700
Kumar, K., Aggarwal, R., & Jain, A. (2012). A Hindi speech recognition system for connected words
using HTK. International Journal Of Computational Systems Engineering, 1(1), 25. doi:
10.1504/ijcsyse.2012.044740
Kumar, S. and Cohn, E. (2013). Telerehabilitation. 1st ed. London: Springer, pp.2-3.
Mahé, J., Vercauteren, T., Rosa, B., & Dauguet, J. (2013). A Viterbi Approach to Topology Inference for
Large Scale Endomicroscopy Video Mosaicing. Advanced Information Systems Engineering, 1, 404-411.
doi: 10.1007/978-3-642-40811-3_51
Peretti, A., Amenta, F., Tayebati, S., Nittari, G. and Mahdi, S. (2017). Telerehabilitation: Review of the
State-of-the-Art and Areas of Application. JMIR Rehabilitation and Assistive Technologies, [online] 4(2),
p.e7. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5544892/ [Accessed 20 Jun. 2019].
Rangpariya, N. (2018). 11 Reasons to Use Node.Js For Web Application Development - Medium.
Retrieved from https://medium.com/swlh/11-reasons-to-use-node-js-for-web-application-development-
5b2ed5877ff0
Reddivari, H., Yang, C., Ju, Z., Liang, P., Li, Z., & Xu, B. (2014). Teleoperation control of Baxter robot
using body motion tracking. 2014 International Conference On Multisensor Fusion And Information
Integration For Intelligent Systems (MFI), 1. doi: 10.1109/mfi.2014.6997722
Sharma, A., Golubchik, L., & Govindan, R. (2010). Sensor faults. ACM Transactions On Sensor
Networks, 6(3), 1-39. doi: 10.1145/1754414.1754419
Sarfo, F., Ulasavets, U., Opare-Sem, O. and Ovbiagele, B. (2018). Tele-Rehabilitation after Stroke: An
Updated Systematic Review of the Literature. Journal of Stroke and Cerebrovascular Diseases, 27(9),
pp.2306-2318.
Simonsen, D., Hansen, J., Spaich, E. and Andersen, O. (2014). Kinect-Based Tele-rehabilitation System
for Hand Function. Biosystems & Biorobotics, 7, pp.871-872.
Tee, J. (2015). Simplified cloud-based application development with Node.js. Retrieved from
https://www.theserverside.com/feature/Simplified-cloud-based-application-development-with-Nodejs
Tey, C., An, J., & Chung, W. (2017). A Novel Remote Rehabilitation System with the Fusion of
Noninvasive Wearable Device and Motion Sensing for Pulmonary Patients. Computational And
Mathematical Methods In Medicine, 2017, 1-8. doi: 10.1155/2017/5823740
Wang, X., Xia, M., Cai, H., Gao, Y., & Cattani, C. (2012). Hidden-Markov-Models-Based Dynamic
Hand Gesture Recognition. Mathematical Problems In Engineering, 2012, 1-11. doi:
10.1155/2012/986134
Yu, S. (2010). Hidden semi-Markov models. Artificial Intelligence, 174(2), 215-243. doi:
10.1016/j.artint.2009.11.011
Zia Uddin, M., Kim, T., & Jeong Tai Kim. (2010). Video-based Indoor Human Gait Recognition Using
Depth Imaging and Hidden Markov Model: A Smart System for Smart Home. Indoor And Built
Environment, 20(1), 120-128. doi: 10.1177/1420326x10391140