MSc Dissertation: Human Activity Recognition from Wearable Sensors

This dissertation, submitted to the University of Greenwich for an MSc in Data Warehousing & Data Mining, investigates human activity recognition using wearable sensors. The research explores the potential of ubiquitous computing and the challenges of acquiring annotated data for training machine learning algorithms. It delves into the applications of wearable sensors in assisted living, healthcare, and other fields, reviewing existing literature on activity recognition, modeling, and machine learning techniques. The dissertation analyzes datasets, features, and clustering methods, presenting results and findings related to activity recognition. It concludes with a discussion of the study's limitations, recommendations, and potential future research directions, including personal evaluations and references.

A dissertation submitted to the University of Greenwich
in partial fulfilment of the requirements for the Degree of
Master of Science
in
Data Warehousing & Data Mining
Human Activity Recognition Data from Wearable
Sensor
Name:
Student ID:
Supervisor:
Submission Date:
Word count: 12,713

Human Activity Recognition Data from Wearable Sensor
Computing & Mathematical Sciences, University of Greenwich, 30 Park Row, Greenwich, UK.
(Submitted Date)
ABSTRACT
The debate revolving on how we should live and interact freely when there are many small
computer systems that can be hidden so that we cannot see them. Ubiquitous computing has the
vision that with time computers will be part of our lives such that we cannot do anything without
them. One primary advantage of wearable sensors is the chance to view the world from the
perspective of the first-person all the time without having to setup external infrastructure. The
existing applications can be improved by wearable activity recognition abilities ranging from
assisted living and personal healthcare to arts, entertainment, and even industrial applications.
Most part of the work on activity recognition has been carried out by individuals because of the
challenges associated with acquiring annotated data for training. This has caused several issues
including the fact that the available data is insufficient to make reliable approximations of how
better the approach will conclude to various users or settings (how relevant is the approach).The
wearable sensor technology was first identified in the early 1990s because of hardware evolution
making it possible for computing equipment light enough to support integration of sensor
technology in mobile systems that can be worn by one person for a long time. Initially, the
prototypes were a long way from what we have today and were relatively bulky but promised an
exciting future in sensor technology to make computer perceive the life of a human being from
first-person.
One of the main objectives in context-aware computing and activity recognition is to facilitate
development of new technologies and applications for the elderly and other patients. The
declining rates of fertility and the extended life expectancy are increasing the number of the aged
population in our communities across the world bring about numerous challenges to the current
healthcare systems. Many expect that the activity recognition and wearable sensors will address
these challenges by enabling the aged in the society to live independently without having the
need of caregivers around.
Keywords: Wearable Sensors; Ubiquitous Computing; Activity Recognition; Context-
aware Computing

Acknowledgements
I would especially like to thank XXXXXXXXXX for agreeing to be my supervisor
and for his consistent advice, feedback, guidance and support throughout the
lifecycle of this Msc data warehouse project.
I want to thank both XXXXXXXXXXXXX and XXXXXXXXX for agreeing to
have the project demonstration on the schedule day.

Table of Contents
ABSTRACT.....................................................................................................................................2
Acknowledgements..........................................................................................................................4
Introduction......................................................................................................................................6
Challenges....................................................................................................................................6
The need for less supervision...................................................................................................7
High-level and Long-term activities:.......................................................................................7
Literature Review.............................................................................................................................8
Introduction..................................................................................................................................8
Applications.................................................................................................................................9
Sensors.......................................................................................................................................11
Activities....................................................................................................................................12
Modelling and Recognizing Activities.......................................................................................13
Unsupervised Learning of Activities.........................................................................................15
Multiple Eigenspaces.............................................................................................................15
Problem Description...............................................................................................................16
Overview of the Multiple Eigenspace Algorithm..................................................................17
Initialization...........................................................................................................................18
Eigenspace Growing..............................................................................................................18
Eigenspace Selection..............................................................................................................19
Extension to Multiple Time Scales........................................................................................20
Modified Eigenspace Selection..................................................................................................21
Recognizing daily Routines.......................................................................................................21
Using Topic Models to Model Daily Routines......................................................................23
Generative Versus Discriminative Models............................................................................25
Analysis of the System...................................................................................................................27
Introduction................................................................................................................................27
Dataset........................................................................................................................................28
Feature Analysis.........................................................................................................................28
Commonly used Features from Accelerometers....................................................................29
Features used for this.............................................................................................................29
Clustering...............................................................................................................................29
Results....................................................................................................................................30
Recognition............................................................................................................................33
Findings..................................................................................................................................34
Summary and Conclusion..........................................................................................................37
Conclusion......................................................................................................................................39
Overview....................................................................................................................................39
Summary of Findings.................................................................................................................40
Recommendations......................................................................................................................40
Future Work...............................................................................................................................41
Personal Evaluation....................................................................................................................42
Reference List................................................................................................................................43

Introduction
The pervasiveness of computers is growing and they are being integrated in cameras, mobile
phones, cars, buildings, music players, clothing and many other things that lack the resemblance
of the traditional desktop computers with a mouse, keyboard, and a monitor. The debate
revolving on how we should live and interact freely when there are many small computer
systems that can be hidden so that we cannot see them. Ubiquitous computing has the vision that
with time computers will be part of our lives such that we cannot do anything without them
(Baghiyan, 2011). Basically, the number of computers will increase and will enhance our way of
life. However, we may not be aware of them as we will be concentrating more on the tasks that
we are doing rather than technology around us. As such, the ubiquitous computing has been
challenged with the task to identify new ways of how the users will interact and use the new
generation of computers.
By reducing the explicit interaction, the designers will be able to do away with the big computers
and increase the implicit interaction. A good example of implicit interaction is a case like using
motion sensors to turn on or off the lights rather than using a switch (Bulling, Blanke and
Schiele, 2014). Also motion sensors could be used to register students as they walk into class
rather than signing a manual register. Lack of standard acceptable protocols to carry out
experiments and present the outcomes is one of the main concerns of carrying out study on
human activity recognition data on wearable sensors. Basically, many issues have not been
addressed including evaluations metrics, generating datasets from raw signals, and the datasets
that are challenging by the existing research work.
This paper will explore a section of context-wear technology. First, the focus will be on context
of the user which is a very crucial part. Secondly, we concentrate on combining machine learning
methods with the use of wearable sensors to learn, record, recognize, and model the activities of
the user. One primary advantage of wearable sensors is the chance to view the world from the
perspective of the first-person all the time without having to setup external infrastructure. The
existing applications can be improved by wearable activity recognition abilities ranging from
assisted living and personal healthcare to arts, entertainment, and even industrial applications
(Dunne et al., 2016). The next session will describe the main challenges that the researchers often
face when handling the concept of human activity recognition data from wearable sensors.
Challenges
This section will attempt to describe essential research challenges in human activity recognition.
They include:
 Exploration of high-level and long-term activities

 The need for less constant supervision
These challenges are further described below in detail, and finally give a summary of other
challenged related to the topic.
The need for less supervision
To train a machine learning algorithm, nearly all approached in human activity recognition
depend on glossed or annotated records of the data associated with the specific activity. It is quite
difficult, erroneous, time-consuming, tedious, and sometimes nearly impossible to acquire such
data with ground truth or enough annotations and this poses a serious challenge to the
enhancement in this field. Most part of the work on activity recognition has been carried out by
individuals because of the challenges associated with acquiring annotated data for training (Eagle
and Pentland, 2016). This has caused several issues including the fact that the available data is
insufficient to make reliable approximations of how better the approach will conclude to various
users or settings (how relevant is the approach).
It would be much easier to exploit sparsely, or no-annotated data since they can be acquired more
easily, and this will further reduce the problems and issues in the activity recognition field.
Furthermore, there exist several big databases which contains data that is uncategorized such as
location traces, activity diaries, or even cell phone logs. It is quite expensive to carry out
annotation on such big databases, and it can also introduce unexpected data biasness.
High-level and Long-term activities:
Several of the tasks in activity recognition concentrates on quite shorter activities that can be
measured in seconds or minutes, rather than days or hours (Hamid et al., 2015). Activity
exploration on a bigger time scale is both challenging and interesting for several reasons: high-
levels time scale activity recognition has unclear understanding in several aspects, and long-term
activities have not been researched on extensively.

Literature Review
Introduction
This chapter will provide related information current trends, state of the art, and issues in human
activity recognition data from wearable sensor. One of the major expectations of wearable
computing is the ability to facilitate development of personal applications that can react and
adapt to the current user context (Hastie, Tibshirani and Friedman, 2011). In this scenario,
context refers to any type of information related to the current state of the user or his/her
surrounding. This paper focuses on human activity, that is mostly regarded as one of the most
essential part of the warble sensor technology. The remaining sections of this chapter will focus
on the applications of wearables sensors to refer to the current user activity. The chapter is
subdivided into different sections including application areas of wearable sensor technology, the
various technologies that have been employed in activity recognition, type of human activities,
and finally machine learning techniques that have been used fir activity recognition.
The wearable sensor technology was first identified in the early 1990s because of hardware
evolution making it possible for computing equipment light enough to support integration of
sensor technology in mobile systems that can be worn by one person for a long time. Initially, the
prototypes were a long way from what we have today and were relatively bulky but promised an
exciting future in sensor technology to make computer perceive the life of a human being from
first-person. Early research and study were focused on keyboard, text, and traditional
applications and gradually adopted new techniques of interaction and input using microphones or
cameras and adding other user context details topic of conversation, current location, or the
identity of the person the user is conversing with for later retrieval.
The medical community has adopted this technology to measure the physical activity of the
patients through the utilization of objective technology.
The initial uses of wearable sensory system were to estimate global measures like the amount of
oxygen used by a particular subject while carrying out the various activities and the total energy.
As technology advances, both emergence of machine learning and advancement in hardware
technology, more systems have been designed that uses inertial sensors to recognize and separate
different human activities and their unsolicited advantages in the new paradigm of wearable
computing.
Applications
This section will discuss the various application area of human activity recognition technology in
mobile and wearable environments. First, we will focus on assisted living and healthcare which
represents an essential category of uses. Apart from applications on the healthcare industry, there

are other applications such as gaming, entertainment, and industrial applications which will be
discusses later in the chapter.
One of the main objectives in context-aware computing and activity recognition is to facilitate
development of new technologies and applications for the elderly and other patients (Huynh et
al., 2016). The declining rates of fertility and the extended life expectancy are increasing the
number of the aged population in our communities across the world bring about numerous
challenges to the current healthcare systems. Many expect that the activity recognition and
wearable sensors will address these challenges by enabling the aged in the society to live
independently without having the need of caregivers around. One way of utilizing such
technology in healthcare is designing a sensory system that will have the ability to use human
activity to identify any possible dangerous situation in the life of an elderly person and
automatically call for help.
Another objective of this technology is to prevent the occurrence of severe medical conditions or
age-related diseases before they occur because the wearable sensor applications use long-term
monitoring to discover any unusual patterns or changes in the daily routine of a person as any
changes may indicate early symptoms of a particular disease (Junker et al., 2013). Applications
that have shown the abilities to record and monitor physiological parameters and daily activities
of a person have proven to be very valuable to care-givers and physicians to assess the physical
wellness of a person.
Thirdly, the medical sector aims at employing context-information to foster for a healthier and
more active lifestyle, or to support the disabled or elderly people actively in carrying out their
everyday tasks. The fluctuating signals in mobile phones can be used to summarize and
approximate the levels of activities of a person to encourage and motivate contemplation of
everyday activities (Kunze and Lukowicz, 2014). Wearable sensors can be used to discover
particular activities and symbolize them as distinct types of flowers in the display of a mobile
phone. Location and activity data collected by wearable sensors can be combined to imply
impulsive exercises like recording that a particular user walked to the next eat out instead of
queuing in the current one (Kańtoch, 2018).
Wearable Optical sensors can be used in monitoring the posture of the spinal code to prevent or
detect any back problems due to poor body postures. A similar system has been proposed by
Leonardis, Bischof and Maver (2012) for the mentally challenged to discover any anomalies,
assess user location, and help in navigation to notify the user where to alight. Stork, Yom-Tov
and Duda (2014) have put a lot of effort to develop context-aware reminders for the people
suffering from dementia to offer the relevant aid when needed. A mobile system to recognize

sign languages has been proposed by Vogel and Schiele (2016) that uses accelerometers and
wearable cameras.
In an industrial setting, wearable sensor technology has been identified to have a great potential
to support employees at performing their tasks and help then increase throughput and avoid
mistakes. Wearable sensors help the employees in processes such as collection of data, data
access, and communication and this has been around since the 1990s for industrial enterprises
such as Xybernaut. Wahle et al. 2016) have mention that the early users of such expensive
systems were those involved in complex and expensive operations such as manufacturing of
airplanes, ships, and telecommunication networks. Today, numerous researches are being carried
out by scientists, engineers, and IT professionals to discover the next generation technologies that
can be applied in an industrial setting which and take advantage of the benefits they come with
them by include enhancement of multimodal sensing abilities of a wearable device by gathering
the context data about the current activities of the user. Zahid, Mittal and Joshi (2017), gave a
scenario to evaluate the use of wearable technology in emergency responses, hospital
environment, car production, and aircraft maintenance. In these cases, activity recognition and
wearable technology are employed to offer hands-free and interactive data access such as
assisting in training of new employees, patient records, electronic manuals, help in
communication and navigation, and generate summaries of the activities performed by the user.
According to Baghiyan (2011), data collected from wearable sensors can be used to track
employee activities in a car production plat such as generate real-time responses to the employee
concerning the next steps in the car assembly process and issuing warnings if the steps are not
done correctly. Bulling, Blanke and Schiele (2014) integrated the data collected from
accelerometers and microphones to monitor/track the activities in a wood shop such as
hammering or sawing. Dunne et al. (2016) have reported on wearable sensor project that targets
at aiding the hospital employed in their everyday activities like showing the health conditions of
the nearby patients and giving precedence to the to the patients based on their health condition
and enhancing the communication between the nurses and the patients. Dunne et al. (2016) have
proposed a wearable sensor in the form of a bracelet that will be worn by nurses and the LED
will change the color based on the current condition of the patient health.
Eagle and Pentland (2016) have utilized activity recognition and wearable sensors to evaluate the
social traits in organizations, thus improving the techniques such as context aware computing for
potentially bigger enterprise network. For example, Hamid et al. (2015) describes that wearable
sensors can be used to analyze one-on-one conversation in order to find experts within the
institution, map social network, and aid in creating a project team. The context data collected

from wearable sensors cane be used to determine the common constructions in the everyday
routines of the user.
In games and entertainments, adopting activity recognition wearable systems are attractive for
use in performing arts such as, enabling dance artist to intensify their performance by using
interactive media that are in line with their body movements. Hastie, Tibshirani and Friedman
(2011) describes such systems as wearable inertial sensors integrated with machine learning
methods to visualize, record, and classify the movement of dancers. Adoption of wearable sensor
systems may be at a higher rate in gaming and entertainment industries since classification
accuracy is not essential as in manufacturing and healthcare system where accuracy is
paramount. Huynh et al. (2016) gives an example of an application used in gaming world where
clamp attached to the body with the ability to sense body motion are used to control video games
or discover moves to regulate martial arts games. Motion-sensing systems for game control has
become popular and has been inspired by applications like Nintendo’s Wii dais which have
familiarized many users to the concepts that commenced in the activity recognition computing
and are currently being implemented widely by independent developers and companies.
There are other possible application areas that human activity recognition data from wearable
sensors can be used. For example, in educational context to learn foreign vocabulary, context-
aware advertising, and can also be used to recognize the activities of a soldier to generate action
reports automatically.
Sensors
The kinds of sensors used in human activity recognition varies from sensors with constant output
such as accelerometers to simple sensors with discrete output like ball switches to more complex
sensor techniques like computer visioning and audio processing. Similarly, there exist other kinds
of sensors have been proposed by Junker et al. (2013) and Dunne et al. (2016) which include
physiological sensors, foam pressure sensors for measuring the rate of respiration, body
temperature sensors, skin conductivity sensors, and electrocardiographs.
One of the popular kinds of sensor is the accelerometers used for activity recognition together
with wearable sensors. These sensors are fairly insensitive to the conditions of the environment,
require less processing power, memory, and energy, they are cheap and small in size, and provide
good results in recognizing physical activities. Many users have regarded them as less invasive as
compared to cameras and microphones.
Improvement of the performance of activity recognition can be done based on the type of activity
using similar kind of sensor at different parts of the body. Using a combination of more than one

complementary kind of sensor helps in recognizing activities. For instance, combining audio-data
and motion-data, proximity-data and motion-data, or location-data and motion-data.
Adaskevicius (2014) has taken the same approach to gather data from environmental infrared
sensors and wearable accelerometers. The data that Adaskevicius used in their methodology
forms a part of the bigger dataset that Kańtoch (2018) introduced by combining a range of
environmental sensors and wearable sensors for discovering usage of objects (like motion sensors
for detecting cabinet, windows, or doors usage) and environmental conditions (like humidity,
temperature, light).
Activities
There are so many types of sensors and envisioned application to choose from and it is not a
surprise the number of human activities that experts have tried to recognize using wearable
sensors is endless. Section 2.2 has already identified a number of different activities. Human
activities such jogging, sitting, standing, and walking are natural physical activities that present
themselves for recognition with motion sensors or inertial sensors because they are defined
clearly by the relative and motion position of the body parts of the user. 3D and 2D acceleration
information has been employed fruitfully for recognizing such activities (Kunze and Lukowicz,
2014).
Activities of Daily Living (ADLs) are the essential category of activities in assisted living and
healthcare. Leonardis, Bischof and Maver (2012) initially proposed that these activities have
developed over time and are now being used by caregivers and physicians as a standard measure
to assess the physical wellness of those in need of assisted living and the elderly patients. Some
of the core set of activities of daily living include toileting, bathing, feeding, dressing,
continence, and transferring. These activities are supplemented by Instrumental Activities of
Daily Living (IADLs) which is made up of activities like food preparation, handling finances,
taking medications, using a phone, doing laundry, housekeeping, shopping, and transportation.
Bulling et al. (2018) have demonstrated recognition of such specific sets of IADLs and ADLs
through recognizing activities like taking a shower, washing dishes, cleaning the windows,
vacuuming, ironing, dusting, or making tea. It is still a challenge to recognize the whole set of
IADLs and ADLs using sensors because some of the activities like handling finances are not
defined clearly and some are difficult to detect for instance, continence. Moreover, the physicians
are very interested both the fact the that patient has done an activity and how well it was done.
Aside from the activities identified, there are other like sporting activities that can be recognized