MSc Dissertation Interim Report: Human Activity Recognition Analysis

This interim report details a student's progress on an MSc dissertation focused on human activity recognition (HAR) using wearable sensors. The report is divided into two parts: Part A provides a progress update, including the current situation, problem areas, and planned future work. It discusses the pervasiveness of computing, the need for implicit interaction, and the challenges in HAR, such as lack of standardized protocols and the need for less supervision in training machine learning algorithms. The report also highlights the objectives of the research, including facilitating new technologies for the elderly and preventing severe medical conditions. Part B presents a literature review, covering the applications of wearable sensors in healthcare, gaming, and industrial settings. The report explores the evolution of wearable computing, the various technologies used in activity recognition, and machine learning techniques. The report also reviews challenges like the need for less constant supervision and the exploration of high-level and long-term activities.

An interim report for a dissertation that will be submitted in partial fulfilment of a
University of Greenwich Masters Degree
Human Activity Recognition Data from Wearable Sensor
Name:
Student ID:
Programme of Study:
Date Proposal Submitted:
Project Hand In Date:
Supervisor:
MSc Project Proposal Page 1

Table of Contents
Part A – Progress Report....................................................................................................3
1. Current situation........................................................................................................3
2. Problem Areas............................................................................................................4
The need for less supervision.....................................................................................5
High-level and Long-term activities:.........................................................................5
Part B – Project Report to Date..........................................................................................5
Literature Review...............................................................................................................5
Introduction....................................................................................................................5
Applications...................................................................................................................6
Sensors...........................................................................................................................9
Activities......................................................................................................................10
Modelling and Recognizing Activities.........................................................................11
Reference List..................................................................................................................14
MSc Project Proposal Page 2

Part A – Progress Report
1. Current situation
The pervasiveness of computers is growing and they are being integrated in cameras,
mobile phones, cars, buildings, music players, clothing and many other things that lack
the resemblance of the traditional desktop computers with a mouse, keyboard, and a
monitor. The debate revolving on how we should live and interact freely when there are
many small computer systems that can be hidden so that we cannot see them. Ubiquitous
computing has the vision that with time computers will be part of our lives such that we
cannot do anything without them (Baghiyan, 2011). Basically, the number of computers
will increase and will enhance our way of life. However, we may not be aware of them
as we will be concentrating more on the tasks that we are doing rather than technology
around us. As such, the ubiquitous computing has been challenged with the task to
identify new ways of how the users will interact and use the new generation of
computers.
By reducing the explicit interaction, the designers will be able to do away with the big
computers and increase the implicit interaction. A good example of implicit interaction
is a case like using motion sensors to turn on or off the lights rather than using a switch
(Bulling, Blanke and Schiele, 2014). Also motion sensors could be used to register
students as they walk into class rather than signing a manual register. Lack of standard
acceptable protocols to carry out experiments and present the outcomes is one of the
main concerns of carrying out study on human activity recognition data on wearable
sensors. Basically, many issues have not been addressed including evaluations metrics,
generating datasets from raw signals, and the datasets that are challenging by the
existing research work.
This paper will explore a section of context-wear technology. First, the focus will be on
context of the user which is a very crucial part. Secondly, we concentrate on combining
machine learning methods with the use of wearable sensors to learn, record, recognize,
and model the activities of the user. One primary advantage of wearable sensors is the
chance to view the world from the perspective of the first-person all the time without
having to setup external infrastructure. The existing applications can be improved by
wearable activity recognition abilities ranging from assisted living and personal
healthcare to arts, entertainment, and even industrial applications (Dunne et al., 2016).
The next session will describe the main challenges that the researchers often face when
handling the concept of human activity recognition data from wearable sensors.
MSc Project Proposal Page 3

Objectives
One of the main objectives in context-aware computing and activity recognition is to
facilitate development of new technologies and applications for the elderly and other
patients (Huynh et al., 2016). The declining rates of fertility and the extended life
expectancy are increasing the number of the aged population in our communities across
the world bring about numerous challenges to the current healthcare systems. Many
expect that the activity recognition and wearable sensors will address these challenges
by enabling the aged in the society to live independently without having the need of
caregivers around. One way of utilizing such technology in healthcare is designing a
sensory system that will have the ability to use human activity to identify any possible
dangerous situation in the life of an elderly person and automatically call for help.
Another objective of this technology is to prevent the occurrence of severe medical
conditions or age-related diseases before they occur because the wearable sensor
applications use long-term monitoring to discover any unusual patterns or changes in the
daily routine of a person as any changes may indicate early symptoms of a particular
disease (Junker et al., 2013). Applications that have shown the abilities to record and
monitor physiological parameters and daily activities of a person have proven to be very
valuable to care-givers and physicians to assess the physical wellness of a person.
Thirdly, the medical sector aims at employing context-information to foster for a
healthier and more active lifestyle, or to support the disabled or elderly people actively
in carrying out their everyday tasks. The fluctuating signals in mobile phones can be
used to summarize and approximate the levels of activities of a person to encourage and
motivate contemplation of everyday activities (Kunze and Lukowicz, 2014). Wearable
sensors can be used to discover particular activities and symbolize them as distinct types
of flowers in the display of a mobile phone. Location and activity data collected by
wearable sensors can be combined to imply impulsive exercises like recording that a
particular user walked to the next eat out instead of queuing in the current one (Kańtoch,
2018).
2. Problem Areas
This section will attempt to describe essential research challenges in human activity
recognition. They include:
 Exploration of high-level and long-term activities
 The need for less constant supervision
MSc Project Proposal Page 4

These challenges are further described below in detail, and finally give a summary of
other challenged related to the topic.
The need for less supervision
To train a machine learning algorithm, nearly all approached in human activity
recognition depend on glossed or annotated records of the data associated with the
specific activity. It is quite difficult, erroneous, time-consuming, tedious, and sometimes
nearly impossible to acquire such data with ground truth or enough annotations and this
poses a serious challenge to the enhancement in this field. Most part of the work on
activity recognition has been carried out by individuals because of the challenges
associated with acquiring annotated data for training (Eagle and Pentland, 2016). This
has caused several issues including the fact that the available data is insufficient to make
reliable approximations of how better the approach will conclude to various users or
settings (how relevant is the approach).
It would be much easier to exploit sparsely, or no-annotated data since they can be
acquired more easily, and this will further reduce the problems and issues in the activity
recognition field. Furthermore, there exist several big databases which contains data that
is uncategorized such as location traces, activity diaries, or even cell phone logs. It is
quite expensive to carry out annotation on such big databases, and it can also introduce
unexpected data biasness.
High-level and Long-term activities:
Several of the tasks in activity recognition concentrates on quite shorter activities that
can be measured in seconds or minutes, rather than days or hours (Hamid et al., 2015).
Activity exploration on a bigger time scale is both challenging and interesting for
several reasons: high-levels time scale activity recognition has unclear understanding in
several aspects, and long-term activities have not been researched on extensively.
Part B – Project Report to Date
Literature Review
Introduction
This chapter will provide related information current trends, state of the art, and issues in
human activity recognition data from wearable sensor. One of the major expectations of
wearable computing is the ability to facilitate development of personal applications that
can react and adapt to the current user context (Hastie, Tibshirani and Friedman, 2011).
MSc Project Proposal Page 5

In this scenario, context refers to any type of information related to the current state of
the user or his/her surrounding. This paper focuses on human activity, that is mostly
regarded as one of the most essential part of the warble sensor technology. The
remaining sections of this chapter will focus on the applications of wearables sensors to
refer to the current user activity. The chapter is subdivided into different sections
including application areas of wearable sensor technology, the various technologies that
have been employed in activity recognition, type of human activities, and finally
machine learning techniques that have been used fir activity recognition.
The wearable sensor technology was first identified in the early 1990s because of
hardware evolution making it possible for computing equipment light enough to support
integration of sensor technology in mobile systems that can be worn by one person for a
long time. Initially, the prototypes were a long way from what we have today and were
relatively bulky but promised an exciting future in sensor technology to make computer
perceive the life of a human being from first-person. Early research and study were
focused on keyboard, text, and traditional applications and gradually adopted new
techniques of interaction and input using microphones or cameras and adding other user
context details topic of conversation, current location, or the identity of the person the
user is conversing with for later retrieval.
The medical community has adopted this technology to measure the physical activity of
the patients through the utilization of objective technology.
The initial uses of wearable sensory system were to estimate global measures like the
amount of oxygen used by a particular subject while carrying out the various activities
and the total energy. As technology advances, both emergence of machine learning and
advancement in hardware technology, more systems have been designed that uses
inertial sensors to recognize and separate different human activities and their unsolicited
advantages in the new paradigm of wearable computing.
Applications
This section will discuss the various application area of human activity recognition
technology in mobile and wearable environments. First, we will focus on assisted living
and healthcare which represents an essential category of uses. Apart from applications
on the healthcare industry, there are other applications such as gaming, entertainment,
and industrial applications which will be discusses later in the chapter.
MSc Project Proposal Page 6

One of the main objectives in context-aware computing and activity recognition is to
facilitate development of new technologies and applications for the elderly and other
patients (Huynh et al., 2016). The declining rates of fertility and the extended life
expectancy are increasing the number of the aged population in our communities across
the world bring about numerous challenges to the current healthcare systems. Many
expect that the activity recognition and wearable sensors will address these challenges
by enabling the aged in the society to live independently without having the need of
caregivers around. One way of utilizing such technology in healthcare is designing a
sensory system that will have the ability to use human activity to identify any possible
dangerous situation in the life of an elderly person and automatically call for help.
Another objective of this technology is to prevent the occurrence of severe medical
conditions or age-related diseases before they occur because the wearable sensor
applications use long-term monitoring to discover any unusual patterns or changes in the
daily routine of a person as any changes may indicate early symptoms of a particular
disease (Junker et al., 2013). Applications that have shown the abilities to record and
monitor physiological parameters and daily activities of a person have proven to be very
valuable to care-givers and physicians to assess the physical wellness of a person.
Thirdly, the medical sector aims at employing context-information to foster for a
healthier and more active lifestyle, or to support the disabled or elderly people actively
in carrying out their everyday tasks. The fluctuating signals in mobile phones can be
used to summarize and approximate the levels of activities of a person to encourage and
motivate contemplation of everyday activities (Kunze and Lukowicz, 2014). Wearable
sensors can be used to discover particular activities and symbolize them as distinct types
of flowers in the display of a mobile phone. Location and activity data collected by
wearable sensors can be combined to imply impulsive exercises like recording that a
particular user walked to the next eat out instead of queuing in the current one (Kańtoch,
2018).
Wearable Optical sensors can be used in monitoring the posture of the spinal code to
prevent or detect any back problems due to poor body postures. A similar system has
been proposed by Leonardis, Bischof and Maver (2012) for the mentally challenged to
discover any anomalies, assess user location, and help in navigation to notify the user
where to alight. Stork, Yom-Tov and Duda (2014) have put a lot of effort to develop
context-aware reminders for the people suffering from dementia to offer the relevant aid
MSc Project Proposal Page 7

when needed. A mobile system to recognize sign languages has been proposed by Vogel
and Schiele (2016) that uses accelerometers and wearable cameras.
In an industrial setting, wearable sensor technology has been identified to have a great
potential to support employees at performing their tasks and help then increase
throughput and avoid mistakes. Wearable sensors help the employees in processes such
as collection of data, data access, and communication and this has been around since the
1990s for industrial enterprises such as Xybernaut. Wahle et al. 2016) have mention that
the early users of such expensive systems were those involved in complex and expensive
operations such as manufacturing of airplanes, ships, and telecommunication networks.
Today, numerous researches are being carried out by scientists, engineers, and IT
professionals to discover the next generation technologies that can be applied in an
industrial setting which and take advantage of the benefits they come with them by
include enhancement of multimodal sensing abilities of a wearable device by gathering
the context data about the current activities of the user. Zahid, Mittal and Joshi (2017),
gave a scenario to evaluate the use of wearable technology in emergency responses,
hospital environment, car production, and aircraft maintenance. In these cases, activity
recognition and wearable technology are employed to offer hands-free and interactive
data access such as assisting in training of new employees, patient records, electronic
manuals, help in communication and navigation, and generate summaries of the
activities performed by the user.
According to Baghiyan (2011), data collected from wearable sensors can be used to
track employee activities in a car production plat such as generate real-time responses to
the employee concerning the next steps in the car assembly process and issuing
warnings if the steps are not done correctly. Bulling, Blanke and Schiele (2014)
integrated the data collected from accelerometers and microphones to monitor/track the
activities in a wood shop such as hammering or sawing. Dunne et al. (2016) have
reported on wearable sensor project that targets at aiding the hospital employed in their
everyday activities like showing the health conditions of the nearby patients and giving
precedence to the to the patients based on their health condition and enhancing the
communication between the nurses and the patients. Dunne et al. (2016) have proposed
a wearable sensor in the form of a bracelet that will be worn by nurses and the LED will
change the color based on the current condition of the patient health.
Eagle and Pentland (2016) have utilized activity recognition and wearable sensors to
evaluate the social traits in organizations, thus improving the techniques such as context
MSc Project Proposal Page 8

aware computing for potentially bigger enterprise network. For example, Hamid et al.
(2015) describes that wearable sensors can be used to analyze one-on-one conversation
in order to find experts within the institution, map social network, and aid in creating a
project team. The context data collected from wearable sensors cane be used to
determine the common constructions in the everyday routines of the user.
In games and entertainments, adopting activity recognition wearable systems are
attractive for use in performing arts such as, enabling dance artist to intensify their
performance by using interactive media that are in line with their body movements.
Hastie, Tibshirani and Friedman (2011) describes such systems as wearable inertial
sensors integrated with machine learning methods to visualize, record, and classify the
movement of dancers. Adoption of wearable sensor systems may be at a higher rate in
gaming and entertainment industries since classification accuracy is not essential as in
manufacturing and healthcare system where accuracy is paramount. Huynh et al. (2016)
gives an example of an application used in gaming world where clamp attached to the
body with the ability to sense body motion are used to control video games or discover
moves to regulate martial arts games. Motion-sensing systems for game control has
become popular and has been inspired by applications like Nintendo’s Wii dais which
have familiarized many users to the concepts that commenced in the activity recognition
computing and are currently being implemented widely by independent developers and
companies.
There are other possible application areas that human activity recognition data from
wearable sensors can be used. For example, in educational context to learn foreign
vocabulary, context-aware advertising, and can also be used to recognize the activities of
a soldier to generate action reports automatically.
Sensors
The kinds of sensors used in human activity recognition varies from sensors with
constant output such as accelerometers to simple sensors with discrete output like ball
switches to more complex sensor techniques like computer visioning and audio
processing. Similarly, there exist other kinds of sensors have been proposed by Junker et
al. (2013) and Dunne et al. (2016) which include physiological sensors, foam pressure
sensors for measuring the rate of respiration, body temperature sensors, skin
conductivity sensors, and electrocardiographs.
MSc Project Proposal Page 9

One of the popular kinds of sensor is the accelerometers used for activity recognition
together with wearable sensors. These sensors are fairly insensitive to the conditions of
the environment, require less processing power, memory, and energy, they are cheap and
small in size, and provide good results in recognizing physical activities. Many users
have regarded them as less invasive as compared to cameras and microphones.
Improvement of the performance of activity recognition can be done based on the type
of activity using similar kind of sensor at different parts of the body. Using a
combination of more than one complementary kind of sensor helps in recognizing
activities. For instance, combining audio-data and motion-data, proximity-data and
motion-data, or location-data and motion-data. Adaskevicius (2014) has taken the same
approach to gather data from environmental infrared sensors and wearable
accelerometers. The data that Adaskevicius used in their methodology forms a part of
the bigger dataset that Kańtoch (2018) introduced by combining a range of
environmental sensors and wearable sensors for discovering usage of objects (like
motion sensors for detecting cabinet, windows, or doors usage) and environmental
conditions (like humidity, temperature, light).
Activities
There are so many types of sensors and envisioned application to choose from and it is
not a surprise the number of human activities that experts have tried to recognize using
wearable sensors is endless. Section 2.2 has already identified a number of different
activities. Human activities such jogging, sitting, standing, and walking are natural
physical activities that present themselves for recognition with motion sensors or inertial
sensors because they are defined clearly by the relative and motion position of the body
parts of the user. 3D and 2D acceleration information has been employed fruitfully for
recognizing such activities (Kunze and Lukowicz, 2014).
Activities of Daily Living (ADLs) are the essential category of activities in assisted
living and healthcare. Leonardis, Bischof and Maver (2012) initially proposed that these
activities have developed over time and are now being used by caregivers and
physicians as a standard measure to assess the physical wellness of those in need of
assisted living and the elderly patients. Some of the core set of activities of daily living
include toileting, bathing, feeding, dressing, continence, and transferring. These
activities are supplemented by Instrumental Activities of Daily Living (IADLs) which is
made up of activities like food preparation, handling finances, taking medications, using
MSc Project Proposal Page 10

a phone, doing laundry, housekeeping, shopping, and transportation. Bulling et al.
(2018) have demonstrated recognition of such specific sets of IADLs and ADLs through
recognizing activities like taking a shower, washing dishes, cleaning the windows,
vacuuming, ironing, dusting, or making tea. It is still a challenge to recognize the whole
set of IADLs and ADLs using sensors because some of the activities like handling
finances are not defined clearly and some are difficult to detect for instance, continence.
Moreover, the physicians are very interested both the fact the that patient has done an
activity and how well it was done. Aside from the activities identified, there are other
like sporting activities that can be recognized using wearable sensors such as martial arts
move, running, dumbbell exercises, wood work activities, cycling, reading, chewing,
rowing, juggling, or assembly tasks (Bulling et al. 2018).
Fascinatingly, a bigger number of researches has been focused on short-term and low-
level activities. Nevertheless, in many applications varying from modelling of human
behaviour to assisted living to healthcare, the recognition and analysis of long-term and
high-level activities is an essential component. The figure below describes the various
categories of activities that summarized the human activities as either low-level or high-
level.
Figure 1: Categorizing Physical Activities (Source: Stork, Yom-Tov and Duda, 2014)
Modelling and Recognizing Activities
This section will give an overview of the techniques of machine learning that have been
employed in recognizing activities using wearable sensors. It will only give a summary
MSc Project Proposal Page 11

of the machine learning algorithms and concepts for a conclusive mathematical
treatment. As it will be noted, several different techniques have been used. However,
comparing these techniques is not simple because the suitable technique is dependent on
the type of activity, there is no standard dataset in activity recognition, and the data type
has been recorded (Affanni, 2019).
It is quite simple for one to make the difference between unsupervised and supervised
learning techniques. Unsupervised learning basically refers to learning without an
instructor. This technique attempts to construct models directly form unlabelled data
either by identifying groups of similar examples or through density estimation. In
contrast, supervised learning techniques, commonly referred to as learning with an
instructor, involves use of labelled data where training of an algorithm is requiring
before it is capable of classifying the unknown data. Supervised learning until now has
been the dominant technique for activity recognition.
In supervised techniques, the general training process and algorithm testing for activity
recognition is made up of the following five processes: data acquisition, data
transformation, grouping features into a test set and training, and performing training on
the training set. Mostly, processes 3 to 5 are iterative with various subdivision into test-
and training set, and the outcomes are averaged, this is referred to as cross-validation, it
offers a more accurate estimate of the algorithm generalization capacity. Currently, there
exist several models and algorithms for supervised learning including Hidden Markov
Models, Nearest Neighbour technique, C4.5 decision trees, and Naïve Bayes Classifiers
(Vogel and Schiele, 2016). Other techniques that have been used according to
Adaskevicius (2014) include string-matching techniques and support vector machines
(SVMs).
Unsupervised techniques are basically made of the following steps, acquisition of
unlabelled sensor data, converting the unlabelled data into features, and modelling the
unlabelled data through clustering or density estimation. Analysis of unsupervised
techniques is usually challenging because of the insufficient evidence and truth that can
be used to compare the identified structure.
Vogel and Schiele (2016) uses Hidden Markov Models (HMMs) to identify the locations
and scenes like running in the field from video and audio data in an unsupervised
manner. Also, the employ unsupervised learning techniques centred on graphical
models. Their goal was to infer the modes of transportation such as walking car, or bus
and the end goals of the person. Wahle et al. (2016) has combined continuous Hidden
MSc Project Proposal Page 12