AI Mobile & Web Platform to Aid Non-Verbal Autistic Communication

This project outlines the development of an AI-powered platform aimed at facilitating communication for non-verbal autistic children. Recognizing the increasing prevalence of Autism Spectrum Disorder (ASD) and its impact on social interaction, the project proposes an Android mobile application integrated with a web admin panel to provide a communication medium. The platform utilizes content-based image retrieval (CBIR) to cater to the visual strengths of autistic children. The methodology involves an iterative prototyping approach, incorporating feedback from therapists to refine the system. The project details the initial planning, requirements analysis, design considerations, testing procedures, implementation strategies, evaluation metrics, and maintenance plans. The appendices include the project proposal, blueprint, project management plan, questionnaires, questionnaire analysis, and ethical considerations. The goal is to create an accessible and effective communication tool that addresses the fragmented nature of existing interventions for autistic individuals.

Table of Contents
Introduction.................................................................................................................................................2
Problem statement...................................................................................................................................3
Literature Review........................................................................................................................................4
Methodology...............................................................................................................................................5
Initial Planning........................................................................................................................................5
Planning...................................................................................................................................................5
Requirement............................................................................................................................................5
Analysis...................................................................................................................................................6
Design.....................................................................................................................................................6
Testing.....................................................................................................................................................6
Implementation........................................................................................................................................6
Evaluation................................................................................................................................................6
Maintenance............................................................................................................................................6
Development...........................................................................................................................................6
References...................................................................................................................................................7
Appendices..................................................................................................................................................9
1. Project Proposal...................................................................................................................................9
2. Blueprint............................................................................................................................................19
3. Project Management Plan.................................................................................................................20
4. Questions...........................................................................................................................................21
5. Questionnaire Analysis:.....................................................................................................................22
6. Ethics Section A & B:..........................................................................................................................31

Introduction
The recent years have seen the prevalence of Autism Spectrum Disorder(ASD) increase at
unprecedented rate. So far, the development condition is listed among the major cases of
mental disorders in various parts of the worlds including the UK, US, and India. Note that the
incidence rate in the mentioned regions are 1 in 64, 1 in 68, and 1 in 150 respectively
(Kuriakose & Lahiri 2017). Such realization is quite devastating considering the detrimental
impact of the condition on the autistic child social life. The implications are felt by not only
the autistic individual but also their direct guardians (mainly parents). At this point, the
readers are urged to consider the established facts regarding autism. The lifelong condition is
usually evidenced in the children under the age of 3, and it curtails their ability to
communicate normally, from social relationship, and engage in normal behavior. While this
disorder is barely understood from a casual perspective, most researcher and practitioner
argue that the results from defects in the brain regions dealing with memory retention, motor
skills, and empathy.
Fortunately, the escalation in the prevalence rate is concurrent with the digital revolution.
The modern world is characterized by sophisticated level of Information and Communication
Technology(ICT) advancement that appear instrumental in the alleviation of most of the
communication and socialization challenges faced by autistic children. As far as autism
concerned, major process has been made in the fields of Information Technology and
robotics (Teevan, Dumais, & Horvitz 2017). A range of computer, smartphone, and artificial
intelligence (merged with robotics) software have been developed to help this population
cope with their communication and socialization needs (Golestan, Soleiman, & Moradi
2017). The applications have been tailored to meet the special communication capabilities of
autistic children. Nevertheless, there is too much fragmentation in the emergent intervention
on the same. For this reason, we have been always curious about nonverbal children with
autism, how they express their feelings, needs and thoughts with others surrounding them.
We find it a quite interesting idea to give them a communication medium and see what they
will offer to us. Therefore, this dissertation project proposes we inspire to develop a platform
which include an android mobile application and a web admin panel, together this platform
will help the non-verbal autistic children to communication and give them a way to express
themselves.

Problem statement
By now, it is apparent that different interventions have emerged in attempt to improve the
lives of people spectrum from autism. Some of the common applications include Pocket PC
Picture Exchange Communication System (PECS), Let’s Talk Voice Output
Communications Aid (VOCA), dmTEA, and AUTHIC. Note that the first two focus on
developing the children’s communication skills (Shminan, Sharif, Adzani, & Lee 2017).
dmTEA is a diagnosis tool while AUTHIC improves the autistic child emotional capabilities.
It suite to restate that the mentioned applications are among the various interventions
implemented for autistic individuals. Such a trend demonstrates the problematic state of the
current progress with respect to the disorder. Logically speaking, it is almost impossible for a
single autistic child to benefit from such a fragmented approach. Such a bold sentiment
follows the fact that one cannot access all of the applications regardless of the fact that they
focus on different aspects of ASD. Since here in Oman the unavailability or scarcity of using
application tools in the centers to communicate with the nonverbal and verbal autistic
children.
Perhaps, communication platform of digital content based image retrieval for Autistic
children can solve this problem by enhancing the accessibility of the distinct functions –
includes digital library on a single platform(s).

Literature Review
At the moment, there is a conventional agreement among researchers, academicians, and
practitioners regarding the need to invent large-scope solutions for the growing autistic populace.
(Leroy, Chuang, Huang, & Charlop-Christy 2005) demonstrate their agreement by
acknowledging the fact that ASD is the third most prevalent neurological disorder in the United
States. Similar sentiments are held by (Kuriakose & Lahiri 2017) as they believe that its
prevalence expands beyond the U.S. borders. According to them UK and India are also recording
high incidence rates. Such perspectives are vital in understanding the global status attained by
autism. It is important to know this fact since it supports the idea that the interventions should be
geared towards the attainment of broad needs.
It appears wise to assert that the preceding paragraph does not intend to disregard the current
progress made in the field. For instance, (Kuriakose & Lahiri 2017) propose a VR-based
application that utilizes a simulated environment to teach basic communication and socialization
skills to the autistic children. This innovation can be said to be an extension of Iyer and
Kalbande’s proposed autistic game (2014). Unlike Kuriakose and Lahiri’s invention, the game is
played on a personalized graphical user interface (GUI). At this point, one should reflect on this
project’s thesis: a mobile application -based approach can allow the autistic users to access a
range of interventions from a single application.
When placed into perspective, a huge operating system can be created where both GUI and AR
functions can be accessed by the autistic children. For the sake of this argument, the search
engine concept described by (Zamir, Korn, Fikes, & Lawrence 2010) is quite illustrious: such a
system utilizes a user profile to extract relevant information based on the user’s query. With the
help of the guardian/practitioner, an autistic child can utilize this system to access various
functions including the learning medium (AR or tablet mode), the learning concepts
(communication, basic mathematics, or socialization), and level (based on progress). Of course,
such a platform must adhere to the improvements suggested by (Kodgiwar 2016). According to
her, content based image retrieval (CBIR) allows the system to access data based on graphical
searches. This function is quite beneficial for children spectrum from autism since they are more
efficient with visuals than words (Kamaruzaman & Azahari 2014). In their research, (Agarwal,
Sharma, and Dubey 2016) noted that CBIR can be improved to facilitate sketch image queries.
Such an operating system can only be efficient if it is full of visuals and colorful feature.

Methodology
Iterative Methodology: the system development life cycle to be used is the iterative prototyping
methodology. This was arrived after consideration of the requirements specification, which are
understood and predicted to evolve with time. The process allows the developer to start with a
limited part of requirements and the iteration design gives room to return and add to the
requirements to expand the abilities repeatedly until the system is fully developed, and can be
implemented and later deployed (Anon, n.d).
Initial Planning we will design a mobile application using content based image retrieval (CBIR)
Planning Considering the nature of the investigation, as shown in (Table 1), researcher will
utilize information collected from professionals in the fields of The age of autistic children will
be between 4 to 11 years the status of their situation nonverbal. Case study method of selection
institutions that contain English Speech Language Therapists (SLT), special education Teachers,
child psychologist and autistic children. Number of cases will be between 3 to 5 therapists who
use English language in therapy will be in a groups depend on the number of the institutions.
Number of cases will be between 3 to 5 institutions depending on the availability of those
children and therapists. According to the final statistics gathered by The Ministry of Social
Development the total number of autistic children in Oman around (1242, Figure 1). These
statistics does not show which institution uses English language in their therapy. Case studies
will be taken from the capital area/Muscat governorate because of the density of the institutions
and number of children.
We prefer 3 to 5 therapists from each institution due to the scope of the proposed project: Each
group of the therapists with their assigned autistic children will be having a research designed
using semi structure interview questions from the researcher who will be collecting the response
garnered from the posed research questions will be on both Arabic and the English language.
Once we collect the data in Arabic form will be translated into the English language.
Table 1. Planning methodology of gathering data from the institutions
Group
Type of
Sector Institution Language
Therapis
t Participant age (4 to 11) Observation
1 Private 1 English Male Female Non-verbal
A
B
C
2 Private 2 English D
Requirement the requirements provide an understanding of the proposed system. This established
the main essential requirements, at the best level of knowledge for a start, that is expected to
increase or/ and evolve with the time. Case modeling can be used to identify the products
beforehand like the system specification (Table 2). The requirements will then be classified and

prioritization allocated. They can then be enlisted in vision document or a system charter for
easier reference.
Analysis after collection (Appendance 3 Questionnaires Analysis), the information will be
collated then assessed to enhance the researcher’s gaps of the potential merits and demerits of the
proposed project. It is through this process that applicability of the proposed solution can be
tested before the design and development process begins. Measuring social development by
conducting paper impact of the electronic impact image.
Design as mentioned earlier, the integrated effectiveness of digital content-based image retrieval
application for ASD children will utilize the structure of a typical search engine. The indicators
for observation schedule, and testing emotional expression in the autistic children. The only
difference is that it will be highly graphical due to the content/sketch-based image retrieval
algorithms. While designing the application we will sketch a new image using character 2D
sketch for different ages such as mother, father, grandmother, grandfather…etc. we will be
adding some character with traditional Omani and casual cloths it will be familiar for them.
Testing the software will be tested during the prototype stage. Further, enhancements can be
done only after the prototype proves functionality. Also, testing the functionality of emotional
expression in the participant.
Implementation the system iteration developed inclusive of the evolved and additional
requirements. We will review the code using the therapist’s user feedback received from the
testing and implements the changes and extra functionalities to suit the therapist’s requirements.
Each module will be tested and modified or updated accordingly.
Evaluation the process will be based on the number of ASD users and therapists in the system.
Maintenance once the approved modules will be tested to satisfaction, and the incomplete
modules are redone to full functionality.
Development it is the last stage of development. The successful of mobile application will be
deployed into a fully operational system version. It will necessitate various procedures including
licenses acquisition, prototype development, and network expansion. Each stage demands huge
amounts of financial and human resources. Not to mention the fact that such a project can be
taken up to a six month before completion.
Table 2. Software & Hardware Requirements
Details Software & Hardware SW.Version HW.Version
IDE IDE: Android Studio 3.2
Programming Language Java 1.8 and XML 1.8
Web admin Firebase Cloud Firestore and SDK 17.1.3
Android API 17.1.3
Server Database Firebase Cloud Database 17.1.3
IDE(Integrated development environment) Android Studio 3.2

Android device Support android media framework
References
1. Anon, (n.d.). Iterative Model: What Is It And When Should You Use It?. [online]
Available at: https://airbrake.io/blog/sdlc/iterative-model [Accessed 16 Jan 2019].
2. Anon, (n.d.). SDLC - Iterative Model - Tutorials Point. [online] Available at:
https://www.tutorialspoint.com/sdlc/sdlc_iterative_model.htm [Accessed 16 Jan 2019].
3. Anon, (n.d.). What is iterative development? - Definition from WhatIs.com. [online]
Available at: https://searchsoftwarequality.techtarget.com/definition/iterative-
development [Accessed 17 Jan 2019].
4. Argawal, S., Sharma, R. and Dubey, R. 2016, Sketch-based image retrieval using watershed
transformation. Second International Conference on Computational Intelligence & Communications
Technology, 2.
5. Draw.io. (2019). Flowchart Maker & Online Diagram Software. [online] Available at:
https://www.draw.io/?state=%7B%22ids%22:%5B%221yR0iE0z-BiPBeEdFvEEAMD3u-5LoppCh
%22%5D,%22action%22:%22open%22,%22userId
%22:%22111030580720086588443%22%7D#G1yR0iE0z-BiPBeEdFvEEAMD3u-5LoppCh
[Accessed 22 Jan. 2019]
6. Figure
7. Fulbright, R. (2013). Incorporating Innovation into Iterative Software Development
Using the Inventive Problem Solving Methodology. International Journal of Innovation
Science, 5(4), pp.203-212.
8. Golestan, S., Soleiman, P. and Moradi, H. 2017, Feasibility of using Sphero in rehabilitation of children
with autism in social and communication skills. International Conference on Rehabilitation Robotics
(ICORR), 1
9. Google Docs. (2019). Technologies in communication & Learning of Autism children. [online] Available at:
https://docs.google.com/forms/d/1sWhvRPN_-rQ5CC93X1_NUwwHv6CprO6OjZO-oq5dL-o/
edit#responses [Accessed 23 Jan. 2019].
10. Google Docs. (2019).  في التوحدتدخل التكنولوجيا. [online] Available at:
https://docs.google.com/forms/d/1Bf0CAJp97ot-6Zo0mcVJ7eNEBCHsxkn7MufZTawn9eo/edit [Accessed
23 Jan. 2019].
11. Iyer, S. and Kalbande, D. 2014, Research on educative games for autistic children. International
Conference on Circuits, Systems, Communication and Information Technology Applications (CSCITA), 1
12. Kamaruzaman, M.F. and Azahari, M.H.H. 2014, Form design development study on autistic counting skill
learning application. International Conference on Computer, Communication, and Control Technology, 1.
13. Kodgiwar, V. 2016, Application of content-based image retrieval for e-commerce. World Conference on
Futuristic Trends in Research and Innovation for Social Welfare.

14. Kuriakose, S. and Lahiri, U. 2017, Design of a physiology-sensitive VR-based social communication
platform for children with autism. IEEE Transactions on Neural Systems and Rehabilitation Engineering,
25.
15. Leroy, G., Chuang, S., Huang, J. and Charlop-Christy, M.H. 2005, Digital libraries on handhelds for
autistic children. JCDL 5.
16. Managing Iterative Software Development Projects. (2007). Kybernetes, 36.
17. Prod.teamgantt.com. (2019). TeamGantt. [online] Available at:
https://prod.teamgantt.com/gantt/schedule/?
ids=1383084#&ids=1383084&user=&custom=&company=&hide_completed=false&date_filter=
&color_filter= [Accessed 22 Jan. 2019].
18. Shminan, S.A., Adzani, A.R., Sharif, S. and Lee, N.K. 2017, Autipecs: Mobile Based Learning of Picture
Exchange Communication Intervention for Caregivers of Autistic Children. New York, USA: IEEE.
19. Teevan, J., Dumais, S. and Horvitz, E. 2017, Personalizing search via automated analysis of interests and
activities. ACM SIGIR Forum, 51.
20. Zamir, O.O., Korn, J.L., Fikes, A.B. and Lawrence, S.R. 2010, United States Patent. Patent No.
7693827B2.

Appendices
1. Project Proposal
MSc Project Proposal Form and Ethics Form
MSc Project Proposal Form
Student Number 1623746
Student Name Amna Saleh Nasser Al Araimi
Degree Course MSc Management Information System and
Security.
Supervisor Name Dr. Haiming Liu.
Title of Project
Communication Platform for Non-verbal
Autistic children in Oman using Android
mobile
Description of your artefact
The context:
This project is an extension to the research
conducted by H. Liu, F. Dong, C. Tziraki in
the area of supporting people with
Intellectual Disability (ID) to use Content -
Based Image Retrieval (CBIR) technology
for information seeking in order to assist
them to improve their communication (H.
Liu, et.al).
Autistic children are viewed as ID people
whom are in need for a technological tool
to ease their access and interaction with
CBIR to improve their social interaction
and communication (Zhou, Z., 2006). They
are the subject of this project in Oman.
Aim & objectives of the project
The Aim:
The aim of this project is to tackle
communication dependency of nonverbal
of special needs by providing
a substitutional to verbal communication
with visual communication. We have been

always curious about nonverbal children
with autism, how they express their
feelings, needs and thoughts with others
surrounding them. We find it a quite
interesting idea to give them a
communication medium and see what they
will offer to us. Therefore, we are inspired
to develop a platform which includes an
android mobile application and web admin
panel, together this platform will help non-
verbal autistic children to communicate
and give them a way to express
themselves.
The objectives:
1. To extend the CBIR technology to be
applied on autistics children.
2. Most non-verbal autistic children hardly
express their mood or how they feel
since they lack the natural way of self-
expression.
3. Identified that autistic children can and
will express themselves better through
Image based communication and text-
to- speech application.
4. Understanding by learning what images
represent will help them choose the
images that represent what they want
to communicate.
5. To offer an environment of enjoyable
and attractive communication and
personalization platform.
6. Provide a platform of interaction to the
autistics children, therapists, and
supervisors.
7. Provide performance track record and
measurement indicators for follow up.
8. Children with neurological influenced
communication.
9. Economic accessible
10. Social integrated
11. Home accessible
12. Cultural recognizer using character with
traditional Omani and casual cloth

List of features that the artefact will
include:
1. A database will be created for this
purpose including the following:
a. Autistic profile
b. Therapist profile.
c. Admin profile.
2. An application will be created with
relevant layout for each of the three
users: Autistics, therapists, and admin.
3. CBIR interactive library software will be
integrated with the application for the
three users.
4. Use of identity referencing for each
autistic child (given by the ministry of
social development).
5. Ability to support image formats (jpeg
and png)
6. Ability to support YouTube video
through video URL
7. Ability to share and send data (with the
therapist)
8. Ability to log important data for admin
only.
9. Ability to log important data for admin
user only.
10. Ability to assign different privileges to
different user (such as Therapist)
11. Ability to present report and
performance in an easy to understand.
The added value:
1. There is scarcity of technological tools
of information seeking for ID in general
and for autistics in specific. This project
becomes contributing to solve this
problem.
2. This project will bridge the gap in
communication of autistics children to
information seeking and improve their
social interactions.
3. This project will be a good tool to
examine the impact of using CBIR on

autistic skills.
4. The early intervention is very crucial
in detecting autism. This project will
assist therapists to identify and
overcome communication and
socialization difficulties before
becoming severe.
The intellectual challenges involved:
1. The first intellectual challenge is that
this area of autism is a very specialized
area which requires reading a lot of
theories and practice materials. The
design of a proper suitable and
consisted system to assist autistics will
depend on my exposure to these
martials.
2. It’s difficult to track the progress in
communicating (Paul, R.et.al, 2005),
because these are qualitative issues
and if this system will focus on tracking
the improvement of using CBIR by
autistic it will depend to a certain extent
on the subjective judgement of the
therapists.
3. The major issue the data which
provided has been requested and
collected in the Arabic language which I
need to translate to the English
language.
4. Questionnaire written in both
languages the English and the Arabic.
Which will take more time to translate
the Arabic form result into the English
form result. Gathering and analyzing
those result will take more time to have
accurate result.
5. The data which provided from the
Ministry of Social Development was in
the Arabic language, also need to be