Big Data Challenges in IoT and Cloud

This research report discusses the challenges of big data in Internet of Things and cloud. It covers the issues related to data storage, management, and transmission. The report provides insights on how to overcome these challenges.

Running head: BIG DATA CHALLENGES IN IOT AND CLOUD
BIG DATA CHALLENGES IN IOT AND CLOUD
ALAPATI SESHUBABU
University Name
Author Note:

Milestone 1
Executive Summary
The main aim of this research report is to understand the various challenges of big data in
Internet of Things and cloud. Big data is considered as one of the most effective technology that
helps in data analysis. The big data could be easily analyzed for the insights, which eventually
lead to the better business decisions and also strategic business movements. Internet of Things,
on the other hand, is the type of inter connection that help any user in sending or receiving of
data or information. Cloud computing is the most effective method of data transfer over the
connection of internet. Big data can bring out various advantages, when it is being mixed up with
the internet of things and cloud computing. However, there are some of the major issues or
challenges that can occur in the phenomenon. This research report has clearly described all the
important details related to such challenges.

Milestone 2
Introduce of the Problems
One of the main features of utilizing IOT is to observing the behavior of many things to
gain important optimize process, insights and so on. By utilizing the Big data several problems
are solved such as store all the events, run analytical queries over the stored events and perform
analytics (machine learning and data mining) over the data to gain insights (Cai et al., 2017).
However there are main challenges to achieve the ability for performing real time analytics using
big data such as correlate streaming events with my stored data in the operational database,
process streaming events on the fly and store streaming events in the operational database. Rapid
growth of IOT applications also presenting many challenges associate with the Big Data. Though
lot of problems is likely to be solved but still needs sufficient research to migrate problems
confronted. Data storage and management can raise challenges such as velocity, variety and
volume of Big Data (Lee et al., 2013). Big Data velocity involves storage system to enable the
capability to scale up quickly which is hard to gain with traditional data protection mechanism.
However big data on the cloud is super expensive for utilizing a vast data volume. There are
other certain challenges of Big Data in Data transmission process in several stages of lifecycle.
Most of the challenges occur in the process of data collection, data integration and data
management. Transferring large amount of data produces obvious risks in each of stages. It is
also hard for traditional systems to efficiently analyze, manage and visualize unstructured data
(Barnaghi, Sheth & Henson, 2013). The main reason for huge amount of data produced by IOT
enabled devices is the growing number of internet-enabled devices utilized for several purposes
by enterprises, individual and government. The large amount of data generated by IOT devices
needs more computational power to process. Consequently, data produced by IOT devices in

several application domains are time perilous. Processing the vast amount of data in a timely
manner is very demanding. As a result the whole process, plans and effective decisions are
formulated for several applications sets.
Structure of the report
This section provides the clear definition of the structure followed in this report. There is
several section of this report for clearly describing the security and privacy regime in IOT. In
every section there is proper heading with suitable subheading to demonstrate the broad about
security and privacy regime associate with IOT. The demonstration of the followed structure is
following-
 Executive Summary: Executive summary is the first part of the report where the summary
of the whole report is presented. This section discusses about the Big Data challenges in
the IOT. It helps to understand the whole report in short time. These sections also provide
the brief summary of the whole content.
 Introduction: This part discuss about the basic challenges associate with the topic. It
provides basic understanding of the chosen topic. It broadly describes the basic
knowledge of big data and what are the challenges associates with the increment of data
set. It also explains familiar application of big data associate with the IOT and cloud.
 Literature Review: this part of the project discuss about the published literature on the
relevant topic. It provides a broad understanding of previous literature and the finding of
that literature with reflection. This section provides a board review of existed research
papers conducted through many sources such as Google scholar and e-libraries.
 Methodology: this is one of the crucial parts of the project where the suitable method will
be chosen for completing the report. After reviewing and compare past and previous

methodologies my own perspectives are illustrated. This section illustrates five
methodologies which are utilized to propose the desired knowledge of big data challenges
associates with the IOT and cloud.
 Conclusion: in this part the result of the whole report is illustrated. This part widely
discusses about the big data challenges in the IOT and how to measure them in term of
efficiency, feasibility and connectivity.
Milestone 3
Literature Review
Introduction
Nowadays, there are several modifications are happening in the technology field
especially in the platform of cloud, big data and internet of things. The internet and the amount
of the users are growing rapidly, it also producing vast amount of data day by day. According to
some research (Bessis & Dobre, 2014). Everyday more than 2.35 quintillion bytes are produced.
The vast amount of data is produced by the user who is creating this for utilizing many
applications. It is getting hard to manage this vast amount of data which are being produces for
several uses. The collaboration of machine and human is producing many challenges as well.
According to Chen, Mao & Liu, 2014, most of the devices or system used by user in a daily basis
produces vast amount of data which are always not visible as hidden information stored in the
devices. Internet of things is triggering the database management growth in future. The
challenges occur to manage large dataset from every system or devices. Prediction capabilities of
the applications and services and higher level of decision making are very essential to achieve
the full advantage from the services and context aware data intensive applications and make the

important information or valuable transparent and available at a much higher frequency. As per
Chen et al., 2015, big data has become very essential to turn the vast amount of data into
information that can be transferred with a high frequency. Big data can be define as high
velocity, volume an verity information which enable innovative way of information processing
and to gain enhanced insights through automation process, effective decision making and
analytical tool. According to research paper from Gartner, almost every business, enterprise and
individuals are producing vast amount of data which can be analyzed through automation
process to manage it more effectively (Lee & Lee, 2015). In fact the life of data can be very short
and this may become obsolete after some time. So, efficient usage of Big Data analytics results
will bring good useful insights from high volume, variety of data. Sometimes quality of the data
is a concern area because it fetches data from different applications for making decisions. Not all
the data captured from various devices are useful for making decisions (Li, Da Xu & Zhao,
2015). These are just information and the information has to be converted into knowledge for
decision making.
Fan & Bifet, 2013, state that for the large and complex volume of big data it is very tough
to manage and analyze the data. Effective data management would not be possible by utilize
typical Rational dataset management system (RDBMS). For the vast amount of big data it is also
very hard to extract proper information in an effective manner. Generally, the raw big data is the
source of the desired information. Data is consisted with each other as raw input. Individual’s
data has no value. Vast amount of data provide meaningful information with several patterns and
trends.

Present and Past Work
There are several research papers available which are available in the web. Some of the
research tried to estimate all the challenges could possible occurred by the overwhelming amount
of data (Jin et al., 2014). Technology shifts more data through various applications like wireless
sensors, smart devices, social media etc. This paper focuses on the improvement the performance
of the old services and offer new services in an open and dynamic environment.
Big Data Definition
Big data is the specific type of data set, which is extremely complex as well as
voluminous and the traditional data processing application software is not at all sufficient for
dealing with these complex data sets (Gudivada, Baeza-Yates & Raghavan, 2015). Big data is
one of the most important and significant data sets that have the advantage to deal with any type
of data and there is no restriction to data size or data limit. It can be easily referred to as the
evolving term, which describes all types of voluminous amount of unstructured, structured and
semi structured data, which comprises of the string potential for information mining. This
particular type of data, big data is subsequently categorized by the help of 3Vs (Mishra, Lin &
Chang, 2015). These 3Vs are highest volume of the data, broader variety of data and finally the
highest velocity of data. The data that is being processed is checked with the help of this big
data. There is no specific data volume for big data and thus this term is utilized for describing the
petabytes, terabytes and finally exabytes that is captured over time. This specific type of data
inundates any organization on the day to day basis. The big data could be easily analyzed for the
insights, which eventually lead to the better business decisions and also strategic business
movements.

Big data eventually refers to the proper utilization of user behavior analytics, predictive
analytics and any other method of advanced data analytics. These advanced data analytics are
solely responsible for extracting significant value from the data and also to the specific data set
size (Obitko, Jirkovský & Bezdíček, 2013). The proper analysis of the data sets could easily find
out various newer correlations that help in spotting the business trends, preventing the diseases
and combating the crime and many more.
Description of 3Vs in Big Data
As per Hassanalieragh et al., 2015, for the description of the 3Vs, that is volume, velocity
and variety, big data has become explicitly popular and well accepted by every business or
company. The organizations or businesses eventually collect the data from a vast variety of
various sources. These sources majorly include all types of business transactions, social media
and finally sensor information or data and data related to machine to machine. In the previous
days, storage of the volumetric data was extremely difficult, however, with the rise of various
technologies like Hadoop, the complexities or problems are solely reduced to a greater extent
(Sivarajah et al., 2017). The next V amongst the 3Vs is velocity. Data streams in at an
unprecedented speed and must be dealt with in a timely manner. RFID tags, sensors and smart
metering are driving the need to deal with torrents of data in near-real time. This is extremely
important for any organization or business as it increases the velocity or speed of the business
processes or operations. The final V is the variety. It eventually refers to the fact that any type of
data comes in every format. The various formats of data are structured, numeric and unstructured
(Gubbi et al., 2013). The traditional databases are not affected in this sector and hence the
emails, audio, video and various financial transactions are checked in the process.

Apart from the above mentioned 3Vs two more dimensions are present within the big
data. These are variability as well as complexity. Variability refers to the fact that big data does
not comprise of any type of consistency, which means the data is not at all fixed (Hashem et al.,
2015). There is always a high chance of variation or changeability. Thus, easy adaption and data
change is possible in big data. Moreover, this big data is much complex to crack and hence there
is extremely less chance that the complexity is almost zero in this particular data type.
Big Data in Internet of Things and Cloud
Internet of things can be defined as the significant network of various physical devices,
home appliances, vehicles and several other items that are eventually embedded with the
software, electronics, actuators, connectivity and sensors that solely enable all the objects in
proper connection as well as exchanging of confidential information and data (Yang, 2014). All
these things that are related to internet of things and cloud computing could be uniquely
recognized by the specific embedded computing system and has the ability in inter operability in
the existing infrastructure of the Internet connection. This internet of things enables all the
objects that are to be controlled as well as sensed remotely within the existing network
infrastructure. This helps to create various opportunities to directly integrate the typical physical
world within the systems that are completely based on computers. This helps to improve the
efficiency, to provide economic advantages and finally to provide accuracy of the systems. This
eventually reduces the efforts as well as intervention of the human beings (Strohbach et al.,
2015). When IoT or cloud is augmented with sensors and actuators, the technology becomes an
instance of the more general class of cyber-physical systems, which also encompasses
technologies such as smart grids, virtual power plants, smart homes, intelligent
transportation and smart cities.

Big data and Internet of things or cloud computing are both extremely important and
significant technologies in the technological world. When these two popular and famous
technologies are amalgamated together, they bring out one of the best technologies for the world.
The exclusive growth of the number of various devices that are connected to the technology of
IoT or Internet of Things as well as the exponential increment of consumption of data has
eventually reflected on the fact that the growth or development of big data can easily overlap the
technology of IoT or Internet of Things (Wortmann & Flüchter, 2015). The entire management
of the big data within the continuous expansion of network has solely given rise to any non
trivial concern about the efficiency of data collection, data security, and analytics of data and
processing of data. For the purpose of addressing all of these problems or concerns, the
researchers have subsequently examined each and every challenge that is linked with the proper
deployment of Internet of things. Although, there are various studies on the big data and Internet
of things, there is a strong convergence of all of these areas and thus creating various
opportunities to flourish the big data or analytics for the systems of internet of things and cloud
computing.
Big Data Issues in IoT and Cloud with Solutions
Big data comprise of various issues or challenges when it is mixed with the Internet of
things and cloud. The various challenges of big data with their probable solutions are given
below:
i) Heterogeneity and Incompleteness: The first and the foremost challenge of the big
data when present in the technology of Internet of things and cloud is heterogeneity and
incompleteness (Zaslavsky, Perera & Georgakopoulos, 2013). The difficulties of big data
analysis derive from its large scale as well as the presence of mixed data based on different

patterns or rules or heterogeneous mixture data in the collected and stored data. In the case of
complicated heterogeneous mixture data, the data has several patterns and rules and the
properties of the patterns vary greatly. Moreover, there are various uncertainties of data that data
might lose the integrity, while it is being managed. All these occur when big data is working
with internet of things (Zanella et al., 2014). The solution to the above mentioned problem is that
the data should be collected or stored properly for reducing the problem of heterogeneity and the
issue of data incompleteness is solved by my qualitative data analysis.
ii) Working with Large Set of Data: There is a challenge in big data that it cannot work
with smaller data sets and thus only huge amount of data sets are involved in the process. Often
the user suffers from various issues due to this as he is not being able to link internet of things
with big data if the data set is smaller. For solving this problem, various approaches are to be
taken by the user
iii) Scale as well as Complexity: The third significant challenge of the big data in IoT and
cloud is that it is extremely complex and the management of the larger and increasing data
volume is considered as the most challenging issue ever for any user. Thus, the user needs to
manage or control the data properly and perfectly. The solution to the above mentioned problem
is that the data should be managed with the help of various software tools.
iv) Data Clustering: Another significant issue with the big data in internet of things is
data clustering. This data clustering refers to the task of clustering or grouping the specific object
set in the particular method by which the objects within the same cluster work together (Da Xu,
He & Li, 2014). This becomes a major problem for Internet of things and cloud computing as it
cannot work with data clustering. The solution of this challenge is to store the data without being
clustered.

Methodologies
Method 1: Hadoop:
This is an Open source mission which is mainly managed by the Apache Software
Foundation. By using this Hadoop it is easy to collect and handle the big data. This is proposed
for the purpose of parallelizing the processing’s of the data and this is done by computations of
the nodes in order hurry the computations and also to hide the latency (Perera et al., 2014). The
Hadoop mainly consists of two components and this includes the Hadoop Distributed File system
and the Map Reduce engine. The Hadoop Distribution File System is associated with storing the
enormous data in a constant way and is also associated with reproducing it to the user
applications which are at high bandwidth. The MapReduce can be considered as a framework
which is used for eth purpose of process the data sets which are massive and are present in a
distributed fashion through numerous machines.
Method 2: Map Reduce:
This is mainly constructed in the form of board programming paradigm. It is seen that
some of the original employments are offered all the key needs related to the parallel execution,
fault tolerance, balancing of load, and manipulation of data. This has been named as this mainly
includes two kinds of abilities from the functional computer languages which are existing and
this includes the map and reduces (Stankovic, 2014). The MapReduce framework is associated
with gathering all the sets which are consisting of common keys from all the records which is
followed by joining them together. This initially results in the formation of one group for each of
the different keys which are produced. This is one of the new technologies, which is just an
algorithm and a technique by which all the data can be fitted. In order to obtain the best from the
Map Reduce there is just a need of an algorithm. For this there is a need of collection of products

and the technologies which are created in order to manage all the challenges faced by the big
data.
Method 3: HBase:
This is the database model which is present inside the Hadoop framework and looks like
the original system of the big table. The HBase also consist of a column which is associated with
operating as the key and is the only index which can be used for the purpose of getting back to
the row (Cui, 2016). The data which is present in the HBase can also be saved as various sets,
and in this the subjects which are present in the non-key columns is represented by the values.
Method 4: Hive:
The tools which has already been deployed for data warehousing is not at all suitable and
especially in the situation where it is seen that the data is accessible everywhere. Besides this
they are also costly and are often operated privately. Sometimes the Hive may also be thought as
the necessary portion of Hadoop system and is viewed at the top that principally is the
organization for the data warehouse (Al-Fuqaha et al., 2015). Hive cannot treat with applications
and transactions of the real time those are achieved online. The motivation behind it is a
complicated technique.
Advantages and Disadvantages
Advantages of Big Data in IoT and Cloud
There are various advantages of benefits of big data within internet of things. The several
benefits are given below:
i) Detection of errors: The first and the foremost advantage of big data in internet of
things is that it helps to detect any type of error instantly and hence the users are benefitted from

the technology instantly (Chandrakanth et al., 2014). Moreover, the operations or processes of
the business are improved eventually since the error detection is done easily.
ii) Improved Strategies: The second significant benefit of big data in internet of things is
that it helps in improving all the new strategies within the business properly and thus the
business enjoys major advantages and benefits from these technologies. The main reason of this
the presence of data analytics.
iii) Cost Savings: Another important benefit of the big data in internet of things is that it
is extremely cost effective and does not incur huge cost for the user (Mukhopadhyay &
Suryadevara, 2014).
iv) In Memory Databases: The fourth benefit of big data in internet of things is that it
comprises of an in memory database and hence other databases are not required.
Disadvantages of Big Data in IoT and Cloud
The various demerits of big data within internet of things are given below:
i) Requirement of Special Computer Power: Big data in internet of things needs special
computer power and thus is termed as one of the most difficult technologies.
ii) Utilizes Real Time Insights: The next demerit is that utilizes real time insights and
hence the user suffers from major issues (Biswas & Giaffreda, 2014).
Future Directions of Big Data in IoT and Cloud
There are some of the future directions of big data in Internet of things. They are as
follows:

i) Increasing Data Volume: The data volume will be growing as days will pass and thus
data management issue would be reduced (Whitmore, Agarwal & Da Xu, 2015).
ii) Tools for Analysis: Various other tools would analyze the data properly and thus data
loss would be highly reduced.
iii) Machine Learning: This technology would be mixing up with machine learning and
hence the technology would be enhanced.
Conclusion
Therefore, from the above research report, conclusion can be drawn that big data, Internet
of things and cloud computing are the most promising technologies in the modern world. These
two technologies help in analysis of data. Big data deals with various larger sets of data and
makes the data operations much easier for the users. big data not only means vast volume, as
there is three type such as variety (types of content), velocity (rate of the data production and
processing rate) and volume (number and size). The volume expedition is been a one of the most
dominant issue in the IOT industry. The volume rate is increased from terabyte to petabyte and
Exabyte which is equal to one million Terabyte. Where variety refers to the type of content such
as data format, under structured, semi structure which may produce from different type of
sources such as devices, machines, sensors many more. The Speed of the production of data and
to process to the data to generate valuable insights referred as Velocity. These types of data sets
are subsequently analyzed computationally for the purpose of revealing the trends, patterns and
associations. This is specially related to the human interactions or behavior. The three types of
data that are structured, semi structured and unstructured are well managed in this particular
technology and increasing the significant potential of information mining. Internet of things

refers to the particular inter connection via the Internet of computing devices embedded in
everyday objects, enabling them to send and receive data. It is considered as the best technology
and thus is much popular in the entire technological world. Internet of things is also termed the
data transfer that is done on the Internet without needing the human to computer or human to
human interaction. The above research report has clearly outlined various important and
significant features of the amalgamation of internet of things, cloud computing and big data. The
several advantages, disadvantages and future directions of the cloud computing are mentioned
here with relevant details.

References
Al-Fuqaha, A., Guizani, M., Mohammadi, M., Aledhari, M., & Ayyash, M. (2015). Internet of
things: A survey on enabling technologies, protocols, and applications. IEEE
Communications Surveys & Tutorials, 17(4), 2347-2376.
Barnaghi, P., Sheth, A., & Henson, C. (2013). From data to actionable knowledge: Big data
challenges in the web of things [Guest Editors' Introduction]. IEEE Intelligent
Systems, 28(6), 6-11.
Bessis, N., & Dobre, C. (Eds.). (2014). Big data and internet of things: a roadmap for smart
environments (Vol. 546). Springer International Publishing.
Biswas, A. R., & Giaffreda, R. (2014, March). IoT and cloud convergence: Opportunities and
challenges. In Internet of Things (WF-IoT), 2014 IEEE World Forum on (pp. 375-376).
IEEE.
Cai, H., Xu, B., Jiang, L., & Vasilakos, A. V. (2017). IoT-based big data storage systems in
cloud computing: perspectives and challenges. IEEE Internet of Things Journal, 4(1), 75-
87.
Chandrakanth, S., Venkatesh, K., Uma Mahesh, J., & Naganjaneyulu, K. V. (2014). Internet of
things. International Journal of Innovations & Advancement in Computer Science, 3(8),
16-20.
Chen, F., Deng, P., Wan, J., Zhang, D., Vasilakos, A. V., & Rong, X. (2015). Data mining for
the internet of things: literature review and challenges. International Journal of
Distributed Sensor Networks, 11(8), 431047.

Chen, M., Mao, S., & Liu, Y. (2014). Big data: A survey. Mobile networks and
applications, 19(2), 171-209.
Chen, M., Mao, S., Zhang, Y., & Leung, V. C. (2014). Big data: related technologies, challenges
and future prospects (p. 35). Heidelberg: Springer.
Cui, X. (2016). The internet of things. In Ethical Ripples of Creativity and Innovation (pp. 61-
68). Palgrave Macmillan, London.
Da Xu, L., He, W., & Li, S. (2014). Internet of things in industries: A survey. IEEE Transactions
on industrial informatics, 10(4), 2233-2243.
Fan, W., & Bifet, A. (2013). Mining big data: current status, and forecast to the future. ACM
sIGKDD Explorations Newsletter, 14(2), 1-5.
Gubbi, J., Buyya, R., Marusic, S., & Palaniswami, M. (2013). Internet of Things (IoT): A vision,
architectural elements, and future directions. Future generation computer systems, 29(7),
1645-1660.
Gudivada, V. N., Baeza-Yates, R. A., & Raghavan, V. V. (2015). Big Data: Promises and
Problems. IEEE Computer, 48(3), 20-23.
Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A., & Khan, S. U. (2015). The
rise of “big data” on cloud computing: Review and open research issues. Information
Systems, 47, 98-115.
Hassanalieragh, M., Page, A., Soyata, T., Sharma, G., Aktas, M., Mateos, G., ... & Andreescu, S.
(2015, June). Health monitoring and management using Internet-of-Things (IoT) sensing

with cloud-based processing: Opportunities and challenges. In Services Computing
(SCC), 2015 IEEE International Conference on (pp. 285-292). IEEE.
Jin, J., Gubbi, J., Marusic, S., & Palaniswami, M. (2014). An information framework for creating
a smart city through internet of things. IEEE Internet of Things Journal, 1(2), 112-121.
Lee, G. M., Crespi, N., Choi, J. K., & Boussard, M. (2013). Internet of things. In Evolution of
Telecommunication Services(pp. 257-282). Springer, Berlin, Heidelberg.
Lee, I., & Lee, K. (2015). The Internet of Things (IoT): Applications, investments, and
challenges for enterprises. Business Horizons, 58(4), 431-440.
Li, S., Da Xu, L., & Zhao, S. (2015). The internet of things: a survey. Information Systems
Frontiers, 17(2), 243-259.
Mishra, N., Lin, C. C., & Chang, H. T. (2015). A cognitive adopted framework for IoT big-data
management and knowledge discovery prospective. International Journal of Distributed
Sensor Networks, 11(10), 718390.
Mukhopadhyay, S. C., & Suryadevara, N. K. (2014). Internet of things: Challenges and
opportunities. In Internet of Things (pp. 1-17). Springer, Cham.
Obitko, M., Jirkovský, V., & Bezdíček, J. (2013, August). Big data challenges in industrial
automation. In International Conference on Industrial Applications of Holonic and Multi-
Agent Systems (pp. 305-316). Springer, Berlin, Heidelberg.
Perera, C., Zaslavsky, A., Christen, P., & Georgakopoulos, D. (2014). Context aware computing
for the internet of things: A survey. IEEE communications surveys & tutorials, 16(1),
414-454.

Sivarajah, U., Kamal, M. M., Irani, Z., & Weerakkody, V. (2017). Critical analysis of Big Data
challenges and analytical methods. Journal of Business Research, 70, 263-286.
Stankovic, J. A. (2014). Research directions for the internet of things. IEEE Internet of Things
Journal, 1(1), 3-9.
Strohbach, M., Ziekow, H., Gazis, V., & Akiva, N. (2015). Towards a big data analytics
framework for IoT and smart city applications. In Modeling and processing for next-
generation big-data technologies (pp. 257-282). Springer, Cham.
Whitmore, A., Agarwal, A., & Da Xu, L. (2015). The Internet of Things—A survey of topics and
trends. Information Systems Frontiers, 17(2), 261-274.
Wortmann, F., & Flüchter, K. (2015). Internet of things. Business & Information Systems
Engineering, 57(3), 221-224.
Yang, S. H. (2014). Internet of things. In Wireless Sensor Networks (pp. 247-261). Springer,
London.
Zanella, A., Bui, N., Castellani, A., Vangelista, L., & Zorzi, M. (2014). Internet of things for
smart cities. IEEE Internet of Things journal, 1(1), 22-32.
Zaslavsky, A., Perera, C., & Georgakopoulos, D. (2013). Sensing as a service and big data. arXiv
preprint arXiv:1301.0159.