Analysis of Big Data: History, Challenges, Techniques & Apps

This report provides a comprehensive overview of big data analysis, starting with its historical roots traced back to the 17th century and John Graunt's statistical analysis of the bubonic plague. It highlights the evolution of data storage and management through the invention of machines like the Hollerith Tabulating Machine and the development of data centers. The report delves into the challenges of big data analytics, including data security, complexity, integration, capture, mobility, and value. It defines big data as complex and large datasets that require advanced software for management and operation, enabling user behavior analytics and predictive modeling. The document also outlines various techniques for analyzing big data, such as machine learning, association rule learning, and classification tree analysis. Furthermore, it discusses the five key characteristics of big data: volume, variety, velocity, variability, and veracity, emphasizing their importance in data analysis. Finally, the report explores how big data technology supports businesses by enhancing efficiency, improving customer understanding, managing risks, identifying trends, and personalizing marketing efforts. It concludes by referencing several academic sources to support its findings.

History on big Data
The big data is trace first time in 1663 by the John Graunt
because dealt the with the large amount of information and that time the John studied the
bubonic plague which was persistent Europe at the time. Graunt was first person who use
the statistical data analysis to include the analysing data and collecting data because hey
was face the problems and issues in storing the data and it was not secure at the time due to
the lack of technology and information systems. And the statistical fields is expanded in the
1800s. Due to the lack of technology, they face the problems in stored the data and manage
in the appropriate and systematic ways and overwhelming in 1880. during the census
program, the US census Bureau announce that estimate and understand the to handle and
process the data collected.
In the 1881, the calculation work was reduced by the invented
machine Hollerith Tabulating Machine. In the 20 century, the data are developed the
unexpected speed and ways and its impossible to store in the small size the big data are
come with the core of evolution for store the data and manage the data in the huge size. The
machines are also invented for storing the data information in the huge size as well as
computers are also invented at the time. The governments are build the data centre to store
the huge data with the purpose of storing millions of fingermarks sets and return tax.
Nowadays the technologies in the big data are included in the various ways such as cloud
computing, business intelligence, stream processing, distributed file systems, In-memory
data fabric, etc.
Information Systems and Big Data Analysis
What is big Data
The big data is referred the managing the data set complex and
large that traditional processing of application software's is adequate to manage, capture,
curate and operation the data with the re-seasonable quantity of time. It can be used for
the user behaviour analytics and predictive. Big data is contained the structure of the data
only with the various data types such as cloud storage services, Hadoop clusters and other
big data platforms. it is the combination of the structured, unstructured and semi-
structured data which are collected by the business and used in the analytical applications,
machine learning projects and other predictive modelling. The company and organization
are use the big data in their application systems to provide better customers services,
create marketing campaigns, enhance revenues and profits and improve the operations of
organization.
The big data is important for the organization and enterprise
because it is a data with the huge size and volume. There are various examples of big
data such as social media and stock exchange, etc. the big data include the three types that
include the structured, unstructured and semi structured. in the context of structured, any
types of data can be accessed, processed and store in the fixed format form is known as
the structured data. In the terms of unstructured data, it is the heterogeneous data and
unknown form is classified as the unstructured data. In the context of semi structured
data, it contains the both forms of data such as fixed format and unknown format.
Characteristics of Big data
The big data have the five's characteristics such as volume, velocity, variability, variety and veracity. The big
data are measured by the five characteristics.
Volume: The volume is referred as the total amount of the store data and it also refers as the size of data because
the size of data are play an important roles for determining the value of data. It is need to be considered while
dealing with the big data solution.
Variety: The range of data sources and types. It is referred as the heterogeneous sources and contain the both
data such as structured and non structured data. The all data are considered in the data analysis such as audio,
videos, photos and emails. The various ways of unstructured data poses the certain issues for mining and
storage.
Velocity: Velocity is referred as the sped that data is proceeded and generated and speed of generation of data. It
determines that how fast data is proceeded and generated o meet the needs and demands. The flow of data is
continuous and massive and data flow sources like networks, social media sites, application logs, sensors,
mobile device, etc.
Variability: The variability is referred as the lack of consistency among data proceed and gathered. It is referred
as the inconsistency which can be shown by data and constraint the process the manage and handle the data
efficiently and effectively.
Veracity: It refers as the data quality for the accurate analysis and degree of accuracy in the set of data.
Collected data form the various sources can cause the data quality issues and it may be difficult to moment. The
data analytics and management teams require to ensure that they have the correct and accurate data accessible to
produce right results.
The challenges of big data analytics
There are various challenges of big data in data analytics which are face
such as data security, complexity, integration, capture, scale, data mobility, data values, etc.
Data security: The collecting and storing the information is not enough because the data must be
secured at any cost because it is the sensitive and important for the business. With the helps of
creation of technologies, the data is stored in the cloud platforms but the security of data are most
important so, it provides the features to protect and secure the data.
Data complexity: The data flowing is the complexity on the regular basis and it can be in the form
of various sources such as consumers, operations and sales person in the organization. There are
various technologies and that functions are increase the new level of complexity and challenges of
big data.
Data mobility: it is also the challenges for the big data because it can be traced back to mobility
data with the worldwide network so, the data must be collected and analysed.
Data integration: Data integrations is the challenges for the big data because the data is collected in
the numerous sources and it's hard to supervisor the effectualness of the integration process.
Data capture: The Data capture is the biggest challenges because the large size and volume of data
capturing is the difficult and storing and managing are most challenging for the enterprise and
business. Capture the data analytics are most essential for the business to analyse the performance
and position in the market, and they can handle the large volume of data easily and effectively.
Data value: it is also the challenges because the value of the data are most important in the business
and organization. The data are stored for the long term of periods so the data values are occurred
challenges and data analytics have a tendency effect.
References
Carpo, M., 2019. Big Data and the End of History. International Journal for Digital Art History: Issue 3, 2018: Digital Space and
Architecture. 3. p.21.
Ghasemaghaei, M. and Calic, G., 2020. Assessing the impact of big data on firm innovation performance: Big data is not always
better data. Journal of Business Research. 108. pp.147-162.
Hariri, R.H., Fredericks, E.M. and Bowers, K.M., 2019. Uncertainty in big data analytics: survey, opportunities, and
challenges. Journal of Big Data. 6(1). pp.1-16.
Palanivel, K. and Surianarayanan, C., 2019. An approach for prediction of crop yield using machine learning and big data
techniques. International Journal of Computer Engineering and Technology.10(3). pp.110-118.
Qi, G.J. and Luo, J., 2020. Small data challenges in big data era: A survey of recent progress on unsupervised and semi-supervised
methods. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Wang, J., Yang, Y., Wang, T., Sherratt, R.S. and Zhang, J., 2020. Big data service architecture: a survey. Journal of Internet
Technology. 21(2). pp.393-405.
Younas, M., 2019. Research challenges of big data. Service Oriented Computing and Applications. 13(2). pp.105-107.
Zhu, L., Yu, F.R., Wang, Y., Ning, B. and Tang, T., 2018. Big data analytics in intelligent transportation systems: A survey. IEEE
Transactions on Intelligent Transportation Systems. 20(1).pp.383-398.
How Big Data technology could support business & Examples
The big data technology used in the business helps in increase the efficiency and effectiveness of
business and bid data on the business environments. The big data technology applications in
business that includes the understand the better customers, know competitors in better ways,
implement the risk management, identify the trends and personalize marketing. It helps in store
and manage the large volume of data and information of the business with the better security as
well as collect and analyse the information to make decisions. The big data in the business
provide the management with information to analyse and identify the trends and positions in the
competitive market place. The big data technology helps in manage the human resources because
its hire the best people as well as understanding and improving the work force. The tools of the
bid data helps in safe the data of business as well as analyse the internal threats.
……………………………………………………………………………………………………………………………………………………………………………………...
Techniques that are currently available to analysis big data
There are various techniques which are available to analysis the big data such as machine learning,
sentiment analysis, genetic algorithms, social network analysis, association rule learning, classification tree analysis and regression
analysis.
Machine learning: The machine learning is the best technique for the big data analysis because it considers software that can acquire
form data. It helps in distinguish the spam and non spam email messages, it helps in make recommendation on the specific information
as well as find the top-quality content for attractive the consumers.
Association rule learning: It is the method of discovering the absorbing associate between variables in big database. It was used in the
supermarket chains for discover the relations between the point of sales and products. It helps in used to examine the biological data to
bring out the new relationships, it helps in order to increase the sales of products and services, monitor the systems logs to detect the
activities such as malicious and intruders activities. It also helps in extract the visitor's information and data content to websites from
the network server logs.
Classification of tree analysis: the methods of statistical classification for identifying the characteristics that a new observations
belongs and training set are also require to identify the observations. It is used for categorize organisms into the groupings, modify
depute documents to aggregation and develop the profiles of students and children's who proceeds online education and courses.