Data System and Integration: A Study on Data Cleaning, Merging, RESTful Web Services, and Mashups

This study focuses on data and system integration, particularly on data cleaning, merging, RESTful web services, and mashups. It discusses the key concepts, functions, and tools used in the process, including R function, Python, petl, parser, bottle, HTML, and CSS. The study also includes a demo running instruction for combining two files, finding RESTful web services, searching and displaying the location using Google Map. The conclusion highlights the successful performance of the integration system.

DATA SYSTEM AND
INTEGRATION

Abstract
The main objective of this project is to provide the web page in the way of accepting the
user queries and the process will be done based on the queries. The user input is a kind of
service or postal code. From this code, our application will display cluster map of clinics to the
user. This project contains HTML, python, petl, parser, bottle and some other technologies.
They are required for the proper functioning of this application. The demonstration code
contains four data sources and also the sample data. The clinics.csv files contain the data about
required clinics. The XML file contains the location details about each clinic. The third CSV files
in the way of services.csv contain the details about several clinical services. The last CSV files
contain a list of clinics and the services. Data integration carried out by using the python scripts.
Usually, this script file performs the cleansing operation first and then the merging operation. In
merging operation, it merges the four data sources into one CSV file. The CSV file contains
several attributes in the way of service ID, service, Email, clinic ID, clinic, suburb, lat and so on.
This application also shows restful web service demo. Here also we use the python script. This
web service should accept clinic query from the user and provide appropriate service to the
user. Then we provide mashup demo that contains HTML and CSS files. It includes error
handling mechanism to handle any error when something went wrong in the process.

Table of Contents
1.0 Introduction...........................................................................................................................1
2.0 Key system concept............................................................................................................1
2.1 Data cleaning and merging............................................................................................1
2.2 RESTful web services.....................................................................................................2
2.3 Mashups.............................................................................................................................2
3.0 Demo Running Instruction:...............................................................................................3
4.0 Conclusion.............................................................................................................................6
References....................................................................................................................................7

Table of figures
Figure 1 Combine two files..............................................................................................................3
Figure 2 Find out the restful web services.......................................................................................4
Figure 3 To search the location.......................................................................................................5
Figure 4 Display the location using Map..........................................................................................5

1.0 Introduction
Data and system integration was the set of processes carried out to enhance the
business by integrating the technical stuff with the business. This integration process is used to
collect different data from the different reliable data sources. This process delivered the output
like data with confidentiality and the main use of this system is to combine the sources for the
vision. Data and system integration is used to analyze the business work with technical by using
the data warehouse. It is used to the operation in the kind of mapping and validation. From the
mapping, we can analyze the system integration and usage of the data warehouse. In that the
information clearly defined for the vision. The map operation and data integration are analyzed
well. Each data in this warehouse should have the name for identification (Kaps et al., 2006).
Normally the data integration analyzes the requirements and it could be functional or
nonfunctional. The system integration mainly concerns about the business development.
2.0 Key system concept
Generally, the system is defined by the collection of elements. The relationship
between two elements is derived by using the key system concept. It mainly focused on
interactions and holism. In this analysis there are some complex issues involved. In this process,
the open system is described based on the relationship between two system elements. Each
system elements has the system boundary. The interaction process is done by the system
elements (SAEKI and SUGITANI, 2011).
2.1 Data cleaning and merging
The concept of this process has two major issues. One is data cleaning and another one
is data merging. Sometimes the user wants to collect the data from multiple files. This
operation is carried out by R function. Generally, the command is used for executing these kind
of operations like VLOOKUP in excel.
1

Functions
Take an example we have two data sets, in that the first data set is sales and the second
data set is customer’s details. The files contain the variables in the way of date, id, product id,
and sales. All these things are load into R, under the name of ‘my data’. And the second one is
customer’s details, these files contain some variables like id, age, and country. These also load
into R, in the same name and location.
2.2 RESTful web services
REST stands for Representational State Transfer. REST is defined by an architectural
style, and it is not a protocol. RESTful web services have many advantages and they are
described below.
 FAST- basically all the RESTful web services has high accessing speed. Because in this
architecture style, there was no strict specification for SOAP. The most important thing
in this protocol was its less bandwidth, also it contains the minimum number of
resources.
 Language and platform independence- it supports all kind of programming language as
well as support all kind of platform. Because RESTful was known as platform
independent web service.
 Use SOAP- normally the RESTful web services are using SOAP web services for the
implementation purpose.
 Use different data formats- it allows different data formats like plain text, HTML, XML,
and JSON (Vinoski, 2008).
2.3 Mashups
2

Mashup is known as the technology and it used to develop the application created by
the user. And mashup used in the data and system integration to collect the information about
the business and customers. And using the programming language knowledge used to make the
good business configuration and also used as application programming interface for the
business development. And also it is used to recognize about the mashup configuration. It has a
logic and it is used to analyze the mash-up operators and Components construction. It had
many tools to describe the operations. Here the data & logics are used to specify the
components. The work progress of the system integration was analyzed by the use of mashup.
Script language was used for the process mapping. And the script language is used for the
dynamic execution (Wang, Shen, and Sun, 2013). And the mashup technology represented as
the kind of gadgets, widgets and the supporting language such as HTML and CSS.
3.0 Demo Running Instruction:
Combining two files:
Explanation about code:
The python program named as “dataMerger.py” is used to merge the given content and
the files were imported by using the keyword “import” and every attributes in the codes are
used to form the tree and In the web services the python program named as
“clinics_locator.py” and it is used to search and locate the address in the nearest tab.
3

Figure 1 Combine two files
Restful web services:
Explanation
Restful web services has contains the execution of python files and save the files in the
location such as csv. And the operation “import CSV” is used to show the results. The function
“Clinic open()” was used to open the information. To read the file we need to use
“ClinicFileReader()” function. For length checking purpose, we need to use the “If (Len! =row)”.
To increase the no of rows we need to use “ClinicList[]=ClinicList[]+row”. For exiting from the
file we need to use “ClinicFile.close”.
4

Figure 2 Find out the restful web services
5

To search the Location:
Figure 3 To search the location
Displayed the Location (Google Map):
Figure 4 Display the location using Map
Code Explain:
The clinics_html file is used to view the exact geolocation and direction of the position
of the clinic who wants to know. Here we can able to see the MAP which contains the direction
for the clinic. And it very useful show the location easily of the clinic's services.
6

4.0 Conclusion
The Integration System has contains the various data and these datum are finally
recovered by using the Python code. The Virtualizing techniques are known as Scalability
system. Finally, the position of the clinic data viewed in Google Map was identified successfully.
The Structure of IT mainly used for the data Access and Functionality based dependent on the
Infrastructure. If the process growth is dynamic it increases slowly. Finally, the integration
system was performed successfully.
7

References
Kaps, A., Dyshlevoi, K., Heumann, K., Jost, R., Kontodinas, I., Wolff, M. and Hani, J. (2006).
The BioRSTM Integration and Retrieval System: An open system for distributed data
integration. Journal of Integrative Bioinformatics, 3(2).
SAEKI, M. and SUGITANI, Y. (2011). Partial Tuning of Dynamical Controllers by Data-Driven
Loop-Shaping. SICE Journal of Control, Measurement, and System Integration, 4(1), pp.71-76.
Vinoski, S. (2008). RESTful Web Services Development Checklist. IEEE Internet Computing,
12(6), pp.96-95.
Wang, X., Shen, J. and Sun, C. (2013). Data Warehouse Oriented Data Integration System
Design and Implementation. Applied Mechanics and Materials, 321-324, pp.2532-2538.
8