Data and System Integration for Desklib

This project for Desklib covers the key concepts of data integration, including data merging and cleaning, RESTful web services, and mashups. It includes a demo running instruction and code explanation, as well as references for further reading.

2018
Data and system Integration

1

Abstract:
The objective of this project is to provide a web page that accepts user’s query and
process the service based upon that query. The user may input a service or postal code. From
this code our application will display cluster map of clinics to the user. This project contains
html, python, petl, parser, bottle and other required technologies for this application. The
demonstration code contains four data sources containing sample data. The clinics.csv files
contains data about required clinics. The xml file contains location details about each clinic. The
third csv files that is services.csv contains details about several clinical services. The last csv files
contains list of clinics and the services they offer. Data integration carried out using python
scripts. This script file first performs cleansing operation then merging operation. In merging
operation, it merges the four data sources to one csv file. The csv file contains several attributes
includes service ID, service, Email, clinic ID, clinic, suburb, lat and so on. This application also
shows restful web service demo. Here also we use python script. This web service should accept
get clinic query from user and provide appropriate service to the user. . Then we provide
mashup demo that contains html and css files. It includes error handling mechanism to handle
any error when something goes wrong.
2

Table of Contents
1.0 Introduction...........................................................................................................................4
2.0 KEY SYSTEM CONCEPTS:................................................................................................4
2.1 Data merging and cleaning:..........................................................................................4
2.2 RESTful Web services:...................................................................................................5
2.3Mashup:...............................................................................................................................5
Merge the two files.........................................................................................................................6
4.0 Conclusion.............................................................................................................................9
References................................................................................................................................9
3

Table of Figures
Figure 1 Combine two files.............................................................................................................6
Figure 2 Find out the restful web services.......................................................................................7
Figure 3 To search the location......................................................................................................8
Figure 4 Display the location using Map.....................................................................................8
4

1.0 Introduction
Large organization definitely processing tons of data for example customer data, financial
data, manufacturing data, and so on. Being able to access all the data is a crucial one. So data
integration plays a significant role in every organization (Wang, Shen and Sun, 2013). Data
integration refers combining several sources and provide unified version of those resources. For
example some companies may need to collaborate each other. Then they combine their
databases. This involves data integration that means several data integrated to provide single
version of those data (ISHII and TEMPO, 2009). Data integration has several categories that
includes core data integration, edge data integration and so on.
There are several examples for core data integration initiatives:
ETL (Extract, transform, load) implementations
EAI (Enterprise application integration) implementations
SOA (Service oriented architecture) implementations
ESB (Enterprise service Bus) implementations
Examples for edge data integration:
Extracting customers list from a host sales force automation application. And then
writing entire results into an excel spreadsheet (SAEKI and SUGITANI, 2011).
In order to manage RSS feeds, creating a script-driven framework.
2.0 KEY SYSTEM CONCEPTS:
2.1 Data merging and cleaning:
It is the main issue when working with data in any context. Removing unwanted data and
merge operation are key concepts of data integration (Huang and Zhu, 2013). Cleaning refers
removing duplicate data, unwanted data when we integrate data from multiple sources.
Two different possibilities for when we find duplicate data:
Same data have been entered more than once.
5

Different data have been entered using same identifier
Now there are several ways available to use merge command:
For example
Simple merge on id
Multiple merge on id
We can use data rules to apply when we merging data from multiple resources.
2.2 RESTful Web services:
These kind of web services based on RESTful Architecture. Everything should be
considered as resources in REST architecture (Vial and Resist, 2014). REST refers
Representational state Transfer. Web services usually refers exchanging data between different
applications or systems.
Features:
Light weight
Highly scalable
Generally used to create APIs for web based applications
It uses HTTP protocol for data communication. REST client’s requests and accesses the
resources. REST server provides access to resources. Resource is identified using URIs. There are
several representations available to represent a resources like Text, JSON and XML (Baldoni,
2006). JSON is most popular format used in web services.
2.3Mashup:
Mashup is also a platform or web page where we combines data from multiple sources.
When we use mashups for data integration, then the process will be easy and fast (Lans, 2012).
There several categories available in mashups, like commercial mashups, business mashups,
enterprise mashups. Main advantage for creating mashups was flexibility, supports self-service
application development, help to make service-Oriented Architecture, situational in nature, and
6

simple. Mashup developers need more precise and scalable way to extract, transform and load
between data sources and target (Oró and Salom, 2015). Business mashups usually differs from
consumer mashups with respect to security, environment, access control, and other factors.
3.0 Demo Running Instruction:
Merge the two files.
Explanation about code:
The python program that is “dataMerger.py” is used to merge given content and the files are
imported by using the keyword “import” and every attributes in the codes are used to form the
tree and the web service side the python program that is “clinics_locator.py” used to search
and locate the address in the nearest tab.
Figure 1 Combine two files
7

Restful web services:
Explanation
Which was carried out to the execution of python files and save the location on .csv files.
For show the results of the operation “import csv” was used. For opening the information
“Clinicopen()” was used. To read the file we need to use “ClinicFileReader()” was used. For
length checking purpose we need to use the “If(len !=row)”. To increase the no of rows we
need to use “ClinicList[]=ClinicList[]+row”. For exiting from the file we need to use
“ClinicFile.close”.
Figure 2 Find out the restful web services
8

To search the Location:
Figure 3 To search the location
Displayed the Location (Google Map):
Figure 4 Display the location using Map
9

Code Explain:
The clinics_html file used to view the exact geolocation and direction of the position of the
clinic wants to know. Here we can able to see the MAP which contains the direction for the
clinic. It very useful show the location of the clinics services location easily.
4.0 Conclusion
The Integration System contains the various data, finally recovered the data exactly using
Python code. The Virtualizing techniques is the Scalability system. Finally the position of the
clinic data viewed in Google Map is identified successfully. The Structure of IT mainly used to
data Access and Functionality based dependent on the Infrastructure. If the process growth is
dynamic and increased slowly. Finally, the integration system is performed exactly and also
demonstrations perfectly.
Reference
References
Baldoni, R. (2006). Global data management. Amsterdam: IOS Press.
Huang, X. and Zhu, W. (2013). An Enterprise Data Integration ERP System Conversion System
Design and Implementation. Applied Mechanics and Materials, 433-435, pp.1765-1769.
ISHII, H. and TEMPO, R. (2009). Computing the PageRank Variation for Fragile Web Data. SICE
Journal of Control, Measurement, and System Integration, 2(1), pp.1-9.
Lans, R. (2012). Data virtualization for business intelligence architectures. Amsterdam:
Elsevier/MK.
10

Oró, E. and Salom, J. (2015). Energy Model for Thermal Energy Storage System Management
Integration in Data Centres. Energy Procedia, 73, pp.254-262.
SAEKI, M. and SUGITANI, Y. (2011). Partial Tuning of Dynamical Controllers by Data-Driven
Loop-Shaping. SICE Journal of Control, Measurement, and System Integration, 4(1), pp.71-
76.
Vial, F. and Reist, M. (2014). Evaluation of Swiss Abattoir Data for Integration in a Syndromic
Surveillance System. Online Journal of Public Health Informatics, 6(1).
Wang, X., Shen, J. and Sun, C. (2013). Data Warehouse Oriented Data Integration System Design
and Implementation. Applied Mechanics and Materials, 321-324, pp.2532-2538.
11