Data merging and Cleaning Assignment PDF

Table of Contents
1. Introduction................................................................................................................................2
2. Data merging and cleaning.......................................................................................................2
3. RESTful web services................................................................................................................4
4. Mashups:....................................................................................................................................5
5. Demo Running Instruction:.....................................................................................................5
Combine the two files....................................................................................................................5
6. Conclusion:.................................................................................................................................9
References.......................................................................................................................................9
1

1. Introduction
Data and system integration has combined data from different sources, the technologies
are stored in various sources and data has unique view. System integration as bring the
component of sub-systems from one aggregation of cooperating subsystems so the system have
able to deliver an overarching functioning and ensure a subsystems function. In data and system
integration the mashups and restful services concept has done in application approach that can be
allows multiple services in one users, for each services it’s have own goals, the purpose as serves
to create a service. Restful services is a style of architecture to design a loosely coupled web
services. It main purpose is to develop a fast and lightweight service. Rest is a distributed
hypermedia application.
2. Data merging and cleaning
In data merging and cleaning the PETL and ETL library package are used in python
programming. PETL package is mainly to extracting, transforming and loading a tables to the
data. The package has no dependencies during writing compare to other python core modules,
the installation and maintenance. It has more third party packages that may be useful for PETL
library. ETL has three process Extract, transform and load. ETL has collection of data and
various reports from one data store, analysis. ETL can also implemented by scripts and ETL tool.
In data cleaning ETL has implement by data generate to create more formats like JSON, XML or
CSV. Data cleaning is changing information in a database to check whether it has correct and
standard data. ETL process is a main part of the cleaning. The ETL system has combined with
many information sources and have representation has repeated data. This approach must have to
detect and recover all errors and both in single information resources have to remove the
unwanted information. Data merging in ETL process have wants to identify related sub processes
into ETL process. The same flow have common sub processes at any stage. The sub processes
have to rewrite an ETL processes. The common steps in ETL processes has less possible. Using
transitive closure all individual results are combing. The independent result generates and
produces more accurate result, in lower cost. The rule programming module is provided by the
system. It is easy to find and locate the duplicate environment. Large amount of data is used in
this application. The real world data base is done by data merging. The final result generates the
statistical data.
2

PETL python package
Petl is a python package index. The following command is used to describe the pip
$ pip install petl
And to download manually, extract and run by following command
python setup.py install
To verify the installation following command is used
$ pip install nose
$ nosetests –v pet1
We are using the python version 2.7 and 3.4. The UNIX and WINDOWS operating
system is used to perform python.
ETL pipelines
Using this package we can easily avoid the lazy evolution and iterations. The pipeline
will not execute accurately, until the data is required.
For instance
>>> example_data = """foo,bar,baz
... a,1,3.4
... b,2,7.4
... c,6,2.2
... d,9,8.1
... """
>>> with open('example.csv', 'w') as f:
... f.write(example_data)
...
petl.util.vis.look() is a calling function. Using this function easily write the data and files
or database.
3

Following codes are some examples
petl. Io csv. tocsv()
petl.io.db.todb()
Table containers are used to perform the data extraction. Each table contains table
containers and table iterations. First we need to accept the requested data otherwise the actual
transformation is not done. All the transformations are run using pipeline.
3. RESTful web services
REST defined as a basic interface that can send a data over a systematize interface
without of other message layers such as SOAP. REST is not a tool to gives a set of order to
design a stateless service to create that view like sources of the data. RESTful web service are
produced by REST principles such as to design on the web. The HTTP method is used in rest
web service. The service often use the Uniform Resource Identifier it gives as service to define
the methods of JSON and HTTP. By implementing RESTful services in python by flask it work
is simple to implement and simple to use and they have not any other extension. To use the
resources the operations CURD will be implement.
The goal of a RESTful service is a resources they gives a permission to access a
resources. They offers an interface in programmatic for web app. The single user offers a
functional process and other kind of third party have should offered in UI services. The
communication of the protocol have been implemented, but HTTP have de facto.
Bottle library packages in python programming has a WSGI micro web framework in
python. It is mainly designed for basic process, and improve it speed and its separate a single file
module without dependencies compared to other python library. They sends a request to URL by
its support of templates, and third party adapter and template engines.
Python standard library is briefly providing a huge range. The library have its own
modules that as given to use its functions of input and output file. The library has methods and
uses that can use to done all actions. In this library we need not to import the statement.
4

4. Mashups:
A mashup is a good web application that has combination of data and functionality from
many other external sources and finally it create a good service. The mashups one can use any
easily, integration speed have increase, using of APIs and to produce a results from data sources
they as no reasons to produce an original data from source. The mashups content are used from
different sources for displaying and creating a new service. It have advanced in the web
technology. The companies have shown the information for using other sources so they have
shared their own data without any permission. The map mashups is a major mashup. The
programmable web have traced the map to the list of comprehensive. The main example of
mashup is creating a map mashup by a google map that shows you an address and location. In
data integration they are many types of mashups that are performing in the system are URL
mashups, HTML mashups, data mashups custom mashups.
The example of data mashups is a purchase funnel. In marketing field the funnel are used
to handle their customer for taking measure at every stage and testing the people whether move
to other stage.
The work of the example is the data of google analytics and salesforce are two source
keys. It can take the google drive spreadsheet for the each stage to describing the team in the
funnel.
5. Demo Running Instruction:
Combine the two files.
Code explanation:
The python program that is “dataMerger.py” is used to merge given content and the files are
imported by using the keyword “import” and every attributes in the codes are used to form the
tree and the web service side the python program that is “clinics_locator.py” used to search and
locate the address in the nearest tab.
5

Restful web services:
Code Explain:
Which was carried out to the execution of python files and save the location on .csv files.
For show the results of the operation “import csv” was used. For opening the information
“storeopen()” was used. To read the file we need to use “StoreFileReader()” was used. For
length checking purpose we need to use the “If(len !=row)”. To increase the no of rows we need
to use “ScoreList[]=ScoreList[]+row”. For exiting from the file we need to use
“ScoreFile.close”.
6

7

To search the Location:
Displayed the Location (Google Map):
8

Code Explain:
The clinics_html file used to view the exact geolocation and direction of the position of the
clinic wants to know. Here we can able to see the MAP which contains the direction for the
clinic. It very useful show the location of the clinics services location easily.
6. Conclusion:
The position of the stores in the MAP was identified. The IT structure are mainly used to
access the data centers and based on the Functionality of the dependent on the type of the
Infrastructure. Growth of the process is slowly increasing and non – dynamic. The techniques are
used to compute the responsibilities of the system. The system integration of various data the
final required data was recovered. Scalability of the system was ensured by the virtualizing
techniques. And finally the integrating the information, demonstrations also performed.
References
Dong, X. and Srivastava, D. (n.d.). Big data integration.
Finkelstein, C. (2006). Enterprise architecture for integration. Boston: Artech House.
Haltiwanger, J., Lynch, L. and Mackie, C. (2007). Understanding business dynamics.
Washington, D.C.: National Academies Press.
SAEKI, M. and SUGITANI, Y. (2011). Partial Tuning of Dynamical Controllers by Data-Driven
Loop-Shaping. SICE Journal of Control, Measurement, and System Integration, 4(1), pp.71-
76.
Wang, X., Shen, J. and Sun, C. (2013). Data Warehouse Oriented Data Integration System
Design and Implementation. Applied Mechanics and Materials, 321-324, pp.2532-2538.
Zhai, L., Guo, L., Cui, X. and Li, S. (2011). Research on Real-time Publish/Subscribe System
supported by Data-Integration. Journal of Software, 6(6).
Huang, X. and Zhu, W. (2013). An Enterprise Data Integration ERP System Conversion System
Design and Implementation. Applied Mechanics and Materials, 433-435, pp.1765-1769.
9

ISHII, H. and TEMPO, R. (2009). Computing the PageRank Variation for Fragile Web
Data. SICE Journal of Control, Measurement, and System Integration, 2(1), pp.1-9.
Joglekar, A. (2016). Prediction of Favourable Rules to Identify Suspected Patients of HIV Using
Integration of Expert System and Data Mining. International Journal of Mechanical
Engineering and Information Technology.
Kaps, A., Dyshlevoi, K., Heumann, K., Jost, R., Kontodinas, I., Wolff, M. and Hani, J. (2006).
The BioRSTM Integration and Retrieval System: An open system for distributed data
integration. Journal of Integrative Bioinformatics, 3(2).
Mynarz, J. (2014). Integration of public procurement data using linked data. Journal of Systems
Integration, pp.19-31.
Oró, E. and Salom, J. (2015). Energy Model for Thermal Energy Storage System Management
Integration in Data Centres. Energy Procedia, 73, pp.254-262.
10