Development of RESTful and Mashup application for Data and System Integration

This report discusses the computing and storage infrastructure design, application/service integration, and information integration for data and system integration. It also includes a demo introduction and conclusion.

Development of RESTful and Mashup application
Name of the Student
Name of the University
Authors note
P a g e

DATA AND SYSTEM INTEGRATION
Table of Contents
Introduction...............................................................................................................................1
Computing and storage infrastructure design...........................................................................1
Application/service integration..................................................................................................2
Information integration.............................................................................................................2
Demo introduction.....................................................................................................................3
Conclusion..................................................................................................................................5
Bibliography...............................................................................................................................6
P a g e

Introduction
As the GE has decided to use the SOA therefore it is important to determine the
computing, storage infrastructure design requirements, data as well as application
integration to have a unified view of the data collected from multiple data sources and are
stored in different formats.
The different sections of this report consist of discussion about the probable
solutions for the storage design, data integrations that will help them in improving the
processes that requires data collected from sources.
Computing and storage infrastructure design
As the organization is operating at different locations and requires a distributed
computing and storage solution for all its IT needs. Thus, considering cost and effectiveness
of different solutions it can be stated that, the organization should use rented services from
the cloud services.
The reasons behind this can be stated as the storage and the computing power of
public clouds are available on internet. In addition to that, the service providers provide
technical expertise for the deployment thus in other words the public clouds are easy to
deploy.
The public cloud service providers operate at quite good speed due to the
optimization provided by the providers, which is also alluring to some enterprises. Thus
require lesser amount of time is required for deployment. The services as well as resources
are accessible quickly that saves time for the organization which may be required to develop
and deploy the private cloud solution.
In addition to that the adoption of the public cloud for the organization frees the
organization from the responsibilities of maintaining the software and hardware systems.
This maintenance is managed by the service providers. The IT staffs of the organization does
not have to manage the cloud framework.
No long term agreements are required for service providers. Most of the cloud
service providers provides their services depending on the pay-as-you-go models. Thus the
organisation have to pay according to their usage of computing resources and storage. In
addition to that, the organization will get more resources whenever they require it for their
business. Thus the public gives the organization more flexibility to the organizations to
manage the resources and storage as per their requirement.
With all this advantage there are certain issues that needs to be kept in mind while
renting some public cloud service. One of the most important issues that should be checked
is the reliability of the services. As the data centres of the service providers are less secure
compared to the private clouds.
P a g e

DATA AND SYSTEM INTEGRATION
Application/service integration
The organizations consist of numerous applications or systems that are used by
different departments of the organization, which provides different services required every
day the organization relies upon to complete their day to day business operations.
This applications or systems may be licensed from a third party vendor or developed
in-house. Thus, integration of this system or applications are important in better
coordination of different operations of different departments of the organization.
Mashups are one of the best example of application integration on the internet.
This type of web application is capable of bringing together numerous sources of data in
order to form and represent some unique combination of meaningful data.
In this project a google map mash up application is developed which will help in
finding and plotting marker on the location of the clinics that are provided in the merged
data source “clinic_location.csv”. The RESTful web service will be responsible for retrieving a
Json object containing the location details as well as other details about a clinic in a specific
region. This mashup application will use ajax calls in order to mark the location using a
marker on the google map.
In order to map a specific location on the google map, it is important to know the
specific longitude and latitude of that area or location.
This mash up or integrated applications are important for following reasons,
Interoperability- It is the most important aspect of the application integration.
Various modules of the employed infrastructure may be deployed on different data formats,
operating systems as well as programing languages thus may prevent connection for other
services using a standard interface.
Data integration for the different applications or systems- For any distributed, and
modular system used in the organization, a standard process for handling the data flow
between different applications is important as it enforces consistency across the database
or the format of the data that is used by different applications.
Stability, Scalability and Robustness- This three attributes acts as vital factors that
impacts on the performance of the system modules that holds the infrastructure. The
integration of the applications must be stable, robust as well as scalable to meet the
requirements of the organization.
Information integration
Presently there are mainly three approaches are used for data integration. Which
are, schema integration, procedural approach and lastly the query oriented approach.
In case of the schema integration based approach, input to this process is a pool of
data sets. Every data set is constituted by actual data and a schema that defines the
structure of the dataset. The main goal is to deliver a reconciled and unified view of the
collected data from the different sources. This process does not interfere into the autonomy
of the datasets as well as their sources. This process deals with the read-only integration of
P a g e

DATA AND SYSTEM INTEGRATION
the data. Therefore, this a reconciled view of the unified dataset is only used for answering
user queries and not for updating the records.
In case of the procedural approach, data collected from different sources are
integrated in an ad-hoc manner according to the set of predefined information needs of an
organization or user.
In procedural approach the integration of data is dependent on the two components
mediators and wrappers. Wrappers are useful in encapsulating the sources of the data. In
addition to that it also helps in the conversion of the core data objects into some common
data model.
The mediators are required in obtaining information from a single or multiple
wrappers. The mediators can obtain or extract data from other mediators. After this stage
the mediators refines the obtained information by resolving conflicts among the different
pieces of information collected from numerous data sources. The refined data is then
provided to either to the other mediators or to the users.
The last approach is query-oriented approach. In this approach, the goal is
modeling of the collected data at the sources by using some suitable language. This can be
extended SQL or some kind of declarative logic-based language. In order to build a unified
representation or view of the collected data and refer to such a demonstration at the time
of querying any global information system. In addition to that deriving query answers by
using some suitable mechanisms to access accessing the data sources or to the materialized
views of the unified data.
Demo introduction
For this project there are two data files (clinic.csv and another is location.xml) are
available that needs to be cleaned, merged and integrated to get some meaningful data. As
two data files are in different formats. Therefore, we used xml, sys and Petl packages in
Python. We first converted the xml file into csv data set so that it can be used with clinic.csv
file.
We used following code snippet for this conversion of the dataset.
import petl as etl
from xml.dom import minidom, Node
import sys
def scanNode(node, file=sys.stdout, level = 0):
if node.hasChildNodes():
for child in node.childNodes:
scanNode(child, file, level + 1)
P a g e

DATA AND SYSTEM INTEGRATION
else:
if node.nodeValue.strip() != "":
file.write(node.nodeValue + ',')
if node.nodeType == Node.ELEMENT_NODE:
if node.hasAttributes():
file.write(node.attributes.item(0).nodeValue + ',')
if level == 2:
file.write("\n")
DOMTree = minidom.parse('locations.xml')
file = open('locations1.csv', 'w')
file.write("Name,Suburb,Lat,Lon\n")
scanNode(DOMTree, file)
file.close()
table1=etl.fromcsv('locations1.csv')
print (table1)
table2=etl.fromcsv('clinics.csv')
print (table2)
table=etl.join(table1,table2,key='Suburb')
P a g e

DATA AND SYSTEM INTEGRATION
print (table)
etl.tocsv(table,'clinic_locations1.csv')
after the merger the cleaned data looks like following,
In order to get results from the RESTful web service demo, at first we have to
execute the clinic locator.py program using any python IDE so that the web service can
retrieve the data from the clinic_locations.csv data file.
The successful execution of the web service server is shown below,
P a g e

DATA AND SYSTEM INTEGRATION
After the web service is started, mashup application needs to be launched which is
the “clinic_map.html”. on this page the user needs to input a postal code in which they
want to find the clinic.
If the web service finds any matching data in the merged csv file that consist the
similar postal code, then the web service returns a JSON data object to the mashup
application which then consumed by the application to plot a marker on the google map.
Conclusion
The system and data integration is important in the present scenario for the
organizations and users as this integration of applications are helpful in providing a unified
view of the data collected from numerous sources and formats.
The web service application and the mashup application developed in this project is
helpful in providing a unified view about the different details of the clinic on the map.
P a g e

DATA AND SYSTEM INTEGRATION
Bibliography
Erlingsson, U., Xie, Y., Livshits, B. and Fournet, C., Microsoft Corp, 2014. Enhanced security
and performance of web applications. U.S. Patent 8,677,141.
Lin, H.Y. and Huang, J.L., 2014, August. A Web API Aggregation Service for Mobile Mashup
Applications. In Intelligent Information Hiding and Multimedia Signal Processing (IIH-MSP),
2014 Tenth International Conference on (pp. 73-76). IEEE.
Zhang, F., Hwang, K., Khan, S.U. and Malluhi, Q.M., 2016. Skyline discovery and composition
of multi-cloud mashup services. IEEE Transactions on Services Computing, 9(1), pp.72-83.
P a g e