MITS5002 Assignment: Software Engineering for Data Warehouse Systems

This presentation provides a comprehensive overview of software engineering methodologies applied to data warehouse systems. It begins by defining different software development approaches such as Waterfall, Agile, and Spiral methodologies. The core of the presentation focuses on data warehouse systems, detailing their role in business intelligence, components like databases, ETL tools, metadata, query tools, and data marts, and the three-tier architecture. It further explores the attributes of an efficient data warehouse, including integration, time-variance, and non-volatility, while also discussing the issues encountered in data warehouse projects, like uncertain requirements and rigid development phases. The presentation then highlights the qualities of a robust software development process, such as reliability, robustness, productivity, and timeliness. It introduces emerging methodological principles like incremental processes, user involvement, and component reuse. The presentation concludes with a discussion of the Four Wheel Drive (4WD) design activities, covering architectural sketches, conformity analysis, data mart prioritization, and design, along with the outcomes of applying these principles, such as risk-based iterations, prototyping, and user engagement. The presentation is supported by several references to academic papers, and it is intended for the MITS5002 course on Software Engineering.

SOFTWARE
ENGINEERING
FOR
DATA
WAREHOUSE
SYSTEMS

Software Engineering
Methodologies
The different methods or approaches by which a software is developed and
delivered for the users are:
 ‘Waterfall Development’
 ‘Rapid Application Development’
 ‘Prototyping-Oriented Software Development’
 ‘Spiral Software Development’
 ‘Model-Driven Architecture’
 ‘Component-Based Software Engineering’
 ‘Agile Software Development’

Data Warehouse Systems
 A model of Business Intelligence Systems
 Helps in generating reports on business
entities on a regular data.
 Does not incorporate the same
performance on real-time basis.
 Provides for a central repository to store all
data (Mercado et al. 2018).
 Incorporates two-way communication with
all operations in a business.
 Adheres to all types of changes in a
business.

Data Warehouse Constituents
 Database- Foundation for the
warehouse environment, provides the
framework to store data.
 ETL Tools- Use to extract data,
transform them and load it back in the
database, removes duplicate data and
deletes unwanted data.
 Metadata- The data that defines the
warehouse and contains the technical
specifications about it.
 Query Tools- Allows to interact with
the database systems, can be classified
into reporting tools, data mining tools,
OLAP tools and app development tools.
 Data Marts- Provides as a subsidiary
of warehouse and an access layer by
which users can access data.

Data Warehouse Architecture
A system of data warehouse incorporates
an architecture of three-tier :
 Bottom Tier- Incorporates the server
of database, it is the RDBMS, that use
the tools in the backend to extract,
transform and load data.
 Middle Tier- Incorporates the server
of OLAP that can be implemented
either by multi-dimensional OLAP or
relational OLAP.
 Top Tier- The layer of front-end client
that embraces the four types of query
tools.

Attributes of an Efficient Data
Warehouse
 Integrated- Data Integrity is always
maintained irrespective of any kind of
operation on data.
 Time-Variant- Organization of data
is based on time(daily, weekly,
monthly or yearly).
 Non-volatile- Momentary changes in
data does not affect the system
because data is not uploaded in real-
time.

Issues in Data Warehouse
Projects
 After data analysis, requirements from
the data warehouse are uncertain and
unclear.
 Fast change in requirements due to
evolution of business activities.
 Incorporates development of one data
mart at a time (Moscoso-Zea, Paredes-
Gualtor and Luján-Mora 2018).
 Development of the data mart follows a
linear approach, thereby the
organization of the phases are rigid.
 Issues in data integration and
performance optimization.
 Inadequacy of the systems.

Qualities of Software Development
Process
 Reliability- The process should be reliable
enough to meet the accurate user requirements.
 Robustness- The process should be flexible
such that it can adapt to any changes without
much error (Ferreira, Almeida and Monteiro
2017).
 Productivity- The process should be productive
such that it can make the maximum use of
resources.
 Timeliness- The process should maintain the
schedule accurately, thereby minimizing
wastage.

Emerging Methodological
Principles
 Incremental Process and
Risk-Based Iteration
 Prototyping
 Involvement of Users
 Reuse of Components
 Readable Documentation
 Transformation of
Automated Schema (Asrani
and Jain 2016)

Four Wheel Drive Design
RELIABILITY ROBUSTNESS PRODUCTIVITY TIMELINESS
Incremental Process Clear
specifications,
constant review
Better change
management
Optimized
management of
resources
Detection of errors
at initial stage
Prototyping Frequent testing,
easy finding out
errors
Faster delivery
User Engagement Improved data
quality
Detection of errors
at initial stage
Reuse Components Robust
components
Speedy design Development can be
seen through.
Documentation Readable and
clear data
Stress-free evolution Speedy design
Automated Schema
Change
Optimized
performance
Stress-free evolution Speedy design Design is predictable

Activities in Data Mart Cycle
 Architectural sketch- The
functional and non-functional
requirements are fitted in the data
warehouse design.
 Conformity analysis- Confirmation
of the hierarchies with respect to
data and schema (Vieira et al. 2018).
 Data mart prioritization- Decided
on the trade-off between the
technical constraints and priorities of
the user.
 Data mart design- Releases the
most optimized data mart amongst
all.

Outcomes of Application on 4WD
 Incrementality and Risk-Based
Iterations are based on the risks. The
earlier the risks are detected while
designing the data mart, the faster
design of the data mart is developed in
an optimal way.
 The process of prototyping involves a
large effort starting from data flow to
front-end by ETL and thus, helps the
architects to validate the requirements
and understand the hierarchies well
(Gurcan and Cagiltay 2019).
 Engagement of the users are a
necessity and thus can be done in the
process of prototyping by clarifying
project goals and explaining multi-
dimensional models.

Outcomes of Application on 4WD
 Reusing components such as
library facts, analysis templates,
ETL blocks, confirmed and library
hierarchies are a necessity to pace
and ensure error-free development.
 The formal documentation for the
development of a data warehouse
will help to promote clarity in
requirements, transparent
communication amongst designers
and users, robust design and
enhanced maintenance.
 Encouragement of automated
transformation of schema to
simplify the process of evolution
and reduce complexity in designs.

References
 Asrani, D. and Jain, R., 2016. DESIGNING AFramework TO STANDARDIZE DATA WAREHOUSE
DEVELOPMENT PROCESS FOR EFFECTIVE DATA WAREHOUSING PRACTICES. International Journal
of Database Management Systems, 8(4).
 Ferreira, J., Almeida, F. and Monteiro, J., 2017. Building an Effective Data Warehousing for
Financial Sector. arXiv preprint arXiv:1709.05874.
 Gurcan, F. and Cagiltay, N.E., 2019. Big data software engineering: Analysis of knowledge
domains and skill sets using LDA-based topic modeling. IEEE Access, 7, pp.82541-82552.
 Mercado, K.E., Perez, C.B., Lopez-Arredondo, L.P., Caro, K., Castro, L.A. and Rodriguez, L.F.,
2018. Agile Dimensional Model for a Data Warehouse Implementation in a Software Developer
Company. Research in Computing Science, 147, pp.27-34.
 Moscoso-Zea, O., Paredes-Gualtor, J. and Luján-Mora, S., 2018. A holistic view of data
warehousing in education. IEEE Access, 6, pp.64659-64673.
 Vieira, A.A., Pedro, L., Santos, M.Y., Fernandes, J.M. and Dias, L.S., 2018, October. Data
requirements elicitation in big data warehousing. In European, Mediterranean, and Middle
Eastern Conference on Information Systems (pp. 106-113). Springer, Cham.