SENG6210A: Software Engineering Masters Project Part A: Log Analysis

This document details a software engineering master's project focused on log analysis. It begins with an introduction to logs, their types (event, transaction, message), and how they are generated and used. The project then explores various log analysis tools like Loggly, Logentries, Splunk, logz.io, and GoAccess. The core of the project involves the design and implementation of a Windows-based log analysis application named "Window live logs identification" using C# and .NET. This application allows users to track and filter system logs, identify errors, and analyze event details. The implementation includes code snippets from the application's form, including the .aspx and .aspx.cs files, along with the Web.Debug configuration file. The student used the EventLog class to fetch the logs, and the application provides filtering options for searching logs by event ID, machine name, message, source, or log type. The project's implementation details are discussed, including the use of IIS (Internet Information Services) to run the application on a local machine, along with sample results from the application’s output.

SENG6210A: Software Engineering Masters Project Part A
Project: Log Analysis
SUBMITTED BY:-DALJEET KAUR
STUDENT ID:-c3298354

What is log
A log is detailed writing of tasks done by a computer program.
In computing, a log file keeps the records of operating system’s actions, messages
between different users of a communication software. Logging is the act of keeping
a log. messages are written in a single log file.
In a clear way, logging is just a fancy word to define a process of writing
down everything you do.
There are various types of logging. We can log every operation of an
application, during the occurrence of errors, or in some critical
operations done by a user, especially if someone wants to get control of
who’s doing what for audit purposes.
Applications can log at a code level for debugging or at a user level for
audits and forbidden access register.
Every operating system has routines or services running on the
background with the task of taking notes of everything that is happening.
This way a person can detect security breaches, malfunctions etc.
In simplest way, logging is keeping a record of all data, processes, and
results in a program. This is part of a much more grand, complex
process, though, so you want to program with a clear goal in mind, and
not try to do several programming disciplines at once.
Event logs
it records events taking place in the execution of a system in order to provide
an audit trail that can be used to understand the activity of the system and to
diagnose problems. They are necessary to understand the complexity of system.
It is beneficial to combine log file entries from multiple sources. This approach along
with statistical analysis, may result in correlations between seemingly unrelated
events on different servers.
Transaction log
These logs record changes in the stored data to allow the database to recover
from crashes or errors and maintain the stored data in a stable state. Thus,
database systems usually have both general event logs and transaction logs.
Message logs
Internet Relay Chat (IRC), instant messaging (IM) programs, peer-to-peer file
sharing clients with chat functions, and multiplayer games (especially MMORPGs)
commonly have the ability to automatically log (i.e. save) textual communication,
both public (IRC channel/IM conference/MMO public/party chat messages) and
private chat messages between users. Message logs are almost universally plain
text files, but IM and VoIP clients (which supports textual chat, e.g. Skype) might
save them in HTML files or in a custom format to ease reading and encryption.

How logs are generated and used
A log is the automatically generated when any task or event occur on computer. It is
the time-stamped documentation of events relevant to a particular system. Every
kind of software application even and systems itself produce log files.
For example: -
On a Web server, an access log lists all the individual files that people have
requested from a website. These files will include the HTML files and their imbedded
graphic images and any other associated files that get transmitted. From the server’s
log files, an administrator can identify numbers of visitors, the domains from which
they’re visiting, the number of requests for each page and usage patterns according
to variables such as times of the day, week, month or year.
Log Analysis Tools
As more and more companies move to the cloud, log analytics, log analysis, and log
management tools and services are becoming more critical. DevOps engineers,
system administrators, site reliability engineers, and web developers can all use logs
to make better data-driven decisions.
There are a lot of different log analysis tools out there, below are 5 of the most
popular ones. Each one of them has its own unique search features, live tail queries,
etc. Some of the tools listed are more log management services, but they also offer
more efficient ways to analyse your logs compared to looking at the raw data.
1. Loggly
Loggly is a cloud-based logging management developed in 2009. Some of its clients
are EA, Autodesk, SendGrid, Atlassian, Sony Pictures, and Citrix.
Loggly features: -
 has both free and paid plans starting at $49 per month.
 ability to see a bird’s eye view of your logs.
 remove the noise with a few simply.
2. Logentries

It is similar tool as loggly because it is also a cloud-based logging management developed in
2010. Some of their clients are the Discovery Channel, Macy’s, LogMeIn, Yummly, and
Parallels.
Logentries Features:-
 Has both free and paid plans starting at $39 per month.
 can filter logs in real-time, giving you much deeper insights into web
application and system events as the occur.
3. Splunk
Splunk is quite famous in the log and application management space. It is developed
in 2003 They offer great solutions for larger enterprise customers.
Splunk Features:-
 has both free and paid plans.
 offers a great way to collect, store, index, search, correlate, visualize, analyze
and report on any machine-generated data to identify and resolve operational
and security issues in a faster, repeatable, and more affordable way.
 a great way to spot and visualize possible trends.
4.logz.io
it provides real-time, actionable insights into your log analytics data with hosted ELK
as a service. Some of its clients are Netflix, Facebook, LinkedIn, Cisco, and
Microsoft. it is probably one of the biggest competitors to Splunk.
logz.io Features: -
 has both free and paid plans .
 can filter results by server, application, or any custom parameter until you
reach the source of the problem
 Alternative of splunk
5. GoAccess
GoAccess is a fast, terminal-based log analyser. It quickly analyses and view web
server statistics in real-time without needing to use your browser.
GoAccess Features:-
 it is open source,
 completely free to use.

 can easily generate reports with this tool
Design and implementation of log parsing
methods
I have created log analyses Application. It is window based app. It will able to track all
system logs.
Name of the method:-
“Window live logs identification”
Application Features:-
 One can easily Know what error occurred when,
 Which application of system throw the error.
 what time it occurred.
 One can filter log by event name
 Number of count of particular event occurred in system log
Technology Use:
I have used Dot net technology with c# language to get my results.
Research and development:-
I know about the window logs and one of the window technologies Asp.net.I just
researched about the classes that I have to fetch out the logs.i found log entity class
which can be able to fetch all logs generated in our window.
EventLog obj=new EventLog();
foreach (EventLogEntry entry in obj.Entries)
view.Rows.Add(entry.EventID,
entry.MachineName, entry.Message, entry.Source,
entry.EntryType);

Details: -
Form1.aspx.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Diagnostics;
using System.Data;
public partial class form1 : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
bindgrid();
}
public void bindgrid()
{
try
{
EventLog obj = new EventLog();
obj.Log = "System";
int num = obj.Entries.Count;
DataTable view = new DataTable();
view.Columns.AddRange(new DataColumn[5] {

new DataColumn("eventID", typeof(int)),
new DataColumn("mn", typeof(string)),
new DataColumn("msg",typeof(string)),
new DataColumn("src",typeof(string)),
new DataColumn("lvl",typeof(string))
});
if (txtsrc.Text == "")
{
foreach (EventLogEntry entry in obj.Entries)
{
view.Rows.Add(entry.EventID, entry.MachineName,
entry.Message, entry.Source, entry.EntryType);
}
}
else
{
foreach (EventLogEntry entry in obj.Entries)
{
if (
txtsrc.Text.Trim().ToLower() ==
entry.EventID.ToString().Trim().ToLower() ||
txtsrc.Text.Trim().ToLower() ==
entry.MachineName.Trim().ToLower() ||
txtsrc.Text.Trim().ToLower() ==
entry.EntryType.ToString().Trim().ToLower() ||
txtsrc.Text.Trim().ToLower() ==
entry.Source.Trim().ToLower() ||
entry.Message.ToLower().Contains(txtsrc.Text.ToLower())
)
{
//SEARCHED EVENT BIND

view.Rows.Add(entry.EventID, entry.MachineName,
entry.Message, entry.Source, entry.EntryType);
}
}
}
// When No record found
if (view.Rows.Count == 0)
{
view.Rows.Add(int.TryParse("-", out num), "-", "No Record
Found", "-", "-");
logGrid.DataSource = view;
}
// when record found on search or all records fetch
else
{
logGrid.DataSource = view;
}
// Bind grid
logGrid.DataBind();
}
catch (Exception)
{
txtsrc.Text = string.Empty;
bindgrid();
throw;
}
}
protected void btnSearch_Click(object sender, EventArgs e)

{
bindgrid();
}
}
Form1.aspx
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
<title></title>
</head>
<body>
<form id="log" runat="server">
<asp:Label ID="sample" runat="server" /><br />
<div class="filter">
<asp:Label Text="Filters" runat="server" Font-Bold="true" />
</div>
<div id="fltrdiv" style="margin-top: 10px; margin-bottom: 10px">
<div class="left">
Search:<asp:TextBox runat="server" ID="txtsrc"
placeholder="Search"/>
<asp:Button Text="Search" ID="btnSearch"
OnClick="btnSearch_Click" runat="server" />
</div>
</div>
<br />
<div id="grid">
<asp:GridView ID="logGrid" runat="server" PageSize="10"
AllowPaging="True" AllowSorting="True" CellPadding="4"
EnableSortingAndPagingCallbacks="True"
AutoGenerateColumns="False" BackColor="White" BorderColor="#3366CC"
BorderStyle="None" BorderWidth="1px">
<Columns>
<asp:BoundField DataField="eventID" HeaderText="Event
ID" />
<asp:BoundField DataField="mn" HeaderText="Machine
Name" />
<asp:BoundField DataField="msg" HeaderText="Message" />
<asp:BoundField DataField="src" HeaderText="Source" />
<asp:BoundField DataField="lvl" HeaderText="Log Type" />
</Columns>
<FooterStyle BackColor="#99CCCC" ForeColor="#003399" />
<HeaderStyle BackColor="#003399" Font-Bold="True"
ForeColor="#CCCCFF" />
<PagerStyle BackColor="#99CCCC" ForeColor="#003399"
HorizontalAlign="Left" />
<RowStyle BackColor="White" ForeColor="#003399" />
<SelectedRowStyle BackColor="#009999" Font-Bold="True"
ForeColor="#CCFF99" />
<SortedAscendingCellStyle BackColor="#EDF6F6" />

<SortedAscendingHeaderStyle BackColor="#0D4AC4" />
<SortedDescendingCellStyle BackColor="#D6DFDF" />
<SortedDescendingHeaderStyle BackColor="#002876" />
</asp:GridView>
</div>
</form>
</body>
</html>
Web.Debug
<?xml version="1.0" encoding="utf-8"?>
<!-- For more information on using web.config transformation visit
http://go.microsoft.com/fwlink/?LinkId=125889 -->
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-
Transform">
<!--
In the example below, the "SetAttributes" transform will change the value
of
"connectionString" to use "ReleaseSQLServer" only when the "Match" locator
finds an attribute "name" that has a value of "MyDB".
<connectionStrings>
<add name="MyDB"
connectionString="Data Source=ReleaseSQLServer;Initial
Catalog=MyReleaseDB;Integrated Security=True"
xdt:Transform="SetAttributes" xdt:Locator="Match(name)"/>
</connectionStrings>
-->
<system.web>
<compilation xdt:Transform="RemoveAttributes(debug)" />
<!--
In the example below, the "Replace" transform will replace the entire
<customErrors> section of your web.config file.
Note that because there is only one customErrors section under the
<system.web> node, there is no need to use the "xdt:Locator" attribute.
<customErrors defaultRedirect="GenericError.htm"
mode="RemoteOnly" xdt:Transform="Replace">
<error statusCode="500" redirect="InternalError.htm"/>
</customErrors>
-->
</system.web>
</configuration>.
Implementation: -
I started over coding and got the result exect what I want. All logs are
generated .now we are able to search all types of logs by a single pharase. I created
one method to filter all types of logs.anyone can search by single word or by single
pharase.

To implement this ,we will open the computer application iis that is internet
information service and on machine name ( my laptop DESKTOP-NAJL7EF)click the
“add new site” option via sys. After it,type test in file name and add port.then the
system will run on localhost website.
I have used this method on my own laptop and got following results:-
Site-localhost
Port-81
Log analysis method-form1.aspx
search:
Even
t ID
Machine
Name Message Source Log Type
43
DESKTO
P-
NAJL7EF
Installation Started: Windows has started
installing the following update: Update
for Windows 10 Version 1709 for x64-
based Systems (KB4041994)
Microsoft-
Windows-
WindowsUpdateCli
ent
Informati
on
19
DESKTO
P-
NAJL7EF
Installation Successful: Windows
successfully installed the following
update: Update for Windows 10 Version
1709 for x64-based Systems
(KB4041994)
Microsoft-
Windows-
WindowsUpdateCli
ent
Informati
on
105
DESKTO
P-
NAJL7EF
The description for Event ID '105' in
Source 'Microsoft-Windows-Kernel-
Power' cannot be found. The local
computer may not have the necessary
registry information or message DLL files
to display the message, or you may not
have permission to access them. The
following information is part of the
event:'false', '40478', '50206'
Microsoft-
Windows-Kernel-
Power
Informati
on
6013
DESKTO
P-
NAJL7EF
The system uptime is 955404 seconds. EventLog Informati
on
43
DESKTO
P-
NAJL7EF
Installation Started: Windows has started
installing the following update: Definition
Update for Windows Defender Antivirus -
KB2267602 (Definition 1.261.404.0)
Microsoft-
Windows-
WindowsUpdateCli
ent
Informati
on
19 DESKTO
P-
NAJL7EF
Installation Successful: Windows
successfully installed the following
update: Definition Update for Windows
Microsoft-
Windows-
WindowsUpdateCli
Informati
on
404 Search

Defender Antivirus - KB2267602
(Definition 1.261.404.0) ent
105
DESKTO
P-
NAJL7EF
The description for Event ID '105' in
Source 'Microsoft-Windows-Kernel-
Power' cannot be found. The local
computer may not have the necessary
registry information or message DLL files
to display the message, or you may not
have permission to access them. The
following information is part of the
event:'false', '36404', '49263'
Microsoft-
Windows-Kernel-
Power
Informati
on
15
DESKTO
P-
NAJL7EF
The description for Event ID '15' in
Source 'Microsoft-Windows-Kernel-
General' cannot be found. The local
computer may not have the necessary
registry information or message DLL files
to display the message, or you may not
have permission to access them. The
following information is part of the
event:'41', '\??\C:\Windows\System32\
config\components', '57737216',
'40407040'
Microsoft-
Windows-Kernel-
General
Informati
on
30
DESKTO
P-
NAJL7EF
The description for Event ID '30' in
Source 'Microsoft-Windows-Kernel-Boot'
cannot be found. The local computer may
not have the necessary registry
information or message DLL files to
display the message, or you may not have
permission to access them. The following
information is part of the
event:'6799404670157', '1161', '1312',
'2154', '2155'
Microsoft-
Windows-Kernel-
Boot
Informati
on
1
DESKTO
P-
NAJL7EF
Possible detection of CVE: 2018-04-
14T08:41:10.500000000Z Additional
Information: 2018-04-
13T15:42:58.494240400Z This Event is
generated when an attempt to exploit a
known vulnerability (2018-04-
14T08:41:10.500000000Z) is detected.
This Event is raised by a User mode
process.
Microsoft-
Windows-Kernel-
General
Informati
on
1 2

Final report
Design
I have made log analyses Application. It is window-based app. It is able to track all system
logs. The name of the application is Window live logs identification.
Characteristics: -
it has following features; -
 One can easily Know what error occurred when,
 Which application of system throw the error.
 what time it occurred.
 One can filter log by event name
 Number of count of particular event occurred in system log
Technology Use:
Dot net technology is used with c# language to get outcomes.
Understanding and knowledge/implementation:-
To run this project, one should have proper knowledge of window logs and one of
the window technologies Asp.net. I just researched about the classes that I have to
fetch out the logs. I found entity class which can be able to fetch all logs generated in
our window.
I started over coding and got the result exact what I want. All logs are generated.
now we are able to search all types of logs by a single phrase. I created one method
to filter all types of logs. Anyone can search by single word or by single phrase.
Before using this, one should install computer application iis that is internet
information service because it cannot work without it.in iis we will click on option of
machine name ( my laptop DESKTOP-NAJL7EF)click the “add new site” option via
sys. Afterwards we will write test in file name and add port. Then the system will run
on localhost website.
Through this method any error in system can be easily detected.
One can easily find out all log events and its source and timing.
For example: -
One can get to know about that which url is searched and when.

One can also come to know about the any error that occur in system.to site an
example, if I searched something (my friend photo on Facebook) but I did not get
and my computer show 404 error. Then by searching it via this method, I come to
know about the source and timing when this error occurred.
References: -
1. DeLaRosa, Alexander (February 8, 2018). "Log Monitoring: not the ugly
sister". Pandora FMS. Archived from the original (html) on February 14, 2018.
Retrieved February 14, 2018. A log file is a text file or XML file used to register the
automatically produced and time-stamped documentation of events, behaviors and
conditions relevant to a particular system.
2. Microsoft Log File Viewer
3.Jump up^ W3C, Extended Log File Format W3C Working Draft WD-logfile-960323
4.Jump up^ Microsoft, The Transaction Log (SQL Server)
5.Jump up^ qlshack.com, A beginner’s guide to SQL Server transaction logs,
February 11, 2014 by Ivan Stankovic
6.Jump up^ techrepublic.com, Understanding the importance of transaction logs in
SQL Server, SQL Server transaction log maintenance, By Crowe, Chizek, November
11, 2004
7.Jump up^ neurobs.com, Logfiles
8.Jump up^ Microsoft, LogFile Class
9.Jump up^ Jansen, B. J. 2006. Search log analysis: What is it; what's been done;
how to do it. Library and Information Science Research, 28(3), 407-432