Operating System Security

This report discusses the use of IPtables and firewall rules to secure Kali Linux systems. It provides a methodology for securing the operating system and demonstrates the use of IPtables to block, filter, and drop incoming data packets. The report covers XMAS tree attack, SYN-FLOOD attack, null packet attacks, and force fragment attacks. It also includes a conclusion and references.

Running head: OPERATING SYSTEM SECURITY
Operating system security
Name of the Student
Name of the University
Authors note

1OPERATING SYSTEM SECURITY
Executive Summary
With the increasing number of data as well as security breaches it becomes important to
secure the operating systems. For this one of the most common tools are used IPtables. The
following table contributes to the discussion on the Use of IP tables to secure the Kali Linux
systems.

2OPERATING SYSTEM SECURITY
Table of Contents
Introduction................................................................................................................................3
Part 1: Methodology for Securing the Kali Linux.....................................................................3
Methodology for Securing the Kali Linux OS...........................................................3
Part 2: Demonstration of the Methodology................................................................................4
IPtable........................................................................................................................4
Use of IPtables to block filter and drop the incoming data packets...........................................6
Conclusion................................................................................................................................15
References................................................................................................................................16

3OPERATING SYSTEM SECURITY
Introduction
In today’s vulnerable computing environment data packets that are flowing inside
and outside a specific network can be are intercepted using different tools and techniques,
analysed and manipulated to bring down that network. For this specific paper, we are
analysing the use of adding entry to the IPtables and firewall rules to restrict the unwanted
data traffic coming or going outside the system (Jung and Sung 2015). We are using Kali
Linux OS in order to implement this security mechanism.
Following report contributes to the discussion of the utilization of IPtables and
firewall rules to secure an operating system. In addition to that, firewall rules that are
implemented and effectiveness are also provided in different sections of this report.
Part 1: Methodology for Securing the Kali Linux
Methodology for Securing the Kali Linux OS
For Linux operating systems the IPtables is one of the best command-line tools that
can be used firewall utility for this operating system. The IPtables uses policy chains in order
to block or allow traffic to the system (Tekeoglu and Tosun 2016). Every time a network
based connection attempts to set up a connection to the system, IPtables checks for a rule in
its created list to match and try to follow the policies.
For this project we created the following methodology to secure the Linux based
systems so that, the servers using the operating system can check analyse and take action
accordingly for any kind of network based attack to the system. Following are the stages in
this methodology,
 Determining the type of the attack

4OPERATING SYSTEM SECURITY
 Setting up the rules for data packets
 Checking the data packets and its content to check validity.
 Defining the action policy if any rule matches for the incoming request data
packets.
 Updating IPtables with rules and policies for different types of attacks and its
nature of the attack.
In the following section we have created the rules and policies to prevent the attacks like
XMAS tree attack, SYN-FLOOD attack, null packet attacks and at last force fragment
attacks.
Part 2: Demonstration of the Methodology
IPtable
In case of Linux operating system the server administrator can use IPtable to
populate with chains of security rules. This rules are helpful in the manipulation of the data
packets at several stages of data packet processing lifecycle by the kernel of the system
(Tekeoglu and Tosun 2016). For an ideal scenario to secure a server, it is suggested to
populate rules that will DENY all the incoming data traffic inside a network. After doing this,
explicit rules that acts as exceptions for the IPtables such as for port 80 requests data traffic
can be added to the table (Jung and Sung 2015).
For the IPtable following are the available options to secure the system,
Screenshot 1: man IPtable result.

5OPERATING SYSTEM SECURITY
For the IPtables [table] {-A|-C|-D}, -A chain is used to append rule to the chain, -C is
used for checking the chain , Check for the existence of a rule. –D is used for deletion of a
matching firewall rule from chain.
For the remaining options in the above diagram, following are the explanations for
them,
-I chain [rulenum]: in order to insert in chain as rulenum.
-R chain rulenum: Replace rule in chain
-L [chain [rulenum]]: In order to list the existing rules in a chain
-S [chain [rulenum]]

6OPERATING SYSTEM SECURITY
-L: In order to List all rules in the selected chain
-F: In order to flush the rules in selected chain
-Z: This option is used to Zero the packet and byte counters in all chains.
-N: Useful for creating a new user-defined chain.
-X: useful for deleting an optional user-defined chain.
-P: mainly used for setting the policy for the chain for a given target.
-E, In order to rename an old-chain to a new-chain.
For this specific project, we are using the IPtables to block the websites Facebook and
Twitter. In case real world scenario, this technique can be used for any suspicious website or
IP which may try to get access to the critical data stored on the system.
Use of IPtables to block filter and drop the incoming data packets
Following are the steps that are used for securing the system,
Step 1: If the IPtable tool is not available for Linux system then at first, it can be
retrieved using the following command “sudo apt-get install iptables”

7OPERATING SYSTEM SECURITY
Screenshot 2
As shown in the above diagram, we can see that, Kali Linux alreadyy
For
Step 2: In this stage, the to modify the existing rules and start with a new IPtable we
will use the following command;
iptables-F
As there are no existing tables in the Kali OS, thus it shows nothing as output.
Screenshot 3:

8OPERATING SYSTEM SECURITY
Step 3: In this stage, we checked the default IPtables are configured in the system,
Screenshot 4:

9OPERATING SYSTEM SECURITY
In the next stage we determined to, protect the system from the XMAS packets.
XMAS packets are mainly used in order accomplish "Christmas tree attack." In this type of
attacks large number data-heavy XMAS are sent by the hackers or attackers in order to slow
down the performance of a network or overload a network with request data packets. These
packets are also used in some kind of network inspection techniques by the hackers. In this
technique the attackers send these heavy data packets to collect information and idea about
the the target machine or network they may infiltrate.
Following is the screen shot for determining the firewall policy using IPtables.
Screenshot 5:
root@kali:~# iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
root@kali:~# iptables -L

10OPERATING SYSTEM SECURITY
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP tcp -- anywhere anywhere tcp
flags:FIN,SYN,RST,PSH,ACK,URG/NONE
DROP tcp -- anywhere anywhere tcp
flags:FIN,SYN,RST,PSH,ACK,URG/NONE
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Step 4: In this stage, the code is executed to protect the server or the system from
the SYN-FLOOD attack. After adding this policy following screen is presented using the
IPtables –L. The policy checks the SYN packets.
Following code is used for the adding to check the data packets,
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
following is the newly added policy in the IPtable
root@kali:~# iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
root@kali:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP tcp -- anywhere anywhere tcp
flags:FIN,SYN,RST,PSH,ACK,URG/NONE
DROP tcp -- anywhere anywhere tcp
flags:FIN,SYN,RST,PSH,ACK,URG/NONE
DROP tcp -- anywhere anywhere tcp flags:!FIN,SYN,RST,ACK/SYN state
NEW
Chain FORWARD (policy ACCEPT)
target prot opt source destination

11OPERATING SYSTEM SECURITY
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
root@kali:~#
Screenshot 6:
IPtable with new added policy
Step 4: In this stage, we implemented as well as added a new security policy in the
IPtable, which will check the null data packets or malformed packets.
We used the following code to implement this policy,
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
and the resultant table becomes,

12OPERATING SYSTEM SECURITY
root@kali:~# iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
root@kali:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP tcp -- anywhere anywhere tcp
flags:FIN,SYN,RST,PSH,ACK,URG/NONE
DROP tcp -- anywhere anywhere tcp
flags:FIN,SYN,RST,PSH,ACK,URG/NONE
DROP tcp -- anywhere anywhere tcp flags:!FIN,SYN,RST,ACK/SYN state
NEW
DROP tcp -- anywhere anywhere tcp
flags:FIN,SYN,RST,PSH,ACK,URG/NONE
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
following is the screenshot of the implemented policy,
Screenshot 8:

13OPERATING SYSTEM SECURITY
Step 5: Adding policy for Force Fragments packets check to protect the fragments if
they intended to forge an attack on the system or the server using kali Linux server.
This kind of attacks is one of the form of DoS (denial of service) attack. In this
technique the attacker overloads the targeted network by exploiting datagram fragmentation
technique which is used for reassembling the fragments of the data packets.
IP fragmentation is a technical procedure at the receivers end in which datagrams are
divided into numerous small packets or fragments. After this, the fragments are transmitted
through the network and reassembled into the original datagram after al of them reach at their
destination.
Fragmentation process is very necessary for transmission of data through the
networks, each network has a limit for the size of datagrams that can be processed by it at a
given time. This maximum limit is defined as the maximum transmission unit (MTU). In this
kind of attack mainly transmission of fraudulent ICMP and UDP data packets are used that

14OPERATING SYSTEM SECURITY
obliviously exceeds the MTU (1500bytes approx.). As most of the packets are fake and thus
could not be reassembled. In the meantime, too many fake packets consume the system
resources quickly. Consequently, the server fails and in few moments gets down as well as
become unavailable.
Following command is used to add this rule to the IPtables.
iptables -A INPUT -f -j DROP
Screenshot 10:
Following is the table that includes all the rules in IPtable that filters the data packets
to prevent the OS from attacks.
root@kali:~# iptables -A INPUT -f -j DROP
root@kali:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP tcp -- anywhere anywhere tcp
flags:FIN,SYN,RST,PSH,ACK,URG/NONE

15OPERATING SYSTEM SECURITY
DROP tcp -- anywhere anywhere tcp
flags:FIN,SYN,RST,PSH,ACK,URG/NONE
DROP tcp -- anywhere anywhere tcp flags:!FIN,SYN,RST,ACK/SYN state
NEW
DROP tcp -- anywhere anywhere tcp
flags:FIN,SYN,RST,PSH,ACK,URG/NONE
DROP all -f anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
root@kali:~#
Conclusion
Even though, there are numerous techniques that can be used to secure the system
from being intercepted. This technique includes Limiting the SSH access to some specific
users inside the network, changing file read write permissions, encrypting the folders as well
files accessible in the network Iptables is a bundle of powerful security directives that resides
inside the any of the Linux kernels. These directives are helpful in checking the data at
various stages of the packet processing lifecycle. Iptables tool is helpful in creating and
managing the security rules that handles the functionalities such as packet manipulation,
connection tracking which in turn helps in hardening the Operating System.

16OPERATING SYSTEM SECURITY
References
Diekmann, C., Hupel, L., Michaelis, J., Haslbeck, M. and Carle, G., 2018. Verified iptables
Firewall Analysis and Verification. Journal of Automated Reasoning, pp.1-52.
Jung, S.J. and Sung, K., 2015. A Study on the Iptables Ruleset Against DoS Attacks. The
Journal of Advanced Navigation Technology, 19(3), pp.257-263.
Muniz, J., 2013. Web Penetration Testing with Kali Linux. Packt Publishing Ltd.
Oesterling, C., 2016. DDoS Protection With IPtables.
Petrucci, L., Bonola, M., Pontarelli, S., Bianchi, G. and Bifulco, R., 2017, April.
Implementing iptables using a programmable stateful data plane abstraction. In Proceedings
of the Symposium on SDN Research (pp. 193-194). ACM.
Tekeoglu, A. and Tosun, A.Ş., 2016, October. A Testbed for Security and Privacy Analysis
of IoT Devices. In Mobile Ad Hoc and Sensor Systems (MASS), 2016 IEEE 13th
International Conference on (pp. 343-348). IEEE.