An Empirical Study into Security Exposure to Hosts of Virtualization

This report presents an empirical study on the security exposure to hosts of hostile virtualized environments, focusing on popular virtual machine implementations for x86 systems. It employs source code auditing and black-box random testing to assess the security risks. The study investigates the assumptions of isolation and containment, testing virtual machine implementations and classifying threats as total compromise, partial compromise, or abnormal termination. Testing procedures involve source code audits, stress testing with crashme, and I/O fuzzing with iofuzz. The results detail failures in various virtual machine emulators, including QEMU, VMware, and Xen, highlighting specific vulnerabilities like heap overflows and integer signedness errors. The report concludes with an analysis of the findings and a discussion of related work, offering insights into the security challenges of virtualized environments. Desklib provides access to this and other solved assignments.

An Empirical Study into the Security Exposure to
Hosts of Hostile Virtualized Environments

Abstract
As virtual machines become increasingly commonplace as a method of separating hostile
or hazardous code from commodity systems, the potential security exposure from
implementation flaws has increased dramatically. This report investigates the state of popular
virtual machine implementations for x86 systems, employing a combination of source code
auditing and black box random testing to assess the security exposure to the hosts of hostile
virtualized environments.

I. INTRODUCTION

Secure isolation is one of the fundamental concepts of virtualization[15], confining a program to a virtualized
environment should guarantee that any action performed inside the virtual machine cannot interfere with the system
that hosts it1. Consequently, virtual machines have seen rapid adoption in situations where separation from a hostile
or hazardous program is critical.
We can consider a virtualized environment hostile when untrusted code is being executed or when untrusted data is
being processed by services inside the virtual machine2. The use of virtual machines in malware analysis is well
documented[22][11], as are numerous proposals for hosting hazardous applications such as honeypots or intrusion
detection systems[10][23].
This report seeks to test how safe the assumptions of isolation and containment are in practice through the analysis
of several popular virtual machine implementations for the x86 architecture in use today. While this is not a new
concept, Ferrie describes how DOS-based, reduced privilege virtual machines could be trivially escaped from[22],
and Tim Shelton describes a flaw in the VMware Workstation NAT service that could result in compromising the
host machine[24], little other research has been published in this area.
A. Virtual Machine Threats
Root Secure Monitors
Commodity operating systems can often be compromised and privileges escalated[18], it is essential that a virtual
machine monitor is ‘root secure’[25], meaning that no level of privilege within the virtualized guest environment
permits interference with the host system.
Detection
Detecting virtualized environments has been explored in detail by other researchers[22]. Virtual machines employed
in malware analysis or honeypots should endeavor to be undetectable or risk changing the behavior of the emulated
code. An emulated malware sample could choose to act benignly, for example, but when executed on a physical
machine immediately performs some malicious action.
Detection will not be considered any further in this report, although being able to detect a virtual machine, or more
specifically, virtual machine implementation, may be a prerequisite for the more serious attacks investigated in this
report. Without being able to identify which virtual machine implementation is present, an attacker would have
difficulty determining how to escape the environment.
Security Threats
The level of threat posed by a hostile virtualized environment that can subvert the normal operation of the virtual
machine will be classified as follows
• Total compromise. The virtual machine monitor is subverted to execute arbitrary code on the host with the
privileges of the vmm process.
• Partial compromise. The vmm leaks sensitive information about the host, or a hostile process interferes with
the vmm (e.g., allocating more resources than the administrator intended) or contaminating state checkpoints.
• Abnormal termination. The virtual machine monitor exits unexpectedly or triggers an infinite loop that
prevents a host administrator from interacting with the virtual machine (for example, suspending, rolling back, etc.),
particularly if accessible as an unprivileged user within the guest.
Overview
The next sections describes the testing procedure and tools used during this investigation. Section IV presents the
results of testing, and in Section X those results are analyzed. Related work in this field known to the author is
described in Section VI., and my conclusions are presented in Section V.

1. Excepting the allocated use of the host’s resources.
2. And thus a compromise of the service by an attacker could result in untrusted code being executed.
II. TESTING

Three of the virtual machine emulators selected for this investigation are open source software, which permitted
manual analysis of the code by inspecting relevant sections for programming errors. The remaining emulators are
proprietary software products distributed without source code available, thus an automated blackbox testing
procedure based on the fuzz technique pioneered by Miller, Fredriksen and so[1] was designed.
Two subsystems were identified through comparison of metrics as the most complex components of the virtual
machine emulators . It is reasoned that the most complex code is most likely to harbour the subtle bugs that can be
exposed through the random blackbox testing methods employed here. The subsystems that have been identified are:
• Instructions; correctly parsing and trapping privileged instructions, and handling illegal, unrecognized and faulting
opcodes correctly.
• Emulated I/O Devices; handling invalid, illegal, or nonsensical I/O activity.
Multiple tools were selected or written to exercise this code in the same manner as [1]. Two of the main tools
employed during this investigation were crashme[26], and iofuzz.
A. CRASHME
Testing hardware and operating system robustness to malformed, illegal or nonsensical instructions has been well
documented[27], and tools are already part of common operating system test suites, such as the Linux Test Project 2.
Wood and Katz[16] proposed validating cache controllers with random data as early as 1990. For the purposes of
this investigation, the simple usermode tool crashme by George Carrette was selected. crashme subjects the
emulated environment to a stress test of fault handling by continually attempting to execute random byte sequences
until failure is encountered.
B. IOFUZZ
Implementing an accurate emulation of a hardware device in software is undoubtedly a challenging task, the
correctness of the implementation is not considered here, only the handling of invalid operations. A simple tool was
developed modeled on fuzz to generate random I/O port activity inside the virtual machines, and any errors
encountered were catalogued and analysed.
To the authors knowledge no similar testing has been performed before.
Table 1: Virtual Machine Emulators Tested
Name Type
Bochs Open source pure software emulator.
Virtual
Machine Xa
Proprietary virtual machine popular on the Macintosh
platform.
QEMU Open source pure software virtual machine emulator.
Virtual
Machine Yb
Proprietary virtual machine popular on the Microsoft
Windows platform.
VMware Proprietary virtual machine.
Xen Open Source paravirtualisation based emulator.
a. The product name has been obscured as the vendor failedto respond in time for publication.
b. The vendor was unable to investigate the issues presentedin time for publication.

III. PROCEDURE

A. Test platform
All tests were performed on a typical commodity system, a Pentium IV 3.2GHz running a Linux 2.6 based
operating system, with the exception of Virtual Machine Y which only supports Microsoft Windows hosts. All
virtual machines tested were the latest versions available at the time of writing and used in their default
configuration state where possible. When some minimal configuration was required, the options selected have
been described in the relevant sections.
All flaws identified as a result of this investigation were reported to the respective developers. In the case of
open source implementations, patches were provided.
Two vendors were unable to respond to the reports presented in time for publication, so the names of their
products have been obscured in the remainder of this report.
B. Failure Criteria
An implementation is considered to have failed the automated tests performed if emulated code in the virtualized
environment can crash or cause the emulator to abnormally exit. Each crash is reproduced and investigated and
its impact determined, which is classified as full, partial, or minor compromise as defined above.
Causing the emulator to crash is not necessarily a security flaw, as a privileged user in the virtualized
environment may legitimately request the emulated machine is halted. However, there are implications for
malware analysis, where a hostile sample can perform some harmless operation that has no effect on a physical
machine but immediately halts any emulator being used to analyse it. Additionally, this can be considered a
simple litmus test as to the quality of the code and the amount of testing performed. An implementation that
continually crashes or aborts is unlikely to have seen significant testing and is a good candidate for further
research. By carefully analysing any crashes encountered it may be determined where the weakest code exists
and where future research may prove to be most fruitful.
C. Testing Procedure
Where source code is available, a source code audit is performed to identify as many flaws as possible. The audit
involves a human researcher familiar with programming errors manually reading the source code and identifying
possible codepaths that could result in security problems. Once the audit has been completed, any flaws
identified are reproduced in a live environment and documented.
Secondly, the virtual machine is subjected to stress testing using the crashme utility described in Section I. If a
fatal error is encountered, the error is reproduced, analysed then corrected if feasible, and the test continues until
run without error for a period of at least 24 hours.
Finally, the machine is subjected to 24 hours of iofuzz, any failures are reduced to the minimum series of peeks
and pokes, and the results are collected and tabulated.

IV. RESULTS

Table 2: Virtual Machine Failure Impact
Name Full Partial Minor
Bochs • •
Virtual Machine Xa • • •
QEMU•b verified, as the fix would be the same. • •
Virtual Machine Y • •
VMware • • •
Xen •c
a. See “Test platform”.
b. these flaws may impact other implementations derived rom the same code.
c. due to lack of appropriate hardware, these findings were not tested.
A. QEMU
QEMU is an open source pure software emulator by Fabrice Bellard that boasts many advanced features and a
dynamic translation system that offers better performance than typically seen in similar emulators.
As QEMU is distributed with full source code under the GPL license, a detailed source code audit was possible
which revealed the presence of multiple exploitable implementation flaw.
QEMU 0.8.2 was the latest version available as of this writing, which was used in its default configuration. A
bootable ISO image was prepared with the software required for testing. According to [28] the proprietary virtual
machine emulator win4lin is based on source code licensed from the QEMU project, a copy of this software
could not be obtained for testing, but could also be affected by the same issues described here. KVM and
VirtualBox also appear to include or derive code from QEMU.
QEMU Cirrus CLGD-54XX “bitblt ” heap overflow
The cirrus_invalidate_region() routine used during video-to-video copy operations in the Cirrus VGA extension
code omits bounds checking in multiple locations, see Table 4, allowing you to overwrite adjacent buffers by
attempting to mark non-existent regions as dirty.
While the overflowing data is not controllable, the overflow allows the LSB of nearby function pointers to be
moved to an attacker controlled location.
QEMU NE2000 “mtu ” heap overflow
Ethernet frames written directly into the NE2000 device registers do not have their size checked against the mtu
before being transferred, see Table 5, resulting in large values in the EN0_TCNT register overwriting a heap
buffer in the slirp library with arbitrary attacker controller data from the devices memory banks.
This flaw is most likely also reachable via the alternative NIC emulated by QEMU, a PCNet32, however this has
not been
QEMU NE2000 “receive” integer signedness error
Nonsensical values in specific device registers can result in sanity checks being bypassed, an integer overflowing
and attacker controlled data overflowing a heap buffer, see Table 6.
QEMU “net socket” heap overflow.
QEMU does not perform adequate sanity checking on data received via the “net socket listen” option, resulting
in an exploitable heap overflow. Other guests, or local attackers, who can send traffic to this port could
potentially compromise the QEMU process.

QEMU Miscellaneous
• An infinite loop was discovered in the emulated SB16 device.
• The DMA code will dereference an uninitialized function pointer if commands are issued to an unregistered
dma channel.
• The unprivileged ‘aam’ instruction does not correctly handle the undocumented divisor operand, resulting
in the qemu process performing an integer divide-by-zero. See Table 7.
• The unprivileged ‘icebp’1 instruction will halt the virtual machine, this feature cannot be disabled. See Table 8.
• VGA BIOS panics can safely be ignored according to the documentation accompanying Bochs, but cause
QEMU to exit immediately. Bochs itself actually prompts the user whether to continue or not on encountering a
panic.
• BIOS panics can also safely be ignored, but cause QEMU to exit immediately.
• Multiple integer signedness errors exist in the emulated IDE controller, allowing very large counts to bypass
sanity checks.
Summary
An attacker with access to a QEMU virtualized environment could potentially compromise the virtual machine
process and execute arbitrary code with the privileges of the emulator. Malware being studied inside QEMU, even in
an unprivileged state, can terminate the virtual machine safely and reliably. Code examples of these flaws are
provided in Section X.
B. VMware Workstation and Server
VMware is a family of popular proprietary virtual machine emulators for the x86 platform. The two major products
bearing the VMware name are VMware Workstation and VMware Server. Both VMware Workstation and Server
were studied during this investigation. As source code is not freely available for these programs, the focus was on
random testing, followed by the analysis of any anomalies encountered.
The latest version of VMware Workstation available at time of writing was 5.5.3.34685, and VMware Server was
1.0.1.29996. These were used in as close to the default configuration as possible.
VMware Security
Vulnerabilities affecting the security of the host of a VMware virtual machine have been published before. In 2005
Tim Shelton reported a heap overflow in the vmnatd service[24], where a specially crafted EPRT or PORT FTP
command would result in an exploitable heap overflow, allowing hostile virtualized code to compromise and thus
execute code on the host machine.
A communication channel exists between the guest and host, and while officially undocumented, has been
successfully reverse engineered and documented by several researchers and is known as the “VMware
Backdoor”[29].
Theoretically this channel could allow hostile guests to steal clipboard data, leak sensitive information about the
host, and other potentially dangerous operations. Care should be taken to avoid it when hosting potentially hostile
machines.
Audit Procedure

VMware was setup with the default options selected for a Linux guest and was subjected to a long period of stress
testing using the crashme and iofuzz tools. Setting up an automated testing procedure proved to be a significant
challenge, as testing revealed a large number of reachable assertions within the VMware emulated hardware could
be triggered by poking I/O ports in an unexpected fashion.
Assertions
The failed assertions encountered during testing were logged and catalogued. Table 9 demonstrates where the most
assertions were found relative to the various I/O regions. This gives a good idea of where the least tested code exists
and perhaps, which demands the most careful inspection in future. Dozens of unique assertions were discovered. The
biggest source appears to be the PIIX4 emulation code. See Table 9, “Map of VMware Assertions relative to I/O
Region.,”
Vulnerabilities
A serious flaw was discovered in the PIIX4 power management code. A specially crafted poke to I/O port 0x1004,
see Table 10, results in an out-of-bounds write to an attacker controlled location. By interacting with the power
management subsystem in specific ways, a write to an arbitrary location can be performed upon restarting a
suspended virtual machine.
VMware Miscellaneous
• A NULL dereference was discovered in the emulated floppy disk controller, which is triggered when reading a
word
from I/O port 0x3f5. The impact of a NULL dereference is generally low, although work has been done on
potentially
exploiting these errors[30].
Summary
An attacker with the necessary privileges to access the guest I/O ports could potentially compromise the virtual
machine monitor, executing arbitrary code with the privileges of the process.
The assertions and segmentation faults, while perhaps of interest to malware researchers, have no direct security
implications, but do suggest subsystems that have received little testing and give hints where future researchers
should concentrate their efforts.
C. Bochs
Bochs2 is a pure software virtual machine implementation written in C++. Bochs is remarkably robust in comparison
to the other implementations tested during this investigation, and with highly configurable error handling and
multiple disparate frontends, the random testing proved to be simple to set up.
Bochs 2.3 was the latest release available at the time of writing. A default bochs installation is very minimal, so a
sample of compile-time options from several popular distributions were taken, and the intersection of those options
was used, see Table 3, “Bochs compile-time configuration,”.