SDN and NFV integrated OpenStack Cloud - Birds eye view on Security
Whitepaper, lynx secure rootkit detection & protection by means of secure virtualization
1. LynxSecure
Low-level & boot-level rootkits revisited:
Real-time inline detection and protection by means of
secure virtualization
-- White Paper --
Phil Yankovsky, Craig Howard, Ed Mooring, Arun Subbarao & Avishai Ziv
LynuxWorks, Inc. San Jose, CA
-- LynuxWorks Proprietary & Confidential --
2. 2
Summary
Low-level and boot-level rootkits (nowadays commonly associated with APT) are the stealthiest and
most potent type of malware. They are stealthy to the extent that they are also capable to escape
common security research and discussions, and for good reason: They are very hard to detect, and when
detected, remediation of the infected targets is harder still.
There’s an ongoing controversy as to the share of low-level rootkits (also known as bootkits) in the
entire malwaredom. Some (Microsoft, Symantec and others) claim rootkits to be less than 5% of all
malware. Others, on the other hand (Kindsight and others) claim low-level rootkits amount to more than
50% of all malware. The stealthy-by-design nature of rootkits makes it hard to even create a commonly
agreed-upon view of the level and dynamics of this cyber-threat.
The commercial availability of rootkits (as software developer kits) and the professional discipline with
which they are developed (even to the extent of version control and customer support!) lead to a
worrisome and growing trend where “benign malware authors” are now adding rootkits to their lot.
Of one thing there’s no doubt: Common endpoint security means are not up to the task of protecting
against low-level rootkits. As a matter of fact, rootkits are specifically designed to evade and disable
them.
Introducing LynxSecure
In this whitepaper we introduce a novel and unique approach to detect rootkits, and protect from them,
all in real time, by means of secure virtualization.
Utilizing LynxSecure – LynuxWorks’ award winning secure hypervisor -- as a real-time inline rootkit
detector, is a completely new approach and methodology to counter the growing threat of cyber-attacks
based on rootkit infection.
We’ll highlight this approach by analyzing a TDL-4 rootkit infection. TDL-4 is the most common rootkit
and one that has been described as “indestructible” by Kaspersky Labs. We’ll provide a step-by-step
description of the detection, interception and remediation of TDL-4 using LynxSecure.
We’ll also claim that since low-level rootkits achieve their goals by assuming equal, or higher, security
posture than the operating system itself, the only viable approach to counter them would be to assume
a higher security posture than the rootkits, and do it in a secure, self-protecting, non-bypassable and
tamper-proof manner. This solution must execute with a higher privilege than the attacked OS; provide
complete control of the platform hardware; and monitor all activities of the OS and its applications.
Namely – use virtualization as a vessel to provide security.
* For more details about rootkits & bootkits see last chapter of this document.
-- LynuxWorks Proprietary & Confidential --
3. 3
LynxSecure: Secure Separation Kernel and Hypervisor
LynxSecure “Type-0” Hypervisor Technology
“Type-0” is a new bare-metal architecture, designed by LynuxWorks, that differentiates from type-1
hypervisors by removing the all un-needed functionality from the “security sensitive” hypervisor mode,
yet virtualizes guest operating systems in a tiny stand-alone package. By shedding the need of support
by a full operating system, the type-0 hypervisor drastically reduces the size and computational
overhead imposed on target systems. Thus, LynxSecure is effectively a virtual mother-board running at
ring -1 (vs. type 1 hypervisors, which are OS-like or full-blown OS).
Combining the best-of-breed capabilities of the separation kernel technology and virtualization,
LynxSecure provides unmatched capabilities to run one or more guest OSes using common PC platforms.
LynxSecure further differs from other hypervisors by offering the underlying security of a separation
kernel to isolate each virtual instance and provide protection to every guest with its own virtual
addressing space. In addition, it guarantees resource availability, such as memory and processor
execution resources, to each guest, so that no software can consume the allocated memory or
scheduled time resources of other guests.
LynxSecure supports the Multiple Independent Levels of Security (“MILS”) architectural approach, with
strict enforcement of data isolation, damage limitation and information flow control policies. Unlike a
traditional security kernel that performs all trusted functions for a secure operating system, the
Separation Kernel’s primary security function is to partition the resources of a system and to control
information flow among those resources.
-- LynuxWorks Proprietary & Confidential --
4. 4
LynxSecure Architecture
LynxSecure Rootkit Detection and Protection Capabilities
Much has been debated in the past about usage of virtualization as a means to counter low-level
rootkits. However, this remained theoretical due to the design and architectural deficiencies of type-1
hypervisors: They were not designed as secure environments, and their sheer size (they are in effect an
operating system with an exceptionally large attack surface) and monolithic architecture prevent them
from addressing these threats.
Overview:
LynxSecure is the first and only technology capable of real-time detection, alert and protection against
zero-day rootkits and bootkits. It is also capable of complete remediation of the compromised/attacked
OS, done in real-time & inline, yet outside of the compromised/attacked OS. Furthermore -- this
remediation can be done remotely by IT staff.
Rootkit detection:
Being the most privileged monitor in the platform, LynxSecure constantly monitors and introspects
malicious and irregular activity in HW areas. The closest entity to the platform’s hardware, LynxSecure’s
fine-tuned introspection can detect the rootkit’s activity even before it installs itself – it’s detected from
the first instance it begins to write to the MBR or other HW areas. LynxSecure’s unique architecture
(effectively – a virtual motherboard running at ring -1) makes it non-bypassable & tamper-proof. It’s also
OS agnostic, as it’s situated below any of the guest OSes. Simply put -- LynxSecure provides hardware
level protection by means of software. LynxSecure monitors:
Key disk areas (MBR, key blocks & sectors etc.)
Key physical memory areas
Key CPU instructions & data structures
-- LynuxWorks Proprietary & Confidential --
5. 5
Alert of rootkit infection-in-progress:
Upon detection, LynxSecure immediately alerts by sending detailed message to its management
system’s dashboard. The alert can then automatically trigger action that is sent back to LynxSecure, all in
real-time.
Protection against rootkit infection:
The protective action can be either block the rootkit from even further writing to the MBR/disk, or block
its install into the MBR/disk. For malware research purposes, the option to let the rootkit complete its
installation also exists. It then allows the researchers to closely monitor the rootkit’s activity.
Remediation of infected targets:
The remediation action can restore the MBR (and other HW parts such as slack disk sectors or last disk
sector/block – the favorite location for rootkits to place their loader and entire file system) to its
pristine/clean state, before it was infected and altered by the rootkit, thus effectively disabling the
rootkit. The remediation takes place inline and in real-time, and does not require the lengthy offline
process currently done by the rootkit-removers.
Low level information/data LynxSecure captures and record:
LynxSecure records and logs detailed low-level data such as: Specific guest OS which was infected; HW
areas affected by the rootkit; specific nature of change the rootkit tries to make; detailed before & after
states of the affected areas, including precise & reliable time-stamps etc..
Schematic view of LynxSecure rootkit detection & protection
-- LynuxWorks Proprietary & Confidential --
6. 6
Features and methodology:
LynxSecure has 3 core modules carrying the various tasks of rootkit detection and protection:
1. The hypervisor (hardened HW-tampering detector)
2. The secure virtualization support layer (FVS)
3. The auxiliary secure virtual machine (Virtual Device Server -- VDS).
Based on configuration, LynxSecure can serve as a self-contained node or a networked node, where
LynxSecure is controlled by remote management system.
Superior vantage point:
LynxSecure has a superior vantage point: It’s the lowest entity on the machine, and resides directly on
the HW. Rootkits can't attack LynxSecure, while LynxSecure uses multiple mechanisms to detect rootkits
and stop them. Furthermore, API’s can be provided for 3rd parties to take advantage of this superior
vantage point.
This vantage point allows LynxSecure unparalleled detection capabilities and the score of its protection
capabilities.
Being an inline entity, rather than static or offline as the other rootkit detection/protection technologies
are, not only is LynxSecure able to detect minute changes to the HW others cannot, but it can do it at
any given time, continuously: Starting BEFORE the OS boots, during boot process, during runtime, and in
shutdown phase (the phase where malware typically tries to hide itself in advance from static offline
anti-rootkit tools).
LynxSecure detection mechanism:
Based on pre-configured policy, LynxSecure hardened HW-tampering detector scans the HW (i.e., boot
sectors, slack disk sectors, hidden partitions, bad disk sectors, memory, CPU etc.) for changes and
irregularities. The scanning is accompanied with micro-snapshotting (see below) of the scanned HW
parts.
The initial state of each scanned part is securely stored as “Golden Image”.
The policy-based scanning scans for “absence of good” and “presence of evil”:
“Absence of good” (aka “zero-day”): When system part has been altered from a known-good state
(such as an MBR which no longer matches a golden MBR), or when it has been accessed or modified
in a manner that is not a known-good manner (i.e., nonstandard API call, call stack chain, I/O port
access pattern, etc., that is used to tamper with the MBR, partition slack space, etc.), or both.
“Presence of evil”: When system part (which can include access patterns) in the system matches a
known-bad state or pattern.
The detector’s dynamic runtime response & introspection capabilities include:
Run in active or passive mode, or both
Detect tampering actions to various disk sectors
Detect boot and reboot attempts by OS (Windows and other OSes)
Run anytime it is specifically instructed to do so by the hypervisor or by external API. In which case,
Windows is completely suspended and its resources (drives, boot devices, file system, RAM, etc.) are
available for capture and export to analysis by external engines/tools.
The detector is capable of scanning and snapshotting other blocks, such as the last block on the drive
(rootkits are known to populate the last block), or any other block on the drive.
-- LynuxWorks Proprietary & Confidential --
7. 7
The detector applies the same to any boot media: Directly assigned HDDs; virtual HDDs; other media,
such as USB boot etc. These capabilities of the detector are extendible to VBRs (Volume Boot Records).
Monitoring storage & boot device heuristics:
Where malware attempts to access devices, it can leave traces Windows and AV clients installed in
Windows typically cannot detect, and definitely not in real-time. For example, where a rootkit hooks
or subverts Windows block I/O layer, it can falsely report disk activity to the kernel, so the kernel
does not see where extra disk activity may have occurred. This can include the use of hidden
sectors on a drive. With LynxSecure, the disk activity and its related events, such as interrupts, are
visible, and can be exported via API to external agents for both analysis and/or immediate
remediation action.
The monitoring mechanisms include monitoring of drive/controller properties, drive/partition
contents and more.
Micro-snapshotting:
One of LynxSecure key features is its ability to take, in real-time, micro-snapshots of every relevant part
of the HW (memory, disk, CPU etc.). Snapshots of Windows entire memory (or parts of it) can also be
taken. This is not a one-time action, but can be configured to take snapshots at any chosen time interval
(polling mechanism).
The use of a virtualized disk allows detection of tampering of the MBR at the block I/O level, meaning --
LynxSecure can detect the writes to the MBR as they occur(!) .
Being OS-agnostic, LynxSecure does not rely upon native Windows APIs and the sanctity of Windows’
virtual memory system to take its view of RAM, nor does it rely upon Windows’ virtual memory
subsystem being intact.
The taken snapshots are stored securely, tamper-proof, and are used to compare various states of these
HW parts vs. previous or pristine versions of them, as well as to restore them. The snapshot comparison
is done dynamically and in real-time, as is the restore to pristine/clean state (i.e., unlike other solutions,
the machine need not be taken offline, or to a forensic lab for analysis and remediation).
Once taken, snapshots can also be exported in real-time to a remote host (either LynxSecure
management system of any 3rd party system connected via API). This feature allows for real-time large-
scale detection and monitoring of rootkit infection, as well as building a big picture of live evolution of
the rootkit infection. It also allows taking countermeasures very rapidly.
Dynamic real-time “compare & restore”:
At any given time, and when instructed to do so, LynxSecure can dynamically restore the pristine un-
tampered image of the infected part (the “Golden Image”), and even succeed doing so if 1st boot of
system is in a dirty environment. For example, if a new Windows installation is booted on LynxSecure,
and that Windows was already infected with a rootkit (prior to being booted on LynxSecure), LynxSecure
can detect that infection and either flag it to a system administrator, or take direct action to restore
(depending on configuration).
This “compare & restore” is done in real-time and can also be configured to be done automatically
without any human intervention.
Based on the policy, LynxSecure can be instructed to boot the infected guest OS (“dirty boot”) for
purpose of analysis.
The “Golden Image” is persistent and survives hard reboots and soft reboots.
-- LynuxWorks Proprietary & Confidential --
8. 8
Configurability:
LynxSecure is highly configurable. Each of the above scanning, snapshotting and restore functions can be
configured and fine-tuned according to needs. Each function can be completely or conditionally turned
on and off.
Network Monitoring:
Rootkits are typically associated with botnets, communicating continuously with their command &
control (C&C) servers. Towards that end LynxSecure can monitor network device access, especially in
conjunction with the boot process.
No single point of failure:
LynxSecure is superior to other hypervisors in that other hypervisors, which rely upon a single point of
failure, like dom0 in Xen (the single super-user domain), can have that failure point affect all analysis
engines and VMs in that system. LynxSecure VBIOS, FVS and VDS, on the other hand, are not a single
point of failure. Each guest OS has its own separate VBIOS/FVS.
Running underneath Windows, LynxSecure does not depend upon Windows (or any other guest OS), nor
rely upon Windows kernel APIs, Windows drivers, or other attack vectors of rootkits (unlike common
Windows-resident security clients).
Highlighted Case: TDL-4 Rootkit
TDL-4 is the most wide-spread rootkit, one of the most potent and persistent of all rootkits. There’s an
abundance of research about this rootkit (and its variants) and its anatomy. In this highlighted case we
showcase LynxSecure ability to detect, block and remediate infected target.
Configuration of target:
The target used was a generic Dell Latitude Laptop, with a vanilla Windows 7 guest installed on top
of LynxSecure.
LynxSecure was configured to include a "Golden Image" of the block containing the Windows
Master Boot Record (MBR) from a native Windows installation, as well as one of the last disk sector.
The “Golden Image” is a block of the disk.
The Golden MBR is stored with LynxSecure in a location that cannot be tampered with by Windows
at runtime. The protection mechanisms of the LynxSecure hypervisor assure this.
Sequence (note: all stages & actions are done in real--time):
Non-infected target:
1. In order to launch Windows, LynxSecure first launches the LynxSecure Virtual BIOS (VBIOS).
2. VBIOS identifies the boot media associated with the Windows installation, then performs a scan of
certain blocks of that device. In this particular configuration, the 0th block was scanned (LBA 0) and
a snapshot of this block was taken. This block contains the MBR and is also stored by LynxSecure in
location that cannot be accessed by Windows. (Note: This snapshot is not the “Golden Image” --
they are two separate disk block images). The same was done with the disk last sector.
On every hard boot and soft boot (reboot) of Windows, a new snapshot is taken, and it is always
compared to the “Golden Image”. The “Golden image” does not change, the snapshot can; in the
infected case it does.
3. Next, the snapshot is compared to the “Golden Image”. As this is the initial boot of Windows, the
snapshot matches the “Golden Image”, so the VBIOS creates an audit record.
-- LynuxWorks Proprietary & Confidential --
9. 9
4. The audit record is exported to the auxiliary guest OS running in the same system (VDS). The VDS
forwards the record to the remote LynxSecure management system.
5. The VBIOS boots Windows, and Windows continues its own boot process.
Instantiation of rootkit infection:
1. The rootkit dropper is activated by manually clicking on the dropper executable.
2. The rootkit installs itself onto the drive, and infects the MBR. Immediately, the rootkit attempts to
reboot Windows in order to activate and avoid detection.
3. Upon the reboot attempt of Windows, LynxSecure VBIOS is activated again by the hypervisor, scans
the 0th block and takes a new snapshot of the 0th block (i.e., the block containing the Windows
MBR).
Remediation sequence:
1. VBIOS takes another snapshot of the MBR then compares it to the “Golden Image” and the prior
snapshot.
2. As this new snapshot does not match the
“Golden Image”, an audit record is
created indicating the lack of a match
against the “Golden Image”.
3. The audit record is exported to the VDS; the VDS sends the data to the remote LynxSecure
management system.
4. Immediately after VBIOS generates the audit record, it suspends Windows before it completes its
boot.
5. LynxSecure management system prompts
a user/agent with a decision choice:
Either proceed with booting the
tampered system, or restore the system
with the “Golden Image”, and reboot it.
6. The user/agent makes the choice to
restore the “Golden Image”.
7. The data is sent to the VDS.
8. VDS provides the data to the VBIOS,
which boots Windows.
Note: VBIOS is a separate runtime entity
from the VDS. LynxSecure provides a
separate VBIOS for each fully virtualized
guest OS at runtime.
9. If “restore” option is chosen:
a. VBIOS over-writes the MBR on the
disk with the “Golden Image”.
b. Windows boots, clean of rootkit.
10. If “restore” option is not chosen:
a. VBIOS boots the existing
(infected) MBR that’s on the disk.
b. The “Golden Image” remains, of course, for possible later usage.
-- LynuxWorks Proprietary & Confidential --
11. 11
Usage of LynxSecure
Usage of virtualization as a means to provide security is a novel and radical concept. Secure
virtualization, such as LynxSecure, is a double-edged security solution:
1. It secures the machine on which it is installed by virtue of its secure design and provides complete
control and manageability
2. It’s capable of proactively protect against cyber-threats.
Scope of detection:
LynxSecure’s protection includes built in-mechanisms and programmatic APIs, to scan for both “absence
of good” and “presence of evil”, in a hardened, in-line, real-time environment.
LynxSecure is capable of detecting rootkits targeting master boot records, volume/partition boot
records, slack disk sectors, platform architecture properties (IDTs, GDTs), guest OS software constructs
(SSDTs), and other portions of guest OS storage and memory at runtime.
Using LynxSecure for rootkit & APT research:
LynxSecure is a vital tool for those engaged in research and analysis of rootkits. It can serve as an acute
sensor for zero-day rootkits and its real-time activity can significantly enhance and speed-up the
capabilities to detect rootkits and generate data about their activity.
A significant “productivity bonus” is the fact that once infected, rootkit test-beds need not be
completely restored in a lengthy offline process, but simply restored in real-time using LynxSecure
native restore function.
Using LynxSecure as a “rootkit sensor” in IT networks:
LynxSecure can serve as a vital and one-of-a-kind rootkit sensor in large IT networks, providing IT staff
with immediate information about rootkit infections & their dynamics. In this configuration, LynxSecure
allows for immediate actions (i.e., remove certain nodes out of the network, block certain network
segments etc.) to block and contain cyber-attacks, saving the currently unbearably-long discovery &
response time. The ability to prevent an infected node from spreading the infection throughout the
network is literally priceless.
-- LynuxWorks Proprietary & Confidential --
12. 12
Rootkits & Bootkits 101:
Much has been said about rootkits and bootkits, but still the unknown is much bigger than the known.
What is a rootkit?
What separates a rootkit from a regular Trojan is that a rootkit, by definition, occupies Ring 0, also
known as root or kernel level, the highest run privilege available, which is where the OS (Operating
System) itself runs. Non-rootkit trojans typically run in Ring 3, or user level, which is where ordinary
applications run, though some sources refer to userland trojans as “rootkits” also. Usually, but not
always, a rootkit will actively obfuscate and attempt to hide its presence from the user and any security
software present. Rootkits subvert the OS through the kernel (core operating system) or privileged
drivers. This enables a rootkit to operate as a part of the OS itself rather than a program being run by
the OS. This high level of sophistication makes rootkits extremely difficult to detect and remove. Often
anti-virus products will be unable to detect or remove a rootkit once it has taken over the OS and more
specialized detection and removal procedures are required. (source: SANS Institute, 2011)
More specifically:
Rootkit installs into the OS file-system and lower – the master boot record (MBR) -- and hooks into
OS data-structures.
It circumvents anti-malware clients and disables or cripples them. It performs its network
communication with its command & control server (the botnet) at levels 1 & 2, and is therefore out
of reach of OS-based security applications and anti-virus software.
It changes its behavior dynamically and utilizes elaborate polymorphism.
There is no existing zero-day/proactive protection against bootkit to date. If some rootkit activity is
detected, the protection and removal must be done in reactive offline mode only.
What is a bootkit?
Bootkit is the stealthiest form of rootkit, the most persistent one and the hardest to remove once
detected. It’s also considered as the most sophisticated form of rootkit. Bootkit installs itself into the
Master Boot Record (MBR), other parts of the boot sectors and hidden disk sectors.
MBR is the portion of the hard drive that tells the BIOS where to find the OS. This is a critical handoff of
responsibility between the BIOS which does the initial boot sequence when the computer is started and
the OS which takes over. By subverting this process the bootkit is able to inject itself between the
computer's hardware and OS, subtly altering data sent back and forth to mask its presence and take
over the system.
Every time the OS tries to read files from the hard drive the bootkit intercepts the attempt and
substitutes either fake data to hide itself or modified data to trick the OS into loading and executing
infected files. By selectively intercepting attempts to read and execute kernel drivers the bootkit loads
itself into memory and takes over the OS. If the user attempts to view the bootkit files, the bootkit can
give a false report of there being no trace of its files. Since the bootkit often never actually modifies the
OS files on the hard drive itself, but only gives modified data when the file is being loaded into memory,
it becomes even harder to detect. It can also detect and intercept any attempt to delete the bootkit
itself or any portion thereof. Even if the bootkit is deleted, since it is loaded in the MBR, the system can
be re-infected when it is rebooted.
By being situated lower than the OS, it enjoys security privilege level higher than those of the OS it’s set
to attack, thus gaining control over the entire OS. Being hidden so low also makes it invisible and
-- LynuxWorks Proprietary & Confidential --
13. 13
undetectable by the OS. This gives the bootkit complete freedom of action, and also allows it reinstall
itself into the OS if those parts have been detected and cleaned by the anti-malware client. (source:
SANS Institute, 2011)
No wonder the most common of all bootkits – TDL-4/TDSS has been described by Kaspersky Labs as
“indestructible”…
# Name Type % of Total
1 Win32.Bot.ZeroAccess Bot 16.87
2 Win32.Backdoor.TDSS Bot 10.03
3 Win32.Downloader.Agent.TK Downloader 6.51
4 Win32.Trojan.Alureon.A Bot 6.28
5 MAC.Bot.Flashback.K/I Bot 4.14
6 Win32.BankingTrojan.Zeus BankingTrojan 3.83
7 Win32.Bot.Alureon/TDL/TDSS Bot 3.39
8 Win32.Virus.Sality.AT Virus 2.21
9 DNS.Trojan.DNSchanger Trojan 1.91
10 Win32.Trojan.Medfos.A Trojan 1.87
* Source: Kindsight 2012 malware report
-- LynuxWorks Proprietary & Confidential --