1. POTASSIUM:
Penetration Testing
as a Service
Richard Li |Hyun-wook Baek | Dallin Abendroth
Xing Lin |Robert Ricci | Yuankai Guo | Jacobus Van der Merwe
Presented by
A. Farár
2. Motivation
• What’s the Problem?
Pentesting on a production system is great, because the exact dynamic state of system
is captured. But there is high risk of damage: data loss, crash. System unavailable or
malware infection…
Testing against a separate system that has been designed to model the live one is also
not ideal. As the production system is not captured, thus reducing the value of the
test.
3. POTASSIUM PTaaS
• Pentesting Service in the Cloud.
POTASSIUM uses techniques developed for live migration of virtual machines to clone them,
capturing their full disk, memory, and network state. The cloned system is then isolated
from the rest of the cloud, so that effects from the penetration test will not damage other
tenants. Because the penetration tester owns the cloned system, testing is more thorough
and efficient.
7. Live Consistent Checkpointing
State Captured at
Single Instance
Snapshots
Transparent
Live ConsistentIterative
P2 must be delivered after time t4 VM state machine for consistent CP
8. Pentesting Process
Pentest Manager AttackersCoordinator
Mirror Subproject data
VM IP addresses,
attackers assignment
sheet
Relays commands to
attackers
Collects session info.
From attackers
Vulnerability report
generated at end of test
Metasploit auto pentest
• Simple, automated.
9. Pentesting Modes
Isolated Automated Scalable
Separate Availability Zones Metasploit Manage Multiple Pentests at Once
Emulate Internal /External Attacks
10. Evaluation
• Measurement: end-to-end time to perform pentesting as a
function of the number of VMs in the production project.
Performance Impact
of Snapshot
Checkpointing
Minimal
Consistent 68.5
Non-consistent 69.6
Pentest
Effectiveness
(2 test cases)
WordPress
Vulnerability
detected
Scalability Up to 100 VMs
11. Evaluation
• Measurement: end-to-end time to perform pentesting as a
function of the number of VMs in the production project.
HTTP Response Times
(ms)
Baseline 67.4
Non-consistent 69.6
Consistent 68.5
Consistency
Packet loss in VM1>VM0
steam
Automated Pentesting
Mirror Creation 227.59
Attacker Creation 77.56
Pentesting 35.99
Miscellaneous 0.87
12. Analysis
Strengths
• Automated pentest
• Economies of Scale
• No performance impact
on production systems
• Availability & Integrity
Weaknesses
• Automated Pentest
• Difficult to bring external
resources into the closed
system (i.e. cloud-wide
storage or DB services.
• Possible Confidentiality
concerns
14. References
Richard Li, et al. (2015), POTASSIUM: Penetration Testing as a Service
Proceedings of the Sixth ACM Symposium on Cloud Computing (SoCC '15)
Hinweis der Redaktion
Pentesting on a production system is great, because the exact dynamic state of system is captured. But there is high risk of damage: data loss, crash. System unavailable or malware infection…
Testing against a separate system that has been designed to model the live one is also not ideal. As the production system is not captured, thus reducing the value of the test.
Penetration testing is the process where you probe network systems for vulnerabilities. POTASSIUM uses techniques developed for live migration of virtual machines to clone them, capturing their full disk, memory, and network state. The cloned system is then isolated from the rest of the cloud, so that effects from the penetration test will not damage other tenants. Because the penetration tester owns the cloned system, testing is more thorough and efficient.
POTASSIUM PTaaS has five design principles:
Validity – results are the same on mirror system as on the production system
Availability & Integrity – the pentest must have low impact on the performance and availability of the production system
Safety – pentesting activity must not affect production projects or other systems on the Internet
Scalability – ability to manage multiple pentest projects at once and deploy different strategies for allocating and positioning attacker VMs
Extensibility – the design can support many pentesting tools
Step A – is the Production system in the cloud. Tenant has a allocated a collection of resources including several VMs and a network.
Step B - A new project or copy of the production system is created using standard APIs of the cloud management system. It contains metadata, and has the same structure as the production project, but not the same internal state. This copy is referred to as the pentest project.
Step C - A consistent snapshot of the production project is created, including all VM memory contents, disk contents, and network packets in flight. That state is inserted into the pentest project.
Step D - Attacking resources are allocated and added to the pentest project; then the pentest is performed. The pentest project consists of two parts: the mirror subproject, which is the set of resources that mirror the production project, and the attack subproject, which is the set of resources introduced for pentesting. The pentest project is isolated so that the effects of the penetration test cannot harm other tenants.
Potassium is based on OpenStack, an open source software for creating private and public clouds.
To ensure Availability, the Project Creator places pentest projects on physical resources that are separate from those used by production projects. This is achievable by using standard cloud APIs such as availability zones. Also, the Snapshot Agents perform live consistent checkpointing, allowing the production system to execute while it is being checkpointed. Although performance may be reduced during the time it takes to checkpoint, the project still remains available to clients.
For Validity, Project Creator, Snapshot Manager, and Snapshot Agents work together to replicate the full state of the production project within the mirror subproject of a pentest project.
The mirror is created in two steps: 1) the Project Creator obtains the metadata of the production project and invokes the cloud platform to create a mirror with the same metadata, and 2) the full state of the VMs and network in the production project is recreated via the Snapshot Agents that perform live, consistent checkpointing over the production project. Consistency means that the state of the production project is captured at a single, logical instant of time. Once a VM has completed its snapshot, any packet that it sends will not be delivered until the recipient VM has also completed its snapshot.
Safety is implemented via the Project Creator, which uses the standard APIs of the cloud platform to disconnect the pentest project from any other network, except for an access route that allows POTASSIUM’s Pentest Manager to communicate with the attack Coordinator within the pentest project. Only the Coordinator has permission to send traffic outside of the pentest project, cannot relay traffic between the “inside” and the “outside” of the pentest project.The cloud platform is trusted, so POTASSIUM is not intended for pentests that aim to compromise the underlying cloud platform or hypervisors.
POTASSIUM implemented the Metaspolit Framework as its automated pentesting tool. However, any pentesting tool can be used, satisfying the Extensibility design principle. The Coordinator serves as an adapter between POTASSIUM’s Pentest Manager and the implementations of the Attackers.
POTASSIUM can insert attacking VMs into the mirror subproject’s internal networks, and capture VMs within the mirror subproject, emulating internal attacks.
Scalability is fulfilled as POTASSIUM can manage multiple pentest projects at once (i.e., two separate users running pentests at the same time or concurrent pentests over a single production project). Additionally, it implements multiple strategies for allocating and positioning Attackers against a mirror subproject, for example, to emulate both external and internal attacks. By allocating large numbers of attacker VMs, POTASSIUM is able to swap space for time by performing pentests against multiple hosts in parallel.
POTASSIUM’s Snapshot Manager and Snapshot Agents implement a live, consistent checkpointing algorithm that creates snapshots of a production project.
The prototype implementation uses QEMU’s live-snapshot mechanism to independently take a live snapshot for each individual VM in the production project. It uses packet coloring and buffering to deal with inconsistent packets.
Snapshots = each VM saves its memory state by performing iterative memory copying
Live = snapshot taken transparently
Consistent = state captured at single instance in time
The figure on the bottom left shows an example of a checkpoint timeline. For live, consistent checkpointing, packet P2 must be delivered after time t4.
The figure on the bottom right shows an example of a VM state machine for consistent checkpointing. “Each VM in a production project is associated with an instance of the state machine shown with a VM beginning in a DEFAULT state. When POTASSIUM needs to checkpoint a production project, the Snapshot Manager sends a START_SNAPSHOT command to each VM, via the Snapshot Agents. Then, each VM transitions to the STARTED state and begins to take its snapshot. When a VM completes its snapshot, it transitions to the COMPLETED state. The Snapshot Manager periodically checks the status of each VM, and when all have completed their snapshots, the Snapshot Manager sends an ALL_SNAPSHOTS_COMPLETE command to every VM. Each VM then transitions to the ALL_COMPLETED state.”
The pentesting process in POTASSIUM is simple and automated. The Pentest Manager forwards Mirror Subproject data, VM IP addresses, and the attackers assignment sheet to the Coordinator.
The Coordinator then relays the commands/data to attackers; then subsequently collects session information from Attackers. A vulnerability report is generated at the end of the test.
The Attackers use the Metasploit framework to run an automated pentest.
There are three pentesting modes: Internal, External and Pivot
Internal – creates multiple Attackers and attaches one to each network within the mirror subproject. Penetration testing may be directly performed by Attackers on the VMs in the mirror subproject, irrespective of whether those VMs can be reached from an external network. Overall vulnerabilities exposed in this mode.
External – the attack subproject is attached to the mirror subproject to emulate an external attacker. Can be used to test correctness of security group rules.
Pivot – pentesting is performed in multiple rounds from Attackers that replace VMs in the mirror subproject. This mode imitates the way an intruder is able to attack new targets from the point of view of an already compromised VM. Useful for “what if” analysis.
Connectivity between mirror subproject and External network is disabled to prevent traffic leakage.
OpenStack Security Groups allow attackers to be controlled by the Coordinator and reach VMs.
Availability Zones is a standard cloud API that separates pentest projects on physical resources from those used by production projects.
Performance impact of Snapshot CP was minimal with a negligible difference between consistent and non-consistent.
The Pentest was effective and positively detected the WordPress vulnerabilities.
POTASSIUM scaled well up to 100 VMs.
HTTP response times ranged from 67% to 69.6%.
Consistency tests shoed a packet loss in VM1>VM0 stream.
Automatic pentest performance test showed mirror creation took the longest time (227.59), followed by attacker creation (77.56), and actual pentesting (35.99).
