1) The document discusses several services that have been developed in the virtualization plane, including Remus for transparent high availability, Parallax for virtual storage, and Tralfamadore for enhancing and understanding systems. 2) Tralfamadore continuously logs system execution over long periods, re-executes slices to generate indexes, and allows querying the history to search for specific events or states. 3) These services demonstrate how the virtualization plane provides opportunities to build low-level extensions that can improve areas like availability, storage, and debugging of software.

1. Services in the Virtualization Plane
Andrew Warfield
Adjunct Professor, UBC
Technical Director, Citrix Systems

2. The Virtualization Plane
Applications Applications
OS
Physical Machine

3. 20ms in the Linux Kernel
20ms in the Linux Kernel
• •355,000 Branches
355,000 Branches
• •350 Syscalls, 312 Interrupts, 255 PFs
350 Syscalls, 312 Interrupts, 255 PFs
• •And this is less than 10% of what
And this is less than 10% of what
happened in that 20ms!
happened in that 20ms!
• Geoff’s call graph goes on this slide.

4. The Virtualization Plane
Applications Applications
OS
Virtual Machine Monitor
Physical Machine

5. The Virtualization Plane
Appliance
VM Appliance
VM Appliance
VM Appliance
VM Appliance
VM Appliance
VM
Virtualization Plane
Virtual Machine Monitor Virtual Machine Monitor Virtual Machine Monitor
Physical Machine Physical Machine Physical Machine
• Huge opportunity for innovation.
• OS agnostic and hardware agnostic.
• Build useful services that are co‐located, but
isolated from VMs.
• Live migration was the first example.

6. Overview

7. Graduate Student: Brendan Cully
REMUS: TRANSPARENT HIGH
AVAILABILITY

8. Example 1: (Brendan Cully)
Remus: Transparent High Availability
• As with process migration, HA is complicated
and difficult to maintain.
• Database HA is generally based around a
replicated log, and a recovery protocol based
around detailed application semantics.
• HA is for “the very rich and the very scared.”
• Idea: Use simple mechanisms in the
virtualization layer to provide universal HA.

9. Remus
3ms
Mail Server Mail Server
<17ms
VM VM
PVM PVM
Xen Xen

10. Remus
3ms
Mail Server Mail Server
<17ms
VM VM
PVM PVM
“checkpoint ok!”
Xen Xen
Internet

11. Remus
Mail Server Mail Server
Remus demonstrates that efficient and complete
Remus demonstrates that efficient and complete
VM VM
PVM PVM
state capture can provide hardware fault tolerant‐
state capture can provide hardware fault tolerant‐
Xen Xen
style whole‐system failover to unmodified
style whole‐system failover to unmodified
applications.
applications.
The simplicity of the approach is critical, because
The simplicity of the approach is critical, because
high availability and fault recovery code is
high availability and fault recovery code is
notoriously difficult to get right.
notoriously difficult to get right.
Internet

12. Remus: Current Work
• Disaster Tolerant Computing.
Extend HA/FT to work in the
wide area. Deployment
between UBC and TRU
~350km fiber connection.
• Exposing Remus to Applications.
Apply paravirtualization to transparent HA.
E.g. let a database know that some memory is
unprotected

13. Remus: Summary
• Published at NSDI 2008 – won “Best Paper”
award.
• Some patches in xen‐unstable, remaining
patches to appear over the next month or two.
• This summer, added support for HVM guests.

14. Graduate Student: Dutch Meyer
PARALLAX: VIRTUAL STORAGE FOR
VIRTUAL MACHINES

15. Parallax: Storage Virtualization for
VMs
• VMs are fantastic, but turn out to be a bit clunky
to work with.
• VM images are really big, and most storage
systems don’t really provide the operations you
want to really innovate with VMs.
• Horizontal scale: create lots of images based on a
gold master.
• Vertical Scale: Lots of snapshots of a single image.
• Thin provisioning is critical, as is low‐cost storage.
• Parallax is basically just page tables for disks!

16. Parallax: Storage Virtualization for
VMs

17. Parallax
•Virtualizes block devices using
•Virtualizes block devices using
address mapping trees.
address mapping trees.
•Very low overhead (2ms) snapshots.
•Very low overhead (2ms) snapshots.
•Equally low overhead image cloning.
•Equally low overhead image cloning.
•Space efficient.
•Space efficient.
•Many very interesting challenges.
•Many very interesting challenges.
metadata
data

18. Parallax Summary
• Parallax is used on a daily basis in our lab.
• Vertical integration of storage from array
through to guest.
• Horizontal integration across hosts in a cluster.
• Storage is no longer a barrier for deploying
new VMs.
• In the process of adding powerful new
features: deduping, linearization, and CAS.

19. Graduate Students: Geoffrey Lefebvre and Brendan Cully
TRALFAMADORE: ENHANCING AND
UNDERSTANDING SYSTEMS

20. Tralfamadore: Motivation
• How much do we really know about what
software is doing, especially when things go
wrong?
• What if we had a detailed recording?
• What if the recording was interactive and
could be queried and changed?

21. Tralfamadore
Remus
RAM
RAM
RAM
RAM
Checkpoints
RAM Disk
RAM Disk
Disk
Disk
Disk
Disk
Parallax
Test/Dev Production
1. Continuously log execution for long periods of time.
1. Continuously log execution for long periods of time. Network Network

22. Tralfamadore
Remus
RAM
RAM
RAM
RAM
Checkpoints
RAM Disk
RAM Disk
Disk
Disk
Disk
Disk
Execution
Index Parallax
Indexing Servers
Test/Dev Production
2. Re‐execute slices of history to generate indexes.
2. Re‐execute slices of history to generate indexes. Network Network

23. Tralfamadore
Remus
RAM
RAM
RAM
RAM
Checkpoints
RAM Disk
RAM Disk
Disk
Disk
Disk
Disk
Execution
Index Parallax
Indexing Servers
“Find points in execution when…”
• “…/etc/passwd was modified”
• “…network receive buffers were
overloaded.”
• “… the stack looked like this.”
Query Servers
• “… control flow ran function b,
shortly after running function a.”
Test/Dev Production
3. Queries to search, select, and interrogate history.
3. Queries to search, select, and interrogate history. Network Network

24. Tralfamadore
Remus
RAM
RAM
RAM
RAM
Checkpoints
RAM Disk
RAM Disk
Disk
Disk
Disk
Disk
Execution
Index Parallax
Indexing Servers
Use piles of existing tools:
• Emulators,
• Profilers,
•Binary Patching,
•Debuggers
Query Servers Re‐execution
Servers
Use repeated re‐execution
to tackle non‐determinism.
Test/Dev Production
4. Re‐execute and analyze modified checkpoints.
4. Re‐execute and analyze modified checkpoints. Network Network

25. Applications of Tralfamadore
• Fault injection / fuzzing of software close to real
• Fault injection / fuzzing of software close to real
execution experience.
execution experience.
• Retrospectively attach a debugger to any node in
• Retrospectively attach a debugger to any node in
this call graph.
this call graph.
• Test patches in the past!
• Test patches in the past!

26. Understanding Execution
• Prototype application of Tralfamadore
performs dynamic analysis of trace data and
maps it back on top of source code.
• Allows developers to understand how source
is behaving in deployments.
• Very early work, but a few examples follow…

30. Tralfamadore Summary
• We hope to be able to do detailed,
retrospective analysis of system behaviour.
• Current focus has been on understanding
execution.
• Future work will involve performance and
security analysis and assisting reproduction of
system failures.

31. Overall Conclusions
• The virtualization plane presents a great
opportunity to build low‐level extensions to
software.
• I have shown three example services, and
expect many more to follow.
• Interesting challenges exist in evolving
virtualization to provide these services while
maintaining isolation and cross‐platform
benefits.

32. Thank You!