1. Embedded Virtual Machines: Architecture Description,
Technical Report ESE-2009-01
Miroslav Pajic and Rahul Mangharam
September, 2009
Abstract
Embedded wireless networks have largely focused on open-loop sensing and monitoring.
To address actuation in closed-loop wireless control systems there is a strong need to re-think
the communication architectures and protocols for reliability, coordination and control. As
the links, nodes and topology of wireless systems are inherently unreliable, such time-critical
and safety-critical applications require programming abstractions where the tasks are assigned
to the sensors, actuators and controllers as a single component rather than statically mapping
a set of tasks to a specic physical node at design time. To this end, we introduce the Embed-
ded Virtual Machine (EVM), a powerful and exible programming abstraction where virtual
components and their properties are maintained across node boundaries. In the context of pro-
cess and discrete control, an EVM is the distributed runtime system that dynamically selects
primary-backup sets of controllers to guarantee QoS given spatial and temporal constraints
of the underlying wireless network. The EVM architecture denes explicit mechanisms for
control, data and fault communication within the virtual component. EVM-based algorithms
introduce new capabilities such as predictable outcomes and provably minimal graceful degra-
dation during sensor/actuator failure, adaptation to mode changes and runtime optimization
of resource consumption. Through the design of a natural gas process plant hardware-in-
loop simulation we aim to demonstrate the preliminary capabilities of EVM-based wireless
networks.
1 Introduction
Automation control systems form the basis for signicant pieces of our nation's critical infrastruc-
ture. Time-critical and safety-critical automation systems are at the heart of essential infrastruc-
tures such as oil reneries, automated factories, logistics and power generation systems. Discrete
and process control represent an important domain for real-time embedded systems with over a
trillion dollars in installed systems and $90 billion in projected revenues for 2008 [1].
In order to meet the reliability requirements, automation systems are traditionally severely
constrained along three dimensions, namely, operating resources, scalability of interconnected
systems and exibility to mode changes. Oil reneries, for example, are built to operate without
interruption for over 25 years and can never be shutdown for preventive maintenance or upgrades.
They are built with rigid ranges of operating throughput and require a signicant re-haul to adapt
to changing market conditions. This rigidity has resulted in proprietary systems with limited scope
for re-appropriation of resources during faults and retooling to match design changes on-demand.
For example, automotive assembly lines lose an average of $22,000 per minute of downtime [2]
during system faults. This has created a culture where the operating engineer is forced to patch a
faulty unit in an ad hoc manner which often necessitates masking certain sensor inputs to let the
operation proceed. This process of unsystematic alteration to the system exacerbates the problem
and makes the assembly line dicult and expensive to operate, maintain and modify.
Embedded Wireless Sensor-Actuator-Controller (WSAC) networks are emerging as a practical
means to monitor and operate automation systems with lower setup/maintenance costs. While
1

2. Figure 1: (a) A wireless sensor, actuator and controller network. (b) Algorithm assignment to a set
of controllers, each mapped to the respective nodes. (c) Three Virtual Components, each composed of
several network elements
the physical benets of wireless, in terms of cable replacement, are apparent, automation manu-
facturers and plant owners have increasing interest in the logical benets.
With multi-hop WSAC networks, it is possible to build modular systems which can be swapped
out for o-line maintenance during faults. Modular systems can be dynamically assigned to
be primary or backup on the basis of available resources or availability of the desired calibra-
tion. Modularity allows for incremental expansion of the plant and is a major consideration in
emerging economies. WSAC networks allow for runtime conguration where resources can be
re-appropriated on-demand, for example when throughput targets change due to lower price elec-
tricity during o-peak hours or due to seasonal changes in end-to-end demand.
While WSAC networks facilitate both planned and unplanned mode changes, runtime pro-
grammable WSAC networks allow for exible item-by-item process customization. For example,
a high demand for fuel-ecient Toyota Prius' will require major retooling of a traditional wired
factory that is designed for the Toyota Camry chassis. With re-programmable WSAC, the assem-
bly line stations can adapt to a schedule where every 3 Camrys are interleaved with 2 Prius' with
synchronized changes in operation modes and assembly line operations.
1.1 Embedded Virtual Machines
The current generation of embedded wireless systems has largely focused on open-loop sensing and
monitoring applications. To address actuation in closed-loop wireless control systems there is a
strong need to re-think the communication architectures and protocols for reliability, coordination
and control. As the links, nodes and topology of wireless systems are inherently unreliable, such
time-critical and safety-critical applications require programming abstractions where the tasks are
assigned to the sensors, actuators and controllers as a single component rather than statically
mapping a set of tasks to a specic physical node at design time (see Fig. 1). Such wireless
controller grids are composed of many wireless nodes, each of which share a common sense of
the control application but without regard to physical node boundaries. Our approach, as shown
in Fig. 2(a), is to decouple the functionality (i.e. tasks) from the inherently unreliable physical
substrate (i.e. nodes) and allow tasks to migrate/adapt to changes in the underlying topology.
To this end, we introduce the Embedded Virtual Machine (EVM), a powerful and exible
programming abstraction where virtual components and their properties are maintained across
node boundaries. EVMs dier from classical virtual machines (VM). In the enterprise or on PCs,
one (powerful) physical machine may be partitioned to host multiple virtual machines for higher
resource utilization. On the other hand, in the embedded domain, an EVM is composed across
multiple physical nodes with a goal to maintain correct and high-delity operation even under
changes in the physical composition of the network. The goal of the EVM is to maintain a set of
2

3. functional invariants, such as a control law and para-functional invariants such as timeliness con-
straints, fault tolerance and safety standards across a set of controllers given the spatio-temporal
changes in the physical network. By incorporating EVMs in existing and future wireless automa-
tion systems, our aim is to realize:
1. Predictable outcomes in the presence of controller failure. During node or link faults, EVM
algorithms determine if and when tasks should be reassigned and provide the mechanisms for
timely state migration.
2. Provably minimal QoS degradation without violating safety. In the case of (unplanned)
topology changes of the wireless control network, potential safety violations are routine occurrences
and hence the EVM must reorganize resources and task assignments to suit the current resource
availability (i.e. link bandwidth, available processing capacity, memory usage, sensor inputs, etc.).
3. Composable and recongurable runtime system through synthesis In the EVM approach,
a collection of sensors, actuators and controllers make a Virtual Component as shown in Fig. 1.
A Virtual Component is a composition of interconnected communicating physical components
dened by object transfer relationships. At runtime, nodes determine the task-set and operating
points of dierent controllers in the Virtual Component. This machine-to-machine coordination
requires task-set generation, task migration and remote algorithm activation which are executed
via synthesis at runtime.
4. Runtime Resource Re-appropriation and Optimization for dynamic changes in service.
For planned system changes such as a factory shift, increase in output or retooling for a dierent
chassis, nodes are required to be re-scheduled in a timely and work conserving manner.
1.2 Challenges with Wireless Control
While there has been considerable research in the general area of wireless sensor networks, a ma-
jority of the work has been on open-loop and non-real time monitoring application. As we extend
the existing programming paradigm to closed-loop control applications with tight timeliness and
safety requirements, we identify four primary challenges with the design, analysis and deployment
of extending such networks:
1. Programming motes in the event-triggered paradigm is tedious for control
networks. It is hard to provide any analytical bounds on the response time and stability of
tasks in an event-driven regime [3, 4]. Real-time tasks are time-triggered while sensor inputs are
event-triggered. It is easier to incorporate sporadic tasks in a time-triggered regime than vice
versa.
2. Programming of sensor networks is at the physical node-level where the tasks are
bound to the node at compile-time. It is non-trivial to decompose a large control problem into
dening components and applications for each mote.
3. Design of systems with exible topologies is hardTaskphysical node-level program-
Runtime with Management
ming as the set of tasks (or responsibility) is associated with the physical node. Thus, a change
• Task_Migrate
in the link capacity, node energy level or connectivity in the current topology will render the ap-
– Task instructions
plication useless. It is necessary to associate – Stack mapping of tasks to nodes and incorporate
a logical
mechanisms to transfer responsibilities during physical and environmental changes in the network
– Data
(as in Fig. 2(a)). – Associate libraries
– Control/Schedule/Resource meta data
VT1 VT2 ... VTn
Task_Migrate(T2)
Node Node
1 2
N1 N2 N3 ... Nm Current Task-set: T1, T2, T3 Current Task-set: T’1, T’2, …, T’7
(a) Decoupled virtual tasks and physical (b) Task migration for real-time operation (instructions, stack,
nodes with runtime task mapping data timing/fault tolerance meta-data) on one physical node
to another. 29
Figure 2: Programmatic control with decoupled virtual tasks and physical controller nodes
3

4. 4. Fault diagnostics, repair and recovery are manual and template-driven for a
majority of networked control systems. In the case of WSAC networks, it is not plausible to
exhaustively capture all possible faults at design time and thus provisions must be made for
runtime diagnostics and recovery.
1.3 Overview of the EVM Approach
While wireless system engineers optimize the physical, link and network layers to provide an ex-
pected packet error rate, this does not translate accurately to the stability of the control problem at
the application layer. For example, planned and unplanned changes in the network topology with
node/link failures are currently not easily captured or speciable in the metrics and requirements
for the control engineer. For a given plant connected to its set of controllers via wireless links it is
necessary that the controller process the sensor inputs and perform actuation within a bounded
sampling interval. While one approach is to design specialized wireless control algorithms that
are robust to specied range packet errors [5, 6], it is non-trivial to design the same for frequent
topological changes. Furthermore, it is dicult to extend the current network infrastructure to
add/remove nodes and to redistribute control algorithms to suit environmental changes such as
battery drain for battery-operated nodes, increased production during o-peak electricity pricing,
seasonal production throughput targets and operation mode changes.
The EVM approach is to allow the control engineer to utilize the same wired-network control
algorithms on the wireless network without knowledge of the underlying network protocols, node-
specic operating systems or hardware platforms. The virtual machine executing on each node
within the virtual component instruments the VC to adapt and recongure to changes while
ensuring the control algorithm is within its stability constraints. This approach is complementary
to the body of network control algorithms [5, 6] as it provides a logical abstraction of the underlying
physical node topology.
2 EVM Design Flow
Our focus is on the design and implementation of wireless controllers and in providing such
controllers with runtime mechanisms for robust operation in the face of spatio-temporal topo-
logical changes. Our focus is exclusively on robust controller design and not on sensors or ac-
tuators, as the latter are largely physical devices with node-bound functionality. In order to
enable control engineers to design wireless control systems in a manner that is both largely plat-
form/protocol/hardware architecture independent and extensible to dierent domains of control
systems in process, discrete, aviation, medical, etc. we present a three-layered design process. We
now describe the design ow from control problem formulation, automatic translation of control
models to platform-independent EVM interpreter-based code and nally to platform-dependent
binaries (see Fig. 2). These binaries are assigned to nodes within a VC using assignment and
scheduling algorithms presented in [7]. The binaries are executed as virtual tasks within the
node-specic architecture described in Section 3.
At design time, control systems are usually designed using software tools, such as Mat-
lab/Simulink, that incorporate both modeling and simulating capabilities. Therefore, in order
to automatize whole design ow, the EVM is able to automatically generate functional models
from the Simulink control system description to dene the processes by which input sampled
data is manipulated into output data for feedback and actuation. These functional models are
represented by generated code and meta data that are platform and node independent system
description, thus providing the system designer to exclusively focus on the control problem design.
From the functional description in the platform-independent and domain specic language,
the EVM design ow automatically extracts additional para-functional properties, like timing,
inter task dependencies, etc from the Simulink model. All these properties, along with functional
description are used to dene a platform optimized binary for each virtual task (VT). VTs have
4

5. the option of static or dynamic binding depending on the capabilities and timing requirements for
the given control problem specication.
2.1 Control Problem Synthesis
A functional description of a VT is automatically
extracted from the Simulink design, using the Design Control System Description
fact that each block can be represented as a com- Time (Matlab/Simulink, C…)
position of other Simulink blocks. Thus, each
model can be presented as a hierarchical com-
position of basic Simulink blocks.This organiza- Platform Domain-specific EVM Description
tion of Simulink models allows for natural extrac- Independent (Common-EVM and Control-EVM)
tion of a functional description using predened
words from the EVM programming language dic-
Platform
tionary (described in the next sub-section). Simi- Dependent Bin.
Virtual Component Manager/RTOS
(nano-RK)
larly, when a new block in Simulink is dened as Format
a composition of previously dened blocks, for
the EVM functional description, a new word is Figure 3: EVM Design ow
described using previously dened words, until a
level is reached where all words belong to the
EVM dictionary. Therefore, a VT description is obtained using a parser that processes the
Simulink .mdl le as a .txt le by searching for a new block denitions along with the inter-
connections between blocks. Since each block in Simulink design has to have an unique name, a
straightforward functional extraction would result in number of EVM words equal to the number
of used (sub)blocks. In order to avoid this, before each new word is dened the parser checks
whether a word with same structure already exist. This check can contain several hierarchical
levels and thus it can increase parser's execution time. An example of creation of a VT's func-
tional description from a Simulink design model in Fig. 4(a) is presented in Fig. 4(b). As the
platform-independent language is stack-based, the notation is in reverse.
2.2 Platform Independent Domain Specic Language
In order to generate functional description of the designed system, the EVM programming language
is based on FORTH, a structured, stack-based, extensible programming language[8]. On the other
hand, since the goal of EVM design is to allow exibility and designing utilities independent
of chosen programming language, the intermediate programming language is not constrained to
FORTH. This allows users to have a degree of freedom to replace Simulink with C, .NET, VB or
other descriptions for control system implementation.
The use of the EVM intermediate programming language enables domain-specic con-
structs, where basic programming libraries are related to the type of application that is being
developed. For example, for use in embedded wireless networks for industrial control, we devel-
oped two predened libraries, Common EVM ( Table 2), based on the standard FORTH library
[8], and Control EVM that contains functionality widely used for developing control applications
Fcn1 1
f(u) out 1
PID_controller 1
R1
2 Sum 2
in out 2
in 2
1 Sum
Saturation
out 2 : VTout1 SWAP f SUM ( x x – y ) ;
in 1
Fcn : VTout2 NEGATE R1 SUM kp1 ki1 kd1 0 0 PID SATURATE ( x – x ) ;
R2
f(u) : VTout4 NEGATE R2 SUM kp ki kd 0 0 PID SATURATE ( x – x ) ;
in out 4
3 Sum 3
out 4
: VT_ID ROT DUP ROT VTout1 DUP VTout4 2SWAP SWAP VTout1 DUP
Sum 1 PID_controller
in 3 Saturation 1 VTout4 2SWAP ( in1 in2 in3 – out1 out2 out3 out4 ) ;
3
out 3
(a) Example of a control algorithm (VT) modeled in (b) Generated code
Simulink
Figure 4: Generation of EVM functional description from Simulink model
5

6. Table 1: Control EVM
PID ( p KP KI KD addr_in addr_out - ) Proportional-Integral-Derivative controller
calculating p outputs in one invocation,
result is updated directly in memory,
since previous values are needed
PI ( p KP KI addr_in addr_out - ) Proportional-Integral controller
TF ( p m n addr_alpha addr_beta addr_in add_out - ) Transfer function described with mth (nth )
order polynomial in nominator (denominator)
with coecients stored in α and β
LTI ( p m n addr_A addr_B addr_C a_in a_x a_out - ) LTI system, x[k + 1] = Ax[k] + Bu[k],
y[k] = Cx[k], where p, m, n
are sizes of vectors n, y and x
(a list of all API's can be seen in Table 2 and Table 1). In addition the extensibility of EVM
allows to dene the Automotive EVM, Aviation EVM or Medical EVM libraries that will contain
functionalities specic to each of these application elds. Using these libraries code generator
creates a system description from a predened component, thus creating a task description le for
each of the Virtual Tasks (VTs).
Timing parameters (period and worst-case execution time) are also extracted from the model.
Since we consider only discrete controllers as potential VTs, Simulink design rules force the designer
to dene a sampling rate for each discrete block, allowing us to extract period as least common
divider for all task timing. These values are also extracted from the model (.mdl) le and added
to the VT's description le, along with the functional description.
List of the Control EVM words is presented in Table 1. First three words specied in the table
are used for the Singe-Input-Single-Output (SISO) systems. Although all these systems can be
described using the last word (LTI - describing Linear Time Invariant system), the fact that they
are widely exploited recommended its use.
6

7. Table 2: Common EVM
: Dene a word
Arithmetic operations (on 16bit):
+ ( n1 n2 - n3 ) Adds n1 and n2, result returned on the stack
- ( n1 n2 - n3 ) Subtract n2 from n1, result returned on the stack
1+ ( n1 - n2) Add 1 to n1, return the result on the stack
1- ( n1 - n2) Subtract 1 from n1, return the result on the stack
* ( n1 n2 - n3 ) Multiply (signed) n1 and n2, result returned on the stack
/ ( n1 n2 - n3 ) Divide n1 with n2, quotient returned on the stack
/MOD ( n1 n2 - n3 n4 ) Divide n1 with n2, quotient n3 and reminder n4 returned on the stack
MOD ( n1 n2 - n3 ) Divide n1 with n2, reminder returned on the stack
L ( x1 u - x2 ) Logical left shift of u places on x1, result returned on the stack
L ( x1 u - x2 ) Logical right shift of u places on x1, result returned on the stack
A ( x1 u - x2 ) Arithmetical left shift of u places on x1, result returned on the stack
A ( x1 u - x2 ) Arithmetical right shift of u places on x1, result returned on the stack
ABS ( n - |n|) Replace the top stack item with absolute value
MIN ( x1 x2 - x3 ) Lower of the two values is returned on the stack
MAX ( x1 x2 - x3 ) Higher of the two values is returned on the stack
Logical operations (on 16bit):
INV ( x1 - x3 ) Invert x1, result returned on the stack
AND ( x1 x2 - x3 ) ands x1 and x2, result returned on the stack
XOR ( x1 x2 - x3 ) xors x1 and x2, result returned on the stack
OR ( x1 x2 - x3 ) ors x1 and x2, result returned on the stack
NOR ( x1 x2 - x3 ) nors x1 and x2, result returned on the stack
NAND ( x1 x2 - x3 ) nands x1 and x2, result returned on the stack
Comparison and testing:
0 ( n - ag )
0 ( n - ag )
0 ( n - ag )
0= ( n - ag )
= ( n1 n2 - ag )
( n1 n2 - ag )
( n1 n2 - ag )
( n1 n2 - ag )
Memory manipulation:
@ ( addr - x) fetch
! ( x addr - ) write
malloc ( size - addr ) Allocation of 128B memory block
mfree ( addr - ) Frees 128B memory block
Sensor/Actuator handling:
RDSensG ( sensID - n1 ) Read sensor value from the sensor described by the global sensor ID
RDSensL ( sensID - n1 ) Read sensor value from the sensor described by the local sensor ID
WRActG ( value actID - ) Write value to the actuator described by the global actuator ID
WRActL ( value actID - ) Write value to the actuator described by the local actuator ID
Networking:
PktSendG ( addr n nodeID - ) Send n-byte message stored
at address addr to node described by global ID
7

8. 3 EVM Architecture
We now describe the node-specic architecture which implements the mechanisms for the virtual
machine on each node. The Common-EVM and Control-EVM description for each set of controllers
are scoped within Virtual Tasks that are mapped at runtime by the the task Assignment Algorithm
described in [7]. This description is interpreted by the interpreter running on each node. The EVM
runtime system is built on top of the nano-RK real-time operating system[9] as a supertask, allowing
node-specic tasks to execute natively and virtual tasks (VTs), i.e. those that are dynamically
coupled with a node, to run within the EVM. The EVM services are shown in Fig. 6 and the EVM
block-level reference architecture is presented in Fig. 5. This allows EVM to maintain node specic
functionalities and be extensible to runtime task evocation of existing or new virtual components.
The interface between nano-RK and all VTs is realized using the Virtual Component Manager
(VCM) supertask. The VCM maintains local resource reservations (CPU, network slots, memory,
etc.) within nano-RK, the local state of the VTs and global mapping of VTs within the virtual
component. The VCM is responsible for memory and network management for all VTs-to-physical
nodes and presents a mapping between local and remote ports which is transparent to all local VTs.
The VCM includes a FORTH-like interpreter for generic and domain-specic runtime operations
and a Fault/Failure Manager (FFM) for runtime fault-tolerant operation.
3.1 nano-RK Real-Time OS
To address the need for timing precision, priority scheduling and ne-grained resource management
the nano-RK resource kernel [9] has been previously developed with timeliness as a rst-class
concern. nano-RK is a fully preemptive RTOS with multi-hop networking support that runs on
a variety of sensor network platforms (8-bit Atmel-AVR, 16-bit TI-MSP430, Crossbow motes,
FireFly). All networking is conducted over the RT-Link [10] real-time link protocol that is native
to nano-RK. It supports xed-priority preemptive scheduling for ensuring that task deadlines are
met, along with support for and enforcement of CPU and network bandwidth reservations. Tasks
can specify their resource demands and the operating system provides timely, guaranteed and
controlled access to CPU cycles and network packets in resource constrained embedded sensor
environments. It also supports the concept of virtual energy reservations that allows the OS to
enforce energy budgets associated with a sensing task by controlling resource accesses. nano-RK
provides various medium access control and networking protocols including a low-power-listen
CSMA protocol called B-MAC, an implicit tree routing protocol and RT-Link.
For networked control systems, it is essential that the underlying sensor operating system ex-
pose precision timing, scheduled tasks and synchronized networking so that the trade-os between
energy-consumption (node lifetime), reliability and responsiveness are speciable and enforceable
both at design-time and runtime. Support for the above services is required for low-duty cycle and
energy-constrained sensor networks too because the computation and communication are packed
into a short duration so all nodes maximize their common sleep time.
nano-RK has been design as fully static OS, congured at the design time. Therefore to allow
Interp. FFM
Task Task
native tasks
VT 1
virtual tasks
...
VCM Task 1 ... Task n
VT m
nanoRK
Figure 5: EVM architecture with Virtual Component Manager running as a supertask along side native
nano-RK tasks
8

9. Motor Overload State Focus of EVM work
Apps
Control Detection Migration
Task Task Task
Reserves Reserves Reserves
Adaptive Virtual Machine
Real-Time Scheduler Network
Runtime System
Management
Kernel
Task Peripheral Parametric Control
Management Drivers Task Partitioning
RT-Link
RX Buffer Scheduleability Analysis
Reservations TX Buffer Software Attestation
Algorithm Activation
Protocol Adaptation
Hardware
Time Sync RX Power Control Policy Negotiation
Data Migration
Microcontroller 802.15.4 Radio Online Fault Diagnosis
Figure 6: nano-RK RTOS architecture with EVM extensions
parametric and programmatic run-time changes in the code nano-RK was redesigned and extended
with several new features (see Fig. 6):
• Runtime Parametric Control: Support for dynamically changing the sampling rates, runtime
task and peripheral activation/deactivation and runtime modication to the task utilization has
been added. These facilities are exposed and executed via the Common-EVM programmer inter-
face.
• Runtime Programmatic Control: As a part of EVM design dynamic task migration has been
implemented. This requires runtime schedulability analysis, capability checks to migrate a subset
of the task data, instructions, required libraries and task control block. Based on the Assignment
[], tasks may be activated or migrated between primary and backup nodes. Such facilities are
triggered by the primary-backup policy implemented on top of the assignment algorithm.
• Dynamic Memory Management: Both Best-t and First-t memory allocation methods are
supported. In addition a Garbage Collector (GC) has been designed to reclaim all memory seg-
ments owned by tasks that had been terminated. The GC is scheduled only when its execution
does not inuence execution of other tasks. To allow for dynamic memory (re)allocation dur-
ing code execution, a memory management mechanism was implemented with functions void ∗
nrk _malloc(uint16_t) and void nrk _f ree(void ∗).
3.2 Virtual Component Interpreter
The Virtual Component Interpreter provides the EVM programmer an interface to dene and exe-
cute all VTs. Every VT is dened as a word within the VCM library. When a new VT description
is received over the network, the VCM calls the interpreter which denes a new word using the
description le of a task and existing virtual component libraries. When a VT is activated, each
execution of VT is implemented as a scheduled activation of the interpreter with the VT's word as
an input. To allow preemptivity of the tasks, each call of the interpreter uses a VT-specic stack
and dedicated memory segments. In addition, since each VT is capable of dynamically allocating
memory during its execution in cases when there is no space in already dedicated memory, the
EVM's memory manager allocates a new block of xed size (currently 128B). Therefore, the inter-
preter is designed to use logical addresses in form (block_index, local_address_inside_block).
In order to be able to use the node's physical memory, the VCM supports address translation and
garbage collection for the interpreter.
Each node has a local copy of standard Common-EVM dictionary and Control-EVM dictio-
naries. If a new word needs to be included in the existing library, upon reception of a message
that contains the word, VCM initiates execution of the interpreter. The interpreter rst checks
the global word identier and revision number in order to discard obsolete versions. If a newer
version is at hand, the interpreter (re)denes the word in the dictionary. Since all VTs executions
9

10. (a) Virtual Task's Description Table (b) nanoRK TCB extension
Global ID Local ID
Processing Units needed CPU Reserve
Deployment constraints Type (Fixed/Flexible) Memory requirements Stack size
Number of nodes (n) 1st Block pointer
Node 1 ...
... nth Block pointer
Node n Number of Tx slots Pointer to slot
Memory requirements Stack size index array
Number of Blocks Number of Rx slots Pointer to slot
Network Dependencies Number of Tx slots index array
Number of Rx slots
EVM description le
Table 3: Local and Global task description
are performed using the interpreter, there is no need for any additional kind of provider-subscriber
relation as described in [11]. In addition, potential failure modes that may result from inconsistent
versions on each node are avoided using this simple approach. In the case when a when a node
receives an update for a word used to dene a task, it checks the revision number in the denition
le of the task in the case of a newer revision, after updating denition of the word, the interpreter
informs the VCM to update the VT's description le. Each local update, also updates the revision
number of the VT word, therefore keeping VT fully updated without requiring consensus across
the VC.
3.3 Virtual Tasks
Each VT is described using Virtual Task's De-
scription Table, comprised of a global and local
INTERPRETER
description of a VT (see Table 2(a)). Copies of
the Descriptor Table are stored on all members
User Defined
of the VC assigned within the VT. While this re-
- Common EVM
quirement for consistency currently results in an
- Control EVM
issue of scalability, the current project with Hon-
eywell Process Control (OneWireless) requires
high-speed control networks with less than 20
nodes and is hence within the practical limits Standardized
- Message format
of the current approach. Each task species its
Deployment constraints which describe specic ...
distribution requirements of a task as to whether
a task is Fixed, meaning that it has to run on
a specic subset of physical nodes, or Flexible, Figure 7: VCM/Interpreter relation
i.e. a task can be deployed on any physical node
in the VC. If a task is declared as Fixed, the number of the suitable nodes is specied along with
the node IDs. Each VT's global description has information about memory requirements, stack
size and number of xed size memory blocks (e.g.128B) that are used. In addition to the above
meta data, Network requirements in terms of number of active RT-Link transmit and receive slots
are specied at design time. The above descriptors are specied as within the VCM's Task Control
Block (TCB) for each task. The EVM TCB is an extension to the native nano-RK TCB[9].
Upon VT denition on a node, VCM creates Task Control Block (TCB) for the task. Beside
parts of native TCB (more details can be found in [6]) VCM initializes the TCB extension presented
in Table 2(b) which describes local resources used by the VT.
10

11. 3.4 Virtual Component Manager
The fundamental dierence between native nano-RK and the VCM is that the scope of nano-RK's
activities is local, node-specic and dened completely at design time, while the scope of the
VCM is the Virtual Component that may possibly span multiple physical nodes. The current set
of functionalities supported is:
1. Virtual Task handling:
1.1 VC state is maintained, keeping every node with a consistent mapping of VTs. The VCM
in each controller node within the VC periodically broadcasts its information about VT mapping
in order to keep consistency between all members of the VC. Currently a centralized consensus
protocol is used and a distributed consensus protocol is needed to scale operations.
1.2 Task migration that can be triggered as a result of a fault/failure procedure or by request of
either the VT or VCM. As a part of a task migration, task's VTDT is sent along with all memory
blocks used by the task. If the VT is already dened on a node (checked by exchange of hash
values), only task parameters are exchanged. In addition, due to physical limitations (network,
dierences between controller/actuator nodes, etc) before migrating VT to a particular node, net-
work and CPU schedulability analysis are performed for nodes that are potential candidates. If
analysis show that no node can execute the task correctly, an error message is returned.
1.4 VT activation: After a VT is dened, the VCM performs local CPU and network schedulabil-
ity analysis prior to task activation. This is done in order to ensure that task will not adversely
aect correct execution of previously dened VTs.
1.5 Control of tasks executed on other nodes: For all VTs in Backup mode, the VCM shadows the
execution of the VT in Primary mode. If a departure from the desired operation is observed (i.e.
low battery level, decreased received packet signal strength), Backup nodes may be assigned to
Primary mode based on policy.
2. Network Management:
2.1 Transparent radio interface: Using the message header which contains information about mes-
sage type, the VCM determines which task should be informed about message arrival. Messages
containing tasks and their parameter denitions are rst processed by a VCM, before the VCM
activates the Interpreter (for remote ports). All local task-specic messages are transferred to
their respective local ports.
2.2 Logical-to-physical address mapping: All communication between VTs is via the VCM. Since
a VT does not have information on which nodes other VTs are deployed, the VCM performs
logical-to-physical address mapping. In cases when both tasks are on the same node, the VCM
directly passes a message to a receiving task buer.
4 Implementation
To evaluate the EVMs performance in a real setting with multiple coordinated controller opera-
tions, we used a factory simulation module (FischerTechnik model factory). The factory consists
of 22 sensors and actuators that are to be controlled in a coordinated and timely manner. A block
of wood is passed through a conveyor, pushed by a rammer on to a turn table and operated upon
by up to three milling/cutting/pneumatic machines. The factory module was initially controlled
by wired programmable logic controllers (PLCs). We converted it to use wireless control with
FireFly nodes controlling all sensors and actuators via a set of electical relays. FireFly [12] is a
low-cost, low-power, platform based on the Atmel ATmega1281 8-bit micro-controller with 8KB
of RAM and 128KB of ROM along with a Chipcon CC2420 IEEE 802.15.4 standard-compliant
radio transceiver. FireFly nodes support tight global hardware-based time synchronization for
real-time TDMA-based communication with the RT-Link protocol [10]. The EVM also works
on TI MSP-EXP430F5xxx and MSP430FG4xxx architectures for more ecient execution of the
FORTH-like stack-based language on a Von Neumann architecture. We have demonstrated:
1. On-line capacity expansion when a node joins the VC
11

12. 2. Redistribution of VTs when adding/removing nodes
3. Planned VT migration triggered by the user
4. Unplanned VT migration as a result of node failure or communication link failure
5. Multiple coordinated work-ows
In a second experiment setup we considered multiple concurrent workows, where each work-
ow uses a subset of the actuators. Each new block type triggers the assignment of the associated
VT and migrates from one controller to the next as the block physically moves through the factory.
Therefore, with every new type of block a reconguration of the VC is performed. We have tested
the setup with a batch of 9 input blocks consisting of 3 dierent types. This is an example of the
logical benets of the EVM as it enables a more agile form of manufacturing. Details on both
experiments with videos can be seen in [13].
5 Related work
There have been several variants of virtual machines, such as Maté [4], Syclla [14] and SwissQM[15],
and exible operating systems, such as SOS[11], Contiki[16], Mantis[17], Pixie[18] and LiteOS[19],
for wireless sensor networks. The primary dierences that set EVM apart from prior work is that
it is centered on real-time operation of controllers and actuators. Within the design of EVM's
operating system, link protocol, programming abstractions and operation, timeliness is a rst-class
citizen and all operations are synchronized. The EVM does not have a single node-perspective
of mapping operations to one virtualized processor on a particular node but rather maintains
coordinated operation across a set of controllers within a virtual component. Furthermore, EVM
has been designed to operate in an autonomous manner and to be extensible at runtime.
One of the rst virtual machines (VM) for sensor networks was Maté. Maté implements
a simple, communication-centric VM built on top of the TinyOS [3]. It is designed as a high
level interface where code is written using limited instruction set, dened at design-time, and
executed with a FORTH-like interpreter. EVM utilizes a similar FORTH-like interpreter but is
extensible at runtime and allows for fully preemptive tasks. Syclla is a more conventional system
VM that allows code mobility by providing a virtualized processor abstraction on each node.
It uses a standard approach where one physical machine exposes interfaces to a single logical
machine. On the other hand, EVM uses several physical nodes and allows user to consider the
virtual component as a single logical entity. SOS operating systems for sensor nodes [11] was
designed to allow exible operation of nodes in a sensor network. As TinyOS, it has an event-
drive execution of all components, with a dierence that components can be installed/modied
during runtime. To allow this, SOS uses dynamically-loaded modules while kernel implements
messaging, dynamic memory, and module loading and unloading. EVM allows for dynamic task
evocation and runtime function loading but in addition has mechanism for interaction with other
nodes, which SOS lacks. In addition, SOS utilizes a signicant portion of the TinyOS code
and all tasks are run-to-completion. EVM is built on the nano-RK sensor RTOS and hence all
tasks are scheduled by xed priority scheduling and are fully preemptable. Finally, the Virtual
Node Layer [20] provides a programing abstraction for a virtual infrastructure where each virtual
node is identied with a particular region and it is emulated by one of the physical nodes in its
region. In last few years several dierent systems for macro-programming in WSN have been
developed[21, 22, 23]. Welsh et al. [21] have dened a set of abstractions representing local
communication between nodes in order to expose control over resource consumption along with
providing feedback on its performance. EVM along with data sharing incorporates performance
tracking of algorithm execution and automatically readjust available resource in order to maximize
system robustness. An extension of these ideas is used to develop the Regiment [23], a high-level
language based on the functional reactive programming. The Regiment uses two types of objects,
Signals used to represent sensor states and Regions, collection of signals, where membership in
Region can vary in time. In addition the authors dene operators for creating new regions along
with three basic operations (applying function to a signal and aggregating values in the region).
EVM is not a generic macroprogramming system as it focuses on closed-loop control and actuation
12

13. problems with native support for task migration and on-line task assignment. Kairos [22] allows
a programmer to describe code execution for each of the nodes in a network using a centralized
approach where details about code generation, remote data access and management along with
node interactions are hidden from the programmer. Kairos uses three constructs: remote data
access, iterating through 1-hop neighbors and addressing arbitrary nodes. Basic of Kairos is
a data sharing algorithm which assumes eventual consistency between shared memories on the
nodes. Although EVM provides consensus between shared memories store on dierent nodes with
a shared memory access for all nodes in the networks its list of features is not only limited to that.
References
[1] Frost and Sullivan, North American Sensor Markets, Technical Report A-761-32, 2004.
[2] Nielsen Research, Downtime Costs Auto Industry, March 2006.
[3] J. Hill and et al. System architecture directions for network sensors. ASPLOS, 2000.
[4] P. Levis and D. Culler. Mate: A tiny virtual machine for sensor networks. ACM ASPLOS-X, 2002.
[5] J. P. Hespanha, P. Naghshtabrizi, and Y. Xu. A survey of recent results in networked control systems.
Proc. of the IEEE, 2007.
[6] W. Zhang, M.S. Branicky, and S.M. Phillips. Stability of networked control systems. IEEE Control
Systems Magazine, 2001.
[7] M. Pajic and R. Mangharam. The Assignemnt Problem. http://mlab.seas.upenn.edu/evm, 2009.
[8] L. Brodie. Starting FORTH. 2nd Ed., Prentice-Hall. ISBN 0-13-843079-9, 1987.
[9] nano-rk sensor rtos. http://nanork.org.
[10] A. Rowe, R. Mangharam, and R. Rajkumar. RT-Link: A Time-Synchronized Link Protocol for
Energy-Constrained Multi-hop Wireless Networks. IEEE SECON, 2006.
[11] C.C. Han and et.al. SOS : A Dynamic Operating System for Sensor Nodes. ACM Mobisys, 2005.
[12] R. Mangharam and et.al. FireFly: A Cross-layer Platform for Real-time Embedded Wireless Net-
works. Real-Time System Journal, 2007.
[13] EVM website - http://mlab.seas.upenn.edu/evm, 2009.
[14] P. Marbell and L. Iftode. Scylla: A smart virtual machine for mobile embedded systems. In WMCSA,
2000.
[15] R Muller, G Alonso, and D Kossmann. A virtual machine for sensor networks. ACM EuroSys, 2007.
[16] A. Dunkels, B. Gronvall, and T. Voigt. Contiki - A Lightweight Operating System for Tiny Networked
Sensors. IEEE LCN, 2004.
[17] Shah Bhatti and et. al. MANTIS OS: an embedded multithreaded operating system for wireless
platforms. Mob. Netw. Appl., 2005.
[18] K. Lorincz and et. al. Resource aware programming in the Pixie OS. ACM SenSys, 2008.
[19] Q. Caoand and et. al. The LiteOS: Towards Unix-Like Abstractions for Wireless Sensor Networks.
IEEE IPSN, 2008.
[20] M. Brown, S. Gilbert, N. Lynch, C. Newport, T. Nolte, and M. Spindel. The Virtual Node Layer: A
Programming Abstraction for Wireless Sensor Networks. WWSNA, 2007.
[21] M. Welsh and G. Mainland. Programming Sensor Networks using Abstract Regions. NSDI, 2004.
[22] R. Gummadi et al. Macro-programming Wireless Sensor Networks Using Kairos. In Dist. Comp. in
Sensor Sys., 2005.
[23] R. Newton et al. The Regiment Macroprogramming System. ACM IPSN, 2007.
13