This document discusses using model checking techniques for safety critical systems at NASA. It begins by introducing model checking and how it can be used to verify that a program or model satisfies a given property. It then discusses challenges like the state explosion problem and presents compositional verification as a way to address this by breaking the verification task into checking smaller components. The document provides several examples of applying these techniques to real NASA systems like rovers and spacecraft software.

1. State Space Exploration
for NASA’s
Safety Critical Systems
Dimitra Giannakopoulou
NASA Ames Research Center

2. model checking
program / model
void add(Object o) {
model checker
property
buffer[head] = o;
head = (head+1)%size;
}
Object take() { always(ϕ or
…
tail=(tail+1)%size;
return buffer[tail];
}
NO + counterexample:
YES (property holds)
(provides a violating execution)

3. model checking vs. testing
testing model checking

4. is it a good idea?
Turing Award 2007
E. Clarke, A. Emerson, J. Sifakis
§ Targets subtle (concurrency) errors § Input language
§ Safety critical applications § Properties
§ Successful in hardware industry § Computational complexity

5. state-explosion problem

6. compositional
verification

7. collaborators
§ Corina Păsăreanu (CMU / NASA Ames)
§ and talented students / visitors:
Howard Barringer (Professor, Univ. of Manchester)
Colin Blundell (Upenn, now IBM)
Jamieson Cobleigh (UMass, now MathWorks)
Michael Emmi (UCLA)
Mihaela Gheorgiu (Univ. of Toronto, now NASA JPL)
Chang-Seo Park (UC Berkeley)
Suzette Person (Univ. of Nebraska, now NASA Langley)
Rishabh Singh (MIT)

8. compositional verification
does system made up of M1 and M2 satisfy property P?
" check P on entire system: too many states!
M1 " use system s natural decomposition into
components to break-up verification task
satisfies P?
" check components in isolation:
A
M2
does M1 satisfy P?

9. §
when we try to pick out anything by itself, we find
it hitched to everything else in the universe
John Muir

10. assume-guarantee reasoning
introduces assumptions / reasons about triples:
〈A〉 M 〈P〉 is true if whenever M is part of a
M1 system that satisfies A, then the system must
also guarantee P
satisfies P?
A simplest assume-guarantee rule (ASYM):
M2 1. 〈A〉 M1 〈P〉 discharge the
2. 〈true〉 M2 〈A〉 assumption
〈true〉 M1 || M2 〈P〉

11. what is a good assumption?
can assumptions be generated?

12. examples of assumptions
§ will not invoke close on a file if open has not previously
been invoked
§ accesses to shared variable X must be protected by lock L
§ (rover executive) whenever thread A reads variable V , no
other thread can read V before thread A clears it first
§ (spacecraft flight phases) a docking maneuver can only be
invoked if the launch abort system has previously been
jettisoned from the spacecraft

13. formalisms
§ components modeled as finite state machines (FSM)
– FSMs assembled with parallel composition operator ||
• synchronizes shared actions, interleaves remaining actions
§ a safety property P is a FSM
– P describes all legal behaviors in terms of its alphabet
– Perr – complement of P
• determinize & complete P with an error state;
• bad behaviors lead to error
– component M satisfies P iff error state unreachable in (M || Perr)
§ assume-guarantee reasoning
– assumptions and guarantees are FSMs
– 〈A〉 M 〈P〉 holds iff error state unreachable in (A || M || Perr)

14. example
require in and out to alternate (property Order)
Input
in send out
Input Output in send
ack
Ordererr ack
in
Output
send out
out
out in
ack

15. parallel composition
Input Output
in send send out
||
ack ack

16. property satisfaction
Ordererr
in
Input
in send 0 1
0 1 2 || out
out in
ack
crex. 1: (I0, O0) out (I0, Oerror)
crex. 2: (I0, O0) in (I1, O1) send (I2, O1) out (I2, O0) out (I2, Oerror)

17. assume-guarantee reasoning
Input
in send
Ordererr
0 1 2 in
ack 0 1
|| out
Assumption
out in
send
ack 0 1
out
send
crex 1: (I0, A0, O0) out X
crex 2: (I0, A0, O0) in (I1, A0, O1) send (I2, A1, O1) out (I2, A0, O0) out X

18. the weakest assumption [ASE 2002]
P
M WA
§ given component M, property P, and the interface ∑ of M
with its environment, generate the weakest environment
assumption WA such that: 〈WA〉 M 〈P〉 holds
§ weakest means that for all environments E:
〈true〉 M || E 〈P〉 IFF 〈true〉 E 〈WA〉

19. learning assumptions (TACAS 2003)
iterative solution +
intermediate results
L* learns unknown regular language
U (over alphabet Σ) and produces
minimal DFA A such that L(A) = U
(L* originally proposed by Angluin)

20. weakest assumption in AG reasoning
1. 〈A〉 M1 〈P〉
2. 〈true〉 M2 〈A〉 weakest assumption makes
rule complete
〈true〉 M1 || M2 〈P〉
for all E, 〈true〉 M || E 〈P〉 IFF 〈true〉 E 〈WA〉
〈true〉 M1 || M2 〈P〉 IFF 〈true〉 M2 〈WA〉
in other words:
〈true〉 M2 〈WA〉 holds implies 〈true〉 M1 || M2 〈P〉 holds
〈true〉 M2 〈WA〉 not holds implies 〈true〉 M1 || M2 〈P〉 not holds

21. L* learner the oracle
queries:
should word w be included in L(A)?
yes / no
conjectures:
here is an A – is L(A) = U?
yes!
no: word w should (not) be in L(A)

22. oracle for WA in assume-guarantee reasoning
true / false 1. 〈A〉 M1 〈P〉
query: string s 2. 〈true〉 M2 〈A〉
〈s〉 M1 〈P〉
(simulate s on M1 || Perr) 〈true〉 M1 || M2 〈P〉
c ↑αA false+crex c
L* conjecture: Ai
〈Ai〉 M1 〈P〉 (model check)
true
(model check)
〈true〉 M2 〈Ai〉 P holds in M1 || M2
false+crex c
query c ↑αA P violated in M1 || M2
c ↑αA false
true
〈WA〉 M1 〈P〉 holds
〈true〉 M2 〈WA〉 holds implies 〈true〉 M1 || M2 〈P〉 holds
〈true〉 M2 〈WA〉 does not hold implies 〈true〉 M1 || M2 〈P〉 does not hold

23. characteristics
assumptions conjectured by L* are not comparable semantically
" terminates with minimal automaton A for U
" generates DFA candidates Ai: |A1| < | A2| < … < |A|
" produces at most n candidates, where n = |A|
" # queries: O(kn2 + n logm),
– m is size of largest counterexample, k is size of alphabet
" for assume-guarantee reasoning, may terminate early with a
smaller assumption than the weakest

24. example
Input Ordererr Output
in
in send send out
out out in
ack ack
Queries A 1:
Oracle 1: Counterexample:
ack
send 〈A1〉 Input 〈Order〉 c = 〈in,send,ack,in〉
A 2: send
Queries Oracle 1:
Return to L*: ack
〈A2〉 Input 〈Order〉
c↑ Σ = 〈send,ack〉
True
out, send
Oracle 2:
property Order holds
〈 true〉 Output 〈A2〉
True on Input || Output
weakest assumption
has 4 states

25. more than 2 components…
§ extension of basic rule ASYM [TACAS 2003, FMSD 2009]
§ symmetric / circular rules [SAVCBS 2003, FMSD 2009]

26. recursive application of ASYM
1. 〈A2〉 M2 〈A1〉
2. 〈true〉 M3 〈A2〉
1. 〈A1〉 M1 〈P〉
2. 〈true〉 M2 || M3 〈A1〉 3. 〈true〉 M2 || M3 〈A1〉
3. 〈true〉 M1 || (M2 || M3) 〈P〉

27. symmetric learning framework [SAVCBS05]
refine A1 refine A2
L* L*
refine A1 A1 A2 refine A2
〈A1〉 M1 〈P〉 〈A2〉 M2 〈P〉
false false
true true
L(coA1 || coA2) ⊆ L(P)
true P holds in M1||M2
false
counterexample P violated in M1||M2
analysis

28. example 1: Mars Exploration Rover
§ tools: LTSA, SPIN
§ model derived from JPL s Mars Exploration
Rover (MER) Resource Arbiter
– local management of resource contention
between resource consumers (e.g. science
instruments, communication systems)
– consists of k user threads and one server
thread (arbiter)
Resource Arbiter
§ checked mutual exclusion between
resources (e.g. driving while capturing a U5
camera image are incompatible) U4
§ compositional verification scaled to >5 U3 request, cancel
users vs. monolithic verification ran out U2
grant, deny
ARB
of memory [SPIN 06] U1 rescind

29. autonomous rendezvous & docking
§ tool: LTSA
§ consists of control software, state estimator, and 4 types of sensors
§ input provided as UML state-charts, properties of type:
– you need at least two operational sensors to proceed to next mode
§ 3 bugs detected
§ scaling achieved with compositional verification:
– monolithic verification runs out of memory after > 13M states
– compositional verification terminates successfully in secs. Largest state-space
explored is less than 60K states, as opposed to > 13M.
star planet
tracker
docking control orbital
inertial
sensor software state
navigation
GPS

30. example 3: K9 Rover Executive
K9 Rover
§ tools: LTSA, JavaPathfinder
§ model of NASA Ames K9 Rover Executive
– executes flexible plans for autonomy
– consists of Executive thread and ExecCondChecker
thread for monitoring state conditions
– checked for specific shared variable: if Executive reads
its value, ExecCondChecker should not read the
variable before the Executive clears it
§ generated assumption of 6 states for model in LTSA [TACAS 2003]
§ used generated assumption to check 8K lines of JAVA code translated from 10K
lines of C++ code using the JavaPathfinder model checker [ICSE 2004]
§ reduced memory used by JavaPathfinder > 3 times
§ used generated assumption to perform assume-guarantee testing of C++ code
using Eagle runtime monitoring framework [SAVCBS 2005, IET Software 2009]

31. interface
generation

32. component interfaces
§ beyond syntactic interfaces (open file before close)
§ document implicit assumptions
§ safe: accept NO illegal sequence of calls
§ permissive: accept ALL legal sequences of calls
§ we use learning to generate interfaces [FASE 2009]
− conjectured interfaces must be safe and permissive
− queries and safety checked as in compositional scheme
− permissiveness checked with heuristics

33. JavaPathfinder
UML statecharts
assume-guarantee reasoning
interface generation / discharge
jpf-cv
http://babelfish.arc.nasa.gov/trac/jpf

34. example: crew exploration vehicle
§ tool: JavaPathfinder
§ UML statechart model of the
Ascent and EarthOrbit flight phases
of a spacecraft
§ properties:
– “An event lsamRendezvous, which
represents a docking maneuver with
another spacecraft, fails if the LAS
(launch abort system) is still
attached to the spacecraft”
– “Event tliBurn (trans-lunar interface
burn takes spacecraft out of the
earth orbit and gets it into transition
to the moon) can only be invoked if
EDS (Earth Departure Stage) rocket
is available”

35. results
Assumption 1: Assumption 2:
lasJetisson lsamRendezvous
lasJetisson tliBurn
lsamRendezvous lsamRendezvous
generated interface assumptions encode Flight Rules in terms of events

36. infinite-state components…

37. infinite components [CAV 2010]
§ use predicate abstraction (e.g., x ≥ 0, x < 0)
§ generate may and must abstraction
Lillegal(Cmay)
Llegal(Cmust)
must transition
L
illegal(C)
L
legal(C)
Lillegal(Cmust)
Llegal(Cmay)
may transition
an interface safe w.r.t. Cmay and permissive w.r.t. Cmust
is safe and permissive w.r.t. concrete component C

38. Query(σ, C)
1. if checkSafe(σ,Cmust) != null Lillegal(Cmay)
Llegal(Cmust)
2. return no
L
illegal(C)
L
legal(C)
3. cex = checkSafe(σ,Cmay)
4. if cex == null
Lillegal(Cmust)
Llegal(Cmay)
5. return yes
6. Preds = Preds U Refine(cex)
7. Query(σ, C)
if concrete component is deterministic, so is the must abstraction…
ARMC model checker: Java2SDK library classes, OpenSSL, NASA CEV model

39. summary
§ automating compositional verification was a breakthrough
§ our techniques are generic
§ not a panacea…
– perform well when alphabets & assumptions are small
§ design for compositional verification
§ discovering good system decompositions
§ timed & probabilistic systems, non functional properties
§ multi core / parallelization?

40. http://babelfish.arc.nasa.gov/trac/jpf
model
checking