Slides of my talk on "Open Source Verification under a Cloud " at OpenCert in Pisa, Italy, September of 2010. The paper appeared in Electronic Communications of the European Association of Software Science and Technology, vol. 33, and a preprint is at http://www.academia.edu/1413629/Open_Source_Verification_under_a_Cloud .

1. Open Source Verification
Under a Cloud
Peter Breuer Simon Pickin
Dept. Comp. Sci. Dpto. Ing. Tel.
Univ. Birmingham U. Carlos III de Madrid
UK Spain

2. What we’ve done
Started with monolithic analysis software
Restructured it into
database plus verification problem server
ad hoc network of remote verification solver clients
Thrown it at verification problems tackled previously
Checking Linux kernel code for SMP locking errors
just over 1,000,000 lines of C code/assembler
look for possible take of spinlock in dangerous context
possible double-take of spinlock before release, etc.
took about 9,000,000s of system time
Outcome is intrinsically 50× as slow as original
takes 100 clients to go 2× as fast
but it scales well!

3. Why we’ve done it
For a future vision
formal methods specialists contribute standard analyses
developers upload code for analysis to cloud
unskilled open source supporters help favourite project
run client solvers at home on their spare CPU cycles
specialists report regressions and new errors
requires humans to eliminate false positives
Hopefully we get a positive reinforcement loop
more developers develop formal methods skills
more formal methods people understand coding problems
more unskilled contributers develop skills
Perhaps to run faster!
500 clients analyse 1,000,000 LOC in 3h

4. What our Ad Hoc Volunteer Cloud of Solvers Looked Like
Internet
3x<1+y?
p/q=>q−>p?
query
store
...
0
0
1
1
Firewall
High
Disk
Speed
DB server

5. How the Calculation is Organised
P Parse produces syntax tree T
1,000,000 LOC produces 10,000,000 syntax tree nodes
A Decorate T with post-conditions post to get T post
E.g. ‘x ≤ 1’ where x counts locks taken
H Add evaluation [post ∧ d] for defect d to get T post
eval
E.g. x ≤ 1 ∧
x ≥ 1 p = lock()
false otherwise
L Where there’s a nonzero evaluation, flag the point p
X Certify intermediates T post
eval and defect list L
customized approximating logic
Symbolic Approximation

6. Rearranging the Calculation for a Cloud
Parsing P is currently done monolithically first of all
Logical analysis is done piecewise at each syntax node
A = ◦
p
Ap
Ditto checking H = ◦
p
Hp
Organised as work-units of analysis and checking
A ◦ H = ◦
p
(Ap ◦ Hp)
Subject to dependencies induced by syntax p = P
i
(pi )

7. Certification
Store the evidence (intermediate and final results) T post
eval
Store parse, logic configuration, defect definitions
Provide list of defects L and certificate X
Certificate contains signatures of all items above
?
Idea is that a doubter can ask to see the data
Any part of the computation can then be repeated . . .
. . . to confirm or refute what the certificate claims
!

8. Experimental Data
746,844 function definitions in >1,000,000 LOC
many turned out to be ‘static inline’ duplications
Reduced to 78,619 syntactically different definitions
Clients each initially given 10m to analyse 1 function
Abort after 10m and try another
only 373 tasks needed longer than 10m
Time-limit on task raised to 15m, etc.
129 functions needed longer than 1h
Were split up syntax-wise and checkpointed every 5m
24 functions remained unanalyzed at experiment end
complexity explosion in logic accounts for most

9. Duplicated Function Definitions are a Problem!
Top-level definitions with multiple instances ( xy = 746844)
1
10
100
1000
10000
100000
1 10 100 1000
#uniquedefs
#instances

10. Time taken per analysis task
1
10
100
1 10 100 1000 10000 100000
#tasks
time in seconds

11. Time taken per analysis task (cumulative count)
0
20
40
60
80
100
1 10 100 1000 10000 100000
%tasks
time in seconds

12. Percentage of total time taken per analysis task
(cumulative)
0
20
40
60
80
100
1 10 100 1000 10000 100000
%time
time taken per task in seconds

13. Practical Difficulties
Populating DB remotely was going to take a month!
parsing/loading locally took about 24h
Remote DB transactions can take seconds each
beaten by cacheing (95% hits) of DB queries on clients
solver attempts 150-500 queries/s (90% reads)
5-10 queries/s escape caches to the net
Solver CPU loading is only a few %, peaking occasionally
DB server limits transaction rate
limit at 100 queries/s for 3GB RAM 1.8GHz 64bit Athlon
needed to up RAM to DB size to avoid disk I/O limits
CPU loaded to about 15% by each active thread
Client/net breakdowns mean only 25-50% operating level

14. Summary
Prototype verification cloud software
Based on symbolic approximation
customized approximating logic
Computation handled incrementally and piecewise
Intermediate results retained for accountability
any part repeated/duplicated if challenged
Produces certificate and a list of defects
Experiment on 1,000,000 LOCC (Linux kernel)
9,000,000s of (normalized 1GHz CPU) system time
< 24h for 100 clients
< 6h for 500 clients
provided can find DB servers to handle 2500 queries/s!
(fortunately queries are strongly localized)