The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Comparing the Effectiveness of Reasoning Formalisms for Partial Models
1. Comparing the
Effectiveness
of Reasoning
Formalisms for
Partial Models
Saadatpanah,
Famelis,
Gorzny, Comparing the Effectiveness of Reasoning
Robinson,
Chechik, Salay Formalisms for Partial Models
Introduction
Designer
Uncertainty Pooya Saadatpanah, Michalis Famelis, Jan Gorzny,
Modeling Nathan Robinson, Marsha Chechik, Rick Salay
Uncertainty
Property
Checking University of Toronto
Process
Verification
Technologies September 30th, 2012
Experiments
MoDeVVa’12
Results
Conclusion
1 / 32
2. Comparing the
Effectiveness
of Reasoning
Formalisms for
Uncertainty in software modeling
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
• Uncertainty: pervasive in MDE
Introduction
Designer
Uncertainty • Models with uncertainty:
Modeling • Represent choice among many possibilities
Uncertainty
• Can be refined to many different classical models
Property
Checking
Process
Verification
Technologies • Our goal:
Experiments
Handle models with uncertainty in MDE
Results without having to remove it [MoDeVVa’11].
Conclusion
2 / 32
3. Comparing the
Effectiveness
of Reasoning
Formalisms for
Existing Work
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
3 / 32
4. Comparing the
Effectiveness
of Reasoning
Formalisms for
In This Paper
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
4 / 32
5. Comparing the
Effectiveness
of Reasoning
Formalisms for
Outline
Partial Models
Saadatpanah,
Famelis, Introduction
Gorzny,
Robinson,
Chechik, Salay
What is uncertainty?
Introduction
Designer How to represent uncertainty with partial models (MAVO).
Uncertainty
Modeling
Uncertainty Process for checking properties
Property
Checking
Process Alternative verification technologies
Verification
Technologies
Experiments
Experiments
Results
Results
Conclusion
Conclusion
5 / 32
6. Comparing the
Effectiveness
of Reasoning
Formalisms for
Introduction to Uncertainty
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay What the designer knows.
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
6 / 32
7. Comparing the
Effectiveness
of Reasoning
Formalisms for
Introduction to Uncertainty
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay What the designer does not know.
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
6 / 32
8. Comparing the
Effectiveness
of Reasoning
Formalisms for
Introduction to Uncertainty
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay What the designer does not know.
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
6 / 32
9. Comparing the
Effectiveness
of Reasoning
Formalisms for
Introduction to Uncertainty
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
What the designer does not know.
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
6 / 32
10. Comparing the
Effectiveness
of Reasoning
Formalisms for
Uncertainty: a Set of Possible
Partial Models
Saadatpanah,
Refinements.
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
If we remove all uncertainty, we have a concrete refinement.
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
7 / 32
11. Comparing the
Effectiveness
of Reasoning
Formalisms for
Modeling Uncertainty with Partial
Partial Models
Saadatpanah,
Models
Famelis,
Gorzny,
Robinson, Explicating uncertainty in a partial model.
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies In a refinement, a May element is optional.
Experiments In a refinement, a Set element can be multiplied to many copies.
Results In a refinement, a Variable element can be unified with some other.
Conclusion In a refinement, an Open world model can be expanded with some
other elements.
Partial models: Syntactic annotations of the points of uncertainty.
8 / 32
12. Comparing the
Effectiveness
of Reasoning
Formalisms for
Modeling Uncertainty with Partial
Partial Models
Saadatpanah,
Models
Famelis,
Gorzny,
Robinson, Explicating uncertainty in a partial model.
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies In a refinement, a May element is optional.
Experiments In a refinement, a Set element can be multiplied to many copies.
Results In a refinement, a Variable element can be unified with some other.
Conclusion In a refinement, an Open world model can be expanded with some
other elements.
Partial models: Syntactic annotations of the points of uncertainty.
8 / 32
13. Comparing the
Effectiveness
of Reasoning
Formalisms for
Modeling Uncertainty with Partial
Partial Models
Saadatpanah,
Models
Famelis,
Gorzny,
Robinson, Explicating uncertainty in a partial model.
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies In a refinement, a May element is optional.
Experiments In a refinement, a Set element can be multiplied to many copies.
Results In a refinement, a Variable element can be unified with some other.
Conclusion In a refinement, an Open world model can be expanded with some
other elements.
Partial models: Syntactic annotations of the points of uncertainty.
8 / 32
14. Comparing the
Effectiveness
of Reasoning
Formalisms for
Modeling Uncertainty with Partial
Partial Models
Saadatpanah,
Models
Famelis,
Gorzny,
Robinson, Explicating uncertainty in a partial model.
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies In a refinement, a May element is optional.
Experiments In a refinement, a Set element can be multiplied to many copies.
Results In a refinement, a Variable element can be unified with some other.
Conclusion In a refinement, an Open world model can be expanded with some
other elements.
Partial models: Syntactic annotations of the points of uncertainty.
8 / 32
15. Comparing the
Effectiveness
of Reasoning
Formalisms for
Modeling Uncertainty with Partial
Partial Models
Saadatpanah,
Models
Famelis,
Gorzny,
Robinson, Explicating uncertainty in a partial model.
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies In a refinement, a May element is optional.
Experiments
In a refinement, a Set element can be multiplied to many copies.
Results
In a refinement, a Variable element can be unified with some other.
Conclusion
In a refinement, an Open world model can be expanded with some
other elements.
Partial models: Syntactic annotations of the points of uncertainty.
8 / 32
16. Comparing the
Effectiveness
of Reasoning
Formalisms for
Outline
Partial Models
Saadatpanah,
Famelis, Introduction
Gorzny,
Robinson,
Chechik, Salay
What is uncertainty?
Introduction
Designer How to represent uncertainty with partial models (MAVO).
Uncertainty
Modeling
Uncertainty Process for checking properties
Property
Checking
Process Alternative verification technologies
Verification
Technologies
Experiments
Experiments
Results
Results
Conclusion
Conclusion
9 / 32
17. Comparing the
Effectiveness
of Reasoning
Formalisms for
Property Checking in Partial
Partial Models
Saadatpanah,
Models
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
10 / 32
18. Comparing the
Effectiveness
of Reasoning
Formalisms for
Property Checking in Partial
Partial Models
Saadatpanah,
Models
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
10 / 32
19. Comparing the
Effectiveness
of Reasoning
Formalisms for
Property Checking in Partial
Partial Models
Saadatpanah,
Models
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
10 / 32
20. Comparing the
Effectiveness
of Reasoning
Formalisms for
Property Checking in Partial
Partial Models
Saadatpanah,
Models
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
10 / 32
21. Comparing the
Effectiveness
of Reasoning
Formalisms for
Verification Technologies I
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
• Alloy
Chechik, Salay
• Lightweight formal methods
Introduction • Model finder based on SAT
Designer
• First order logic specifications expressed in relational logic
Uncertainty • Grounded to CNF representation
Modeling • Finds counter examples
Uncertainty
Property
Checking
Process
• Constraint Satisfaction Problem (Minizinc/Flatzinc)
Verification
Technologies • Assign value to variables to satisfy all constraints
Experiments • Constraint modeling language
Results • Easily translatable to the form required by other CSP
Conclusion solvers
11 / 32
22. Comparing the
Effectiveness
of Reasoning
Formalisms for
Verification Technologies II
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay • Satisfiability Modulo Theory (Z3)
• Constraint satisfaction search with richer theories
Introduction
• Theorem prover
Designer
Uncertainty • Check the satisfiability logical formulas
Modeling
Uncertainty
Property
Checking
Process • Answer Set Programming (Clingo=Gringo+Clasp)
Verification • Answer set solvers
Technologies
• Conflict-driven nogood learning
Experiments
• Normal logic programs
Results
Conclusion
12 / 32
23. Comparing the
Effectiveness
of Reasoning
Formalisms for
Outline
Partial Models
Saadatpanah,
Famelis, Introduction
Gorzny,
Robinson,
Chechik, Salay
What is uncertainty?
Introduction
Designer How to represent uncertainty with partial models (MAVO).
Uncertainty
Modeling
Uncertainty Process for checking properties
Property
Checking
Process Alternative verification technologies
Verification
Technologies
Experiments
Experiments
Results
Results
Conclusion
Conclusion
13 / 32
24. Comparing the
Effectiveness
of Reasoning
Formalisms for
Experimental Setup
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
14 / 32
25. Comparing the
Effectiveness
of Reasoning
Formalisms for
Experimental Setup
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
15 / 32
26. Comparing the
Effectiveness
of Reasoning
Formalisms for
Random Input Generation
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson, • Meta-model : directed graphs
Chechik, Salay
• Minimal meta-model
Introduction • A few constraints
Designer • Most difficult one for solvers
Uncertainty
Modeling
Uncertainty • Randomly decorated with MAVO annotations.
Property
Checking
Process • Parameters are based on real case studies.
Verification • Graph density
Technologies
• Percentage of MAVO annotated elements
Experiments
• Percentages of M-, S- and V-annotated elements
Results
Conclusion
• 3 Model Size : Small, Medium, Large, X-Large
16 / 32
27. Comparing the
Effectiveness
of Reasoning
Formalisms for
Experimental Setup
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
17 / 32
28. Comparing the
Effectiveness
of Reasoning
Formalisms for
Relational Encoding I
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay • Relational Algebra used in DBMSs
Introduction
Designer • Directly translatable into the different formalisms
Uncertainty
Modeling
Uncertainty
• Intermediate representation
Property
Checking • FOL semantics of MAVO
Process
• Reasoning formalisms
Verification
Technologies
Experiments
• Meaningful comparison
Results
• Most efficient encoding in each formalism : impossible!
Conclusion
• Solution : common encoding
18 / 32
29. Comparing the
Effectiveness
of Reasoning
Formalisms for
Relational Encoding II
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
(Very) high level overview.
Introduction
• The metamodel encoded
Designer as a schema.
Uncertainty
• Partial model FOL
Modeling
Uncertainty semantics encoded as
Property
Checking
constraints over the
Process schema.
Verification
Technologies • Creating a concrete
Experiments refinement populates the
Results database.
Conclusion
19 / 32
30. Comparing the
Effectiveness
of Reasoning
Formalisms for
Experimental Setup
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
20 / 32
31. Comparing the
Effectiveness
of Reasoning
Formalisms for
Translation To Formalisms
Partial Models
• Alloy
Saadatpanah,
Famelis, • Relations : Alloy signatures
Gorzny,
Robinson,
• Instances : Atoms
Chechik, Salay • MAVO constraints : quantified predicates over signatures
• Bound is required
Introduction
Designer
• CSP
Uncertainty • Relations : Finite set of Integers
Modeling • Instances : Integers
Uncertainty
• MAVO constraints : cardinality and intersection of sets
Property
Checking • Bound is required
Process
• SMT
Verification • Relations : Uninterpreted boolean functions
Technologies
• Instances : Abstract values
Experiments
• MAVO constraints : Quantified logic over truth table of
Results
functions
Conclusion
• ASP
• Program rules for both instances and relations
• Bound is required
21 / 32
32. Comparing the
Effectiveness
of Reasoning
Formalisms for
Experimental Setup
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
22 / 32
33. Comparing the
Effectiveness
of Reasoning
Formalisms for
Properties Checked
Partial Models
Saadatpanah,
Famelis,
Gorzny, • Inspired by real metamodel constraints.
Robinson,
Chechik, Salay
• No transitive closure, since it is expensive to check.
Introduction
Designer
Uncertainty
P1: There exists a node with a self-loop.
Modeling
Uncertainty P2: All nodes have outgoing edges.
Property
Checking P3: All nodes have outgoing or incoming edges.
Process
Verification P4: For all pairs of nodes n1 , n2 there exists at most one edge e
Technologies e
such that n1 → n2
Experiments
Results P5: For every pair of nodes n1 , n2 , n1 = n2 there exist two edges
e1 e2
Conclusion e1 , e2 such that n1 → n2 and n2 → n1 .
23 / 32
34. Comparing the
Effectiveness
of Reasoning
Formalisms for
Experimental Setup
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
24 / 32
35. Comparing the
Effectiveness
of Reasoning
Formalisms for
Experimental Parameters
Partial Models
Saadatpanah, • Bound (2, 4, 6)
Famelis,
Gorzny, • Solvers (except SMT) use bound for grounding expressions
Robinson,
Chechik, Salay to atoms.
• How many times can an ’S’-annotated element be
Introduction
replicated in a refinement.
Designer
Uncertainty
• Repetitions
Modeling
Uncertainty • 5 times
Property
Checking
• Cutoff time/memory
Process
• less than 10 minutes
Verification
Technologies
• less than 5 gigabyte
Experiments
• otherwise timeout
Results • What we measure
Conclusion
• How long does it take for each solver to return an answer
• A score out of 1200
• if timeout : zero!
25 / 32
36. Comparing the
Effectiveness
of Reasoning
Formalisms for
Findings
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
26 / 32
37. Comparing the
Effectiveness
of Reasoning
Formalisms for
Findings
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
26 / 32
38. Comparing the
Effectiveness
of Reasoning
Formalisms for
Findings
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
26 / 32
39. Comparing the
Effectiveness
of Reasoning
Formalisms for
SMT the champion?
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay • Unaffected by bounds!
Introduction • Works at higher level of abstraction
Designer (Theory of uninterpreted functions.)
Uncertainty
Modeling
• Unaffected by expensive grounding phase.
Uncertainty
Property
Checking
Process
Verification Caveat:
Technologies
Experiments
• SMT can theoretically return “I don’t know”.
Results • (However: we didn’t observe that.’)
Conclusion
27 / 32
40. Comparing the
Effectiveness
of Reasoning
Formalisms for
Threats to Validity
Partial Models
Saadatpanah,
Famelis,
Gorzny, Randomly generated graphs.
Robinson,
Chechik, Salay • Tuned the generator with realistic graph properties.
Introduction • Values of graph properties from case studies.
Designer
Uncertainty
Modeling
Uncertainty Fairness of comparisons.
Property • Common encoding to level the field.
Checking
Process
Verification
Technologies
Choice of specific reasoning engines
Experiments
Results
• When available: winners of competitions.
Conclusion • CSP: most convenient input language.
28 / 32
41. Comparing the
Effectiveness
of Reasoning
Formalisms for
Summary
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Research Question
What is the most efficient formalism for verifying models
Introduction
Designer
containing uncertainty?
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification
Technologies
Experiments
Results
Conclusion
29 / 32
42. Comparing the
Effectiveness
of Reasoning
Formalisms for
Summary
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction
Designer
Uncertainty
Modeling
Uncertainty
Property
Checking
Process
Verification Other results:
Technologies
• Framework for running experiments (now full tool support).
Experiments
Results
• Random generator for arbitrary type graphs.
Conclusion • Relational encoding.
• Translations of the RA encoding to different formalisms.
29 / 32
43. Comparing the
Effectiveness
of Reasoning
Formalisms for
Future Work
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
• Implement symmetry breaking in the SMT encoding.
Introduction
Designer
Uncertainty
Modeling
Uncertainty • Experiment with properties that require transitive closure.
Property
Checking
Process
Verification
Technologies
Experiments • Experiment with partial models containing OW.
Results
Conclusion
30 / 32
45. Comparing the
Effectiveness
of Reasoning
Formalisms for
Bibliography I
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
Introduction Famelis, M., Ben-David, S., Chechik, M., and Salay, R. (2011).
“Partial Models: A Position Paper”.
Designer
In Proceedings of MoDeVVa’11, pages 1–6.
Uncertainty
Famelis, M., Chechik, M., and Salay, R. (2012).
Modeling
Uncertainty “Partial Models: Towards Modeling and Reasoning with Uncertainty”.
In Proceedings of ICSE’12.
Property
Checking Salay, R., Famelis, M., and Chechik, M. (2012).
Process “Language Independent Refinement using Partial Modeling”.
In Proceedings of FASE’12.
Verification
Technologies
Experiments
Results
Conclusion
32 / 32