This presentation describes the new BayesiaLab Knowledge Elicitation Environment. This environment allows reducing biases (cognitive, group and facilitator), and allows to greatly improve the traceability of the brainstorming session.

1. Plan
Modeling by
Brainstorming
BayesiaLab’s Knowledge Elicitation
BAYESIALAB 5.0
Knowledge Environment
Elicitation
Environment
An innovative Brainstorming Tool
Dr. Lionel JOUFFE
May 2010
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
1

2. Plan
Modeling by
Brainstorming
MODELING BY BRAINSTORMING
BAYESIALAB 5.0
Knowledge MODELING BY BRAINSTORMING
Elicitation
Environment
All models are wrong; the practical question is how wrong do
they have to be to not be useful (Box&Draper 87)
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
2

3. Designing a Model for Decision Support
Every Company is faced to complex decisions that need to
be rationally supported
Plan
Sometime, there are too few data available, or no data at all,
to allow using data mining and data analysis technics to
automatically build a Decision Support System
Modeling by
Brainstorming Experts have gathered invaluable Tacit Knowledge through
BAYESIALAB 5.0
their experience
Knowledge
Elicitation We need to Convert this Tacit Knowledge into Explicit
Environment Knowledge and use it to build a model
We want actionable models to allow What-if scenarios
(simulation and/or diagnosis), drivers analysis, ...
Bayesian Belief Networks (BBNs) are ideal models for such
problematics: their graphical representation allows a manual
design by using expert knowledge, and their probabilistic
engines offer powerful simulation capabilities
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
3

4. BBNs are made of Two Distinct Parts
Structure
Directed Acyclic Graph (DAG), i.e. no directed loop
Plan Nodes represent the variables
Each node has a set of exclusive states (e.g.: Young, Adult,
Aged)
Modeling by
Brainstorming
Arcs represent the direct probabilistic influences between the
BAYESIALAB 5.0 variables (possibly causal)
Knowledge
Elicitation
Environment
Parameters
Probability distributions are associated to each node, usually by using
tables
CONDITIONAL PROBABILITY
DISTRIBUTION
MARGINAL A smoker has a 60% of risk of suffering
PROBABILITY DISTRIBUTION from a Bronchitis, whereas the risk of
We consider a population made a non smoker is 30% only
of 40% of Adults
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
4

5. BBNs are Powerful Inference Engines
We get some evidence on the states of a subset of variables:
Hard positive and negative evidence, Likelihoods, Probability
distributions, Mean values
Plan
We take these findings into account in a rigorous way to update our
belief on the states of all the other variables
Modeling by
Brainstorming Probability distributions on their values
BAYESIALAB 5.0
Knowledge Multi-Directional Inference (Simulation and/or Diagnosis)
Elicitation
Environment
Prior Distribution Posterior Distribution
The evidence on Smoker
(a new probability distribution)
allows to update the probability
distribution of Age (Diagnosis)
and Bronchitis (Simulation)
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
5

6. BBN Modeling by Brainstorming
Plan Clear definition of the BBN’s objective(s) (e.g.: Improvement
of the Product/Service Quality, improvement of the Purchase
Intent, improvement of the Company’s performance, ...)
Modeling by
Brainstorming Identification of the conceptual dimensions that are linked to
those objectives (e.g.: Human resources, Management,
BAYESIALAB 5.0 Production, Marketing, ...)
Knowledge
Elicitation
Environment Definition of the group of experts that will fully cover all the
dimensions (and the different geographical zones), with a small
redundancy to allow fruitful debates
Brain Storming Sessions with this group of Experts to
manually build the BBN
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
6

7. The Structure
The Directed Acyclic Graph
The structure elicitation is probably the simplest task of the
Brainstorming session
For each identified conceptual dimension
Plan
Definition of the main variables
Modeling by Definition of the exclusive states of those variables
Brainstorming
Creation of one node per identified variable
BAYESIALAB 5.0
Knowledge
Elicitation
Brainstorming to define the direct relationships between the
Environment variables, and addition of the corresponding arcs between those
dependent variables
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
7

8. The Parameters
Probability Distributions
Probabilities do not have to be exact to be useful
For each root node, i.e. without incoming arc, definition of the marginal
probability distribution over the defined states
Plan
For each node with incoming arc(s), definition of the conditional probability
distribution over the defined states, for each combination of the states of
Modeling by its connected nodes
Brainstorming
Each expert gives his/her belief on the distributions
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
There are various kinds of biases to be aware of
Cognitive (Plausibility, Control, Availability, Anchoring)
Emotional (Mood, Motivation)
Group (Anchoring, Herding)
Facilitator (can be biased toward charismatic experts or toward
the last expressed opinion)
☛ Use the new BayesiaLab’s Knowledge Elicitation environment
©2010 BAYESIA SAS
to reduce these biases, to improve traceability, to gather all the
All rights reserved. Forbidden
reproduction in whole or part
useful knowledge, ....
without the Bayesia’s express
written permission
8

9. Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
BAYESIALAB 5.0 Knowledge Elicitation Environment
Knowledge
Elicitation
Environment
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
9

10. The Experts
Definition of the group of Experts
Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
10

11. The Experts
This Expert Editor allows defining:
The Expert’s name, its Credibility (that will be use globally during the consensus
computation), her/his Picture, a Comment to describe her/his area of expertise.
The last field contains the number of assessments realized by the expert on the
Plan current network
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
- Group of experts can be Imported/Exported
- The Open Session button allows opening an Online Brainstorming Session*
- The Generate Tables button allows generating a Bayesian network by using the
assessments of the selected experts only
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
11
* Available on subscription only

12. The Experts’ Assessments
Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
Selecting a cell in the probability table activates the Assessment button for assessing
the question corresponding to the selected line, i.e. what is the marginal probability
distribution of Age over the 3 defined states?
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
12

13. The Experts’ Assessments
Pressing the Assessment button opens the Assessment Editor that allows the Facilitator
to manually add, delete and modify Experts’ Assessments.
Plan The Post Assessment button can be used by the Facilitator to Post the question to
the BayesiaLab’s secured website for an online assessment
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
13

14. The Expert Online Assessment Tool
The secured website
Plan
Modeling by The Expert’s name, The session name
Brainstorming case sensitive!
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
Once logged in, the
Expert is waiting for a
question
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
14

15. The Expert Online Assessment Tool:
Example with the 3 states variable Age
Once the Facilitator has posted the question with the Assessment Editor, the question is
displayed on the Expert’s webpage.
Plan The question is relative to
the node “Age”, that has 3 states: Young,
Adult and Aged.
There are then 3 sliders for the probability
distribution assessment, and another one
Modeling by for the confidence
Brainstorming There is no context
(root node). This is then a
marginal probability
BAYESIALAB 5.0
Knowledge
Elicitation Check
Environment box for fixing the
probability of the
state
Pie
Chart representing the
probability distribution
specified with the sliders
The label
corresponds to the The
Confidence level the expert comment field can be
has specified with the used for explaining the
©2010 BAYESIA SAS Confidence Slider (ranging assessment
All rights reserved. Forbidden
reproduction in whole or part from “I Do not Know” to “I
without the Bayesia’s express
written permission am Certain”)
15

16. The Expert Online Assessment Tool:
Example with the binary variable Cancer
The context variables in the BBN
Plan
Hovering over the context
variables returns the comment
Modeling by associated to the corresponding
node, if any
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
This question is relative to
node Cancer, and the specific
Context is “Age = Adult” and
“Smoker = Yes”
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
16

17. The Facilitator’s tool
This listener allows
following the status of the
Experts’ assessments
Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation Once the Expert validates her/his assessment, this assessment is sent to the
BayesiaLab’s server and the Facilitator’s listener is automatically updated
Environment
Clicking
on OK makes BayesiaLab harvesting the
assessments.
Closing the window cleared the question from the
webpage of the Experts that do not have
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
17

18. The Facilitator’s tool
Plan This gray part corresponds
to the Experts’ probability distribution
assessments
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
This second part contains the
Expert’s name, the Assessment’s
Confidence, the associated Comment
and the Time (in second) for validating
the assessment
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
18

19. The Facilitator’s tool
The content of this editor is sortable by each column just by clicking on the corresponding header
Plan It is sorted here in the ascending
order on the probabilities assessed for the
state Young
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Selecting the line allows
Environment
displaying the Expert’s picture
Sorting the assessments by state probabilities can be used for:
- detecting Experts’ misunderstanding
- Knowledge sharing, especially by making the 2 “extremes” Experts debate
If some useful knowledge comes out from the debate, the Facilitator can post
again the question for a new Expert Assessment. Each Expert will then be
allowed to update her/his assessment online (each Experts’ webpage is initialized
©2010 BAYESIA SAS
All rights reserved. Forbidden
with the information she/he set in the previous round)
reproduction in whole or part
without the Bayesia’s express
written permission
19

20. The Consensus
Once the assessments validated, a Mathematical consensus is computed by using the Experts’
credibility and their assessment’s confidence. This automatic consensus can be manually
modified by the Facilitator to set a Behavioral consensus, i.e. one issued after a fruitful debate
Plan
Modeling by
Brainstorming
Hovering over this icon A small icon is added at the left of
returns the minimum and the each probability to graphically
BAYESIALAB 5.0
maximum assessments, and the represent the consensus degree:
Knowledge number of assessments from a full transparency when there
Elicitation
all the Experts agree on the
Environment probability, to no transparency when
the range of the assessments is 1
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
20

21. The Consensus
An icon is added to the nodes for indicating the nodes that have Experts
assessments. The darker the icon is, the lower the global consensus is
Plan The probability
distributions that have a set
of assessments are framed
with a green line
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
Here is the list of
assessments corresponding
to the second line of the
table
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
21

22. The Consensus
Pressing the “i”
key while hovering over the
expert icon allows displaying
the information panel below
Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
iiiiiii
Elicitation
Environment
This information panel contains:
- the number of rows ((Conditional) probability distributions) that comes with Experts
assessments
- the total number of assessments that have been set in the probability table
- the number of Experts that have assessed at least one probability distribution in the table
- a measure of the global disagreement that takes into account the deviations from the
mathematical consensus
- the maximum disagreement corresponding to the greatest difference between two
©2010 BAYESIA SAS assessments in the probability table
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
22

23. The Assessment Report
Right clicking on the Expert Icon in the lower left
corner of the Graph window allows generating the
following HTML report.
This report first gives information on the Experts,
Plan then returns a sorted list of the nodes wrt the
global disagreements, and another one wrt the
maximal disagreements.
Finally, for each node, a summary contains all the
Modeling by global information on the assessments of the
Brainstorming (Conditional) Probability Table
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
All these informations can be useful for the Model
Validation, e.g. by checking first the nodes based
on their associated disagreements (global and
maximal), then based on the time for the
assessments (that can reflect a difficulty, or, on the
©2010 BAYESIA SAS
contrary, too prompt assessments)
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
23

24. Exportation of a Bayesian Network per Expert
Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
This exportation tool allows to create a Bayesian Belief Network for each
Expert.
The parameters (probabilities) are those assessed by the Expert. If the
Expert has not assessed all the probabilities, the model will use either the
consensual probabilities, or those manually entered by the Facilitator
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
24

25. Exportation of the Probability Assessments
This exportation tool allows creating a CSV
file with all the assessments of the
probabilities. There is one column per
variable to describe the context (yellow), one
Plan column to indicate the assessed Node
(green), the other columns describing the
assessed probability, the confidence level,
the Expert, and the assessment time.
Each line describes one assessment of a
Modeling by (Conditional) Probability Table cell
Brainstorming
MitA/TiPo MiAt TiPo Node Probability Confidence Expert Time
BAYESIALAB 5.0
Weak Weak Strong MitA/TiPo 0,97 1 Hiro 56
Knowledge
Strong Weak Strong MitA/TiPo 0,03 1 Hiro 56
Elicitation
Weak Weak Strong MitA/TiPo 0,95 1 Haitien 210
Environment
Strong Weak Strong MitA/TiPo 0,05 1 Haitien 210
Weak Weak Strong MitA/TiPo 0,8 0,58 Claire 145
Strong Weak Strong MitA/TiPo 0,2 0,58 Claire 145
Weak Weak Strong MitA/TiPo 0,85 0,77 Matt 65
Strong Weak Strong MitA/TiPo 0,15 0,77 Matt 65
Weak Weak Strong MitA/TiPo 0,4 0,8 Mohinder 76
Strong Weak Strong MitA/TiPo 0,6 0,8 Mohinder 76
Weak Weak Strong MitA/TiPo 0,75 0,9 Nathan 50
Strong Weak Strong MitA/TiPo 0,25 0,9 Nathan 50
... ... ... ... ... ... ...
Weak MiAt 0,7 0,2 Noah 76
©2010 BAYESIA SAS
Strong MiAt 0,3 0,2 Noah 76
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express Weak MiAt 0,75 1 Matt 90
written permission
25 Strong MiAt 0,25 1 Matt 90

26. Exportation of the Expert Assessments
This exportation tool generates a CSV file
with all the assessments of the Experts.
There is one column per Expert, one column
per Expert’s Confidence (yellow), the last
Plan column indicating the weight of the line (1/
number of states of the assessed variable)
(green).
Each line describes the Experts’ assessment
Modeling by of a(Conditional) Probability Table cell
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
Hiro Hiro Haitien Haitien .... Noah Noah Weight
Confidence Confidence Confidence
0,97 1 0,95 1 .... 0,7 0,8 0,5
0,03 1 0,05 1 .... 0,3 0,8 0,5
0,3 0,81 0,05 1 .... 0,3 0,7 0,5
0,7 0,81 0,05 1 .... 0,7 0,7 0,5
0 1 0 1 .... 0 0,79 0,5
1 1 1 1 .... 1 0,79 0,5
0,65 0,79 0,71 1 .... 0,7 0,2 0,5
©2010 BAYESIA SAS 0,35 0,79 0,29 1 .... 0,3 0,2 0,5
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
26

27. Analysis of the Expert Assessments
We then can used this file to analyze the direct probabilistic
relationships that hold between the Experts’ assessments
Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
This network has been
automatically learned with one of
the BayesiaLab’s Association Each node represents the
©2010 BAYESIA SAS Discovering algorithms on a set discretized probabilities
All rights reserved. Forbidden
reproduction in whole or part
of 120 Experts’ assessments assessed by the Expert
without the Bayesia’s express
written permission
27

28. Automatic Segmentation of the Experts
Based on the obtained network, Experts can be clustered into homogeneous
groups by using the BayesiaLab’s Variable Clustering algorithm
Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
Dendrogram Each color corresponds to a
corresponding to that cluster. Three segments of Experts have
segmentation been induced in that example.
The real experts behind those anonymized
experts have indeed three different profiles
(functionally and geographically)
Based on the obtained Expert Segments, one Bayesian network per segment can be
generated (by using the Expert Editor). This can be useful for analyzing the sensibility of
©2010 BAYESIA SAS
the model, but also to get specific networks (depending on the geographical localization
All rights reserved. Forbidden for example)
reproduction in whole or part
without the Bayesia’s express
written permission
28

29. Parameter Sensibility Analysis
BayesiaLab also comes with an Assessment Sensitivity Analysis tool
that allows measuring the uncertainty associated to the consensus.
The general idea is to generate a set of networks by randomly
drawing Experts’ assessments, and then measuring the uncertainty
Plan associated to each probability distribution.
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
29

30. Parameter Sensibility Analysis
Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
Three kinds of analysis are available, depending on the Random selection policy that is
chosen to generate the set of networks (1000 networks in the above example):
- One Expert per network: each network generated is parametrized by using the selected
Expert (or the consensual probability if the selected Expert has not been involved in the
assessment)
- One Expert per node: each network generated is parametrized by selecting for each node
©2010 BAYESIA SAS one Expert. If the selected Expert is not involved, the consensual probability if the selected
All rights reserved. Forbidden - One assessment per Conditional Probability Table’s row (if any)
reproduction in whole or part
without the Bayesia’s express
written permission
30

31. Parameter Sensibility Analysis
States of the Tabs for the selection of
analyzed variable the variable under analysis
Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge Density function illustrating
Elicitation Marginal
the uncertainty associated to probability distribution of the
Environment this node. The Mean over the Target node computed with
1000 networks (one Expert all the consensus
per network) is 70.62%
(versus 70.46% in the
monitor), the Standard
Deviation 2.13%. There are
62% of chance of having a
probability comprise between
70 and 72%
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
31

32. Parameter Sensibility Analysis
Plan
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation
Environment
One Expert per node
One Expert per Conditional
©2010 BAYESIA SAS
All rights reserved. Forbidden
Probability row
reproduction in whole or part
without the Bayesia’s express
written permission
32

33. Contact
Dr. Lionel JOUFFE
Plan President / CEO
Tel.: +33(0)243 49 75 58
Fax: +33(0)243 49 75 83
Modeling by
Brainstorming
BAYESIALAB 5.0
Knowledge
Elicitation 6 rue Léonard de Vinci
BP0119
Environment
53001 LAVAL Cedex
FRANCE
©2010 BAYESIA SAS
All rights reserved. Forbidden
reproduction in whole or part
without the Bayesia’s express
written permission
33