1. ”You get what you measure” or not?
Challenges for fact-based quality management
Juhani Anttila
Venture Knowledgist – Quality Integration
www.QualityIntegration.biz , juhani.anttila@telecon.fi
15.3.2008 These pages are licensed under
Creative Commons Nimeä 2.5 lisenssi
1 http://creativecommons.org/licenses/by/2.5/deed.fi

2. ”You get what you measure (*)” or not?
Challenges for fact-based quality management
1. Searching a balanced position to statistical quality management
2. Factual business performance
3. Facts, data, information and knowledge for business management
4. Using measurements, analyses, and knowledge within business
management
5. Example: Using ZEF methodology for business performance evaluations
6. A topical problem: Quality of networks
7. Summary and conclusions
2
(*) 32500 Google hits
xxxx/2.3.2008/jan

3. What am I striving for?
Statistics Quality
Mathematical science pertaining to the Degree to which a set of inherent
collection, analysis, interpretation or characteristics fulfils needs and
explanation, and presentation of data expectations
Quality management (Quality of management)
Degree to which coordinated activities of management to direct and control the organization
fulfill the needs and expectations of organization’s all stakeholders
Principles of good management
Factual approach to decision making (ISO 9000 - Quality management principles)
Management by fact (Malcolm Baldrige - Core values and concepts)
Management by processes and facts (EFQM - Fundamental concepts of excellence)
Business methodology
Business performance management
Measuring business process performance
3
3462/2.3.2008/jan

4. Positioning to statistics in quality management
Quality management Constraints:
traditions: - Theoretical /
-Statistical process mathematical
control difficulties
-SixSigma - Problems
-Problem solving - Warnings
-Taguchi techniques
Walter Shewhart How to find balance? Darrell Huff
1931 1954
4
3461/16.2.2008/jan

5. Information problems and danger of empirism
• Data hell – Chaotic data disturbance
• Data bulimia or data anorexia
• Data mechanization – overly emphasized
information technology and impoverished
skills
• Knowledge splintering into objective
information pieces
• Data pedantry and data naiveté
• Popularization – Complex realities overly
simplified and information contents
impoverished
• Information insecurity
• Information capitalism or information
proletariat – misuse of information power
5 • Information terror and data crime
3465/14.2.2008/jan (Ref.. Jaana Venkula, Huff, etc.)

6. Warning about the danger of information (*)
Information is not knowledge. Knowledge comes from theory. Without theory, there is no way
to use the information that comes to us on the instant. There is no true value of any
characteristic, state, or observation.
Warning about people rewarding based on quantitative measurements, deadly disease #3 of
leadership:
“Evaluation of performance, merit rating, or annual review nourish short-term performance,
annihilates long-term planning, builds fear, demolishes teamwork, nourishes rivalry and
politics. It leaves people bitter, crushed, bruised, battered, desolate, despondent, dejected,
feeling inferior, some even depressed, unfit for work for weeks after receipt of rating, unable
to comprehend why they are inferior. It is unfair, as it ascribes to the people in a group
differences that may be caused totally by the system that they work in.”
More important than to create scrupulous and even harmful business measurement systems,
is to get tacit knowledge of business leaders and workers effectively interact with each other
for creating extensive profound knowledge within an organization.
6
3466/14.2.2008/jan (*) W. Edwards Deming (e.g. in "The new economics")

7. Pragmatic (*) genuine knowledge
Pragmatism (C.S. Peirce): Action based information / knowledge
approach:
• Genuine knowledge makes possible to act in a new
meaningful way.
• Knowledge can be genuine only if it has been tried and
justified in operation.
• Knowledge is a basis of an organization and a community
and their practical operations.
• Knowledge cannot be objective.
Charles Sanders Peirce
”The knowledge you take
is equal to the knowledge you make.” (**)
(*) Ref. the other paradigms of knowledge theories:
- Heuristic (internal comprehension based knowledge)
- Empirical (external observation and experience based knowledge)
7
2475/4.3.2008/jan (Ref.: Jaana Venkula) (**) Lennon, McCartney

8. Quantitative approach to driving business
performance
Quantitative indication of performance:
- Phenomenon to be considered of a business interest
and its character – especially in business processes
- Measure / indicator, characteristic, quantity
- Metrics, unit of measure
- Numerical values and value range of the quantity
Practicalities of performance management:
- Target / required values, observed values
- Meter, gauge, means to measure / observe the quantity
- Measurement, assessment technique
- Uncertainty
- Conclusions, decisions and actions based on
measurements
- Approach and practices for performance improvement
8
1613/10.3.2008/jan

9. Performance measurements for business
management – Need of business information
Management areas: Purpose of measurements:
- Operational management - Research activities for getting new
- Strategic management knowledge
- Acquisition of information for planning
business or operations
Scope of measurements: - Controlling operations and processes
- Performance of business processes !Performance monitoring
- Performance of products (goods and !Determination measurements
services) !Verification measurements,
hypothesis testing (AQL / LQ)
!Validation testing
Related aspects: - Measurements for problem solving
and performance improvement: Kaizen,
- Measurements management breakthrough and learning
- Business performance management - Measurements for quality assurance
9
3467/2.2.2008/jan

10. From business activities to overall performance of
an organization and to performance excellence
Overall business performance
References:
} Competitive performance
Customer- Operational Product Financial and
focused performance performance marketplace
performance - processes - goods performance PERFORMANCE
- workforce - services EXCELLENCE!
Excellence is originated from
the business activity not only
Financial Non-financial being superficially outstanding
measures (data) measures (data)
DOMAIN OF ACTION - BUSINESS ACTIVITY
(Operational business processes and projects)
10
0877/22.3.2008/jan (Ref.: Malcolm Baldrige Criteria)

11. Interesting information on business performance
• Information available from all key business areas and processes and related
requirements:
– Consistent and balanced quantitative data through measures and
indicators of business related facts
• Level of performance
• Rate of improvement, or slope of trends in data demonstrating development
of business performance
– ”Three generation reporting”: Past, now, future
– Pattern of development: Linear, non-linear, spiral
• Comparisions of performance levels and their development with own
targets, competitors, and World-Class benchmarks
• Relationships of achieved quantitative business results with management
and process activities: Cause-effect linkages
11
3468/2.2.2008/jan (Ref.: Malcolm Baldrige Criteria)

12. Temporal dimensions of the present moment “Now”
Time
The ‘Now’ of future events and performance
The ‘Now’ of present events and performance
The ‘Now’ of past events and performance
} Kairos Kronos
The present “Now” always includes the
impacts of the past events and the
challenges of the future events!
12
2583/5.3.2008/jan

13. What is information security all about?
Basic concepts characterizing security of the information consist of:
• Integrity – information that one is using for his/her actions is accurate
• Availability – one has access to the relevant information when and so long it is
needed
• Confidentiality – information one is using is not manipulated by anybody else.
Additionally authenticity (information is authentic, trustworthy, or genuine) and
authority (one has right to use the information) aspects are significant especially
when using the means of electrical communication.
The most essential information security
aspect is privacy!
13
2476/11.2.2008/jan (Ref.: ISO/IEC 27001 / 27002)

14. You don’t get necessarily what you measure
Measurements in all kinds of business activities are important for management but in
practice:
• Measurements are often excessive, unfocused or wrong (*)
• Business results you get don’t necessarily relate to what you measure
• What you get through measurements is not necessarily meaningful or good for
organization or people being managed
• Business leaders may not know what they don’t know (“Tacit ignorance”)
Measurements results represent explicit knowledge of the business performance but
business intentions and real business results are principally of tacit knowledge of
business leaders and workers in the organization.
No measurements can be objective but they are always affected by the intension and
awareness of somebody – even in physics objectivity is impossible. What is being
measured, by what kinds of means or methodology, what is obtained through
measurements, and how the measurements results are understood – they all depend on
somebody’s intention and awareness.
14 (*) Dr. Juran:
“Vital few” / “Trivial many”
3469/5.3.2008/jan

15. Facts & knowledge for managing business activities
Wisdom
- myths
- values
Knowledge Intervention
- explicit records Reflecting and deciding
- tacit knowledge Plan / Act
(know-how, competence)
Information
”Ba” Analysing
You get what
A P you measure
Data
C D
Environments Measuring ...
Facts Effects
15 The performance reality of the company business processes
0609/25.3.2006/jan

16. PDCA model (*) for a good management: Coordinated
activities to direct and control an organization (**)
P = Planning
D = Doing
ACTING: PLANNING: C = Checking
• Preventing actions • Business and A = Acting
• Improving actions management models
• Re-engineering • Business plan
• Communicating Applying PDCA model:
• Approaches and
• Recognizing and methodology • Rational control (operational)
rewarding • Continual rational small step
A P
improvement (operational),
C D
CHECKING: DOING: “Kaizen” approach
• Assessing the • Deploying the approach • Innovative breakthrough
performance and achieving the results changes (strategic) (***)
• Reviewing the • Controlling operational
performance performance
All three PDCA loops modes and
• Corrective actions
four PDCA phases are strongly
16 information / knowledge dependent
2343x/15.8.2007/jan (*) Deming / Shewhart Cycle (**) ISO 9000 (***) Dr. Shiba

17. Control of the business activities
- feedback
Business Operations Business output
requirement (Business processes)
input
Top
Notes:
Business • Requirements for control :
control Throughput times Tfb<<Top
input • ”A decision must be made
at lowest possible level in
Feedback the organization and as close
Tfb as possible to where its
outcome will be executed.”
(Peter Drucker)
Target
17
1847/26.2.2008/jan (Ref.: Bode, Asby)

18. Traditional quality tools
Traditional ”quality tools” have a long history and are very strongly related with data prosessing and
management of business operations. Assortment of the methods may seem confusing, despite that they
have been arranged into different groups such as Seven basic quality tools, Seven new management and
planning tools, 127 tools, Kaizen tools, SixSigma tools, etc. tools. These tools include e.g.:
Simple basic tools: Process management tools: Inspection, testing or
•Data collection tools •Control charts acceptance methods:
•Idea creation, brainstorming •IDEF0 •Attribute testing
tools, nominal group technique •Process capability methods •Variable testing
•Check lists / sheets •Flow charts Data analysis methods:
•Survey methods •Statistical tests
•Affinity diagram Project management tools:
•Matrix diagram •Pert / Gantt chart •Variance analysis
•Histogram / bar graph Reliability (dependability) tools: •Regression analysis
•Scatter plots •Failure mode effects and •Design of experiments
•Relations diagram criticality analysis (FMECA) •Z-score analysis
•Cause-effect, fishbone or •Fault tree analysis (FTA •Value analysis
Ishikawa diagram •Reliability prediction
•Pareto chart •Reliability testing
•Evaluation and decision-
making tools, decision three, Recommendation: Organizations should establish and maintain
18 force-field diagram their tool sets for quality management, “Business excellence tool
•Voting methods kit”, as well as the measurement and information management at
3470/26.2.2008/jan large.

19. Using measurements, analyses, and knowledge
within business management (*)
Enablers criteria
Category 4: Measurement, analysis, and knowledge management:
!Measurement, analysis, and improvement of organizational performance
(Assessment area 4.1)
!4.2 Management of information, information technology, and knowledge
(Assessment area 4.1)
Results criteria:
Category 7: Results
!Product outcomes (Assessment area 7.1)
!Customer-focused outcomes (Assessment area 7.2)
!Financial and market outcomes (Assessment area 7.3)
!Workforce-focused outcomes (Assessment area 7.4)
!Process effectiveness outcomes (Assessment area 7.5)
!Leadership outcomes (Assessment area 7.6)
19
3472/.20.3.2008/jan (*) Malcolm Baldrige Criteria 2008

20. Using measurements, analyses, and knowledge
within business management (*)
Enablers criteria:
!Criterion 2: Policy and strategy
- Policy and strategy are based on information from performance measurement,
research, learning and external related activities (Criterion part 2b)
!Criterion 4: Partnership and resources
- Information and knowledge are managed (Criterion part 4c)
Results criteria:
!Customer results (Criterion 6)
!People results (Criterion 7)
!Society results (Criterion 8)
!Key performance results (Criterion 9)
20
3473/.20.3.2008/jan (*) EFQM Criteria

21. Information technology serves our knowing and
learning as an extension of the body
Impact of technology:
The form of a medium imbeds itself in the message,
creating a symbiotic relationship by which the medium
influences how the message is perceived, creating
subtle change over time.
We shape our tools and afterwards our tools shape us.
Use of technology:
– Collecting data
– Processing data and information Marshall McLuhan 1911-1980
– Storing data and information
– Sharing / communicating information and
knowledge
(IT = Interactive technology)
21
3474/2.1.2008 /jan

22. Collaborative knowledge working / learning
environment with Web 2.0 tools, “Enterprise 2.0”
Collaborative knowledge working environment for activity XXX at organization XYZ:
Group- Common shared knowledge-base accumulated,
members discussed, and updated by the group-members
(Wiki)
Knowledge feeds from group-members'
findings, observations, and reflection,
and comments
(Blog)
Discussion forums Automatic information
feeds from external
Documents files sources, e.g. Internet
(Aggregators)
" Links to
• Other tools (calendar, to-do list, diary,
22 Skype, virtual meeting room, etc.)
• Administration of the environment
3372/15.9.2007/jan

23. Web 2.0 challenges
Web 2.0: Second-generation of Internet-based services - such as social networking sites,
wikis, communication tools, and folksonomies - that let people collaborate and share
information online in ways previously unavailable
RSS: (Rich Site Summary) Really Simple Syndication
Mashup: A website or application that combines content from more than one source into
an integrated experience
Tag cloud: A visual depiction of content tags used on a website
Facebook: a social utility that connects people with friends and others who work, study
and live around them
RTE – Real Time Economy: Speeding up information flow between companies, each
having capabilities to monitor their businesses continuously and quickly react to changes
and exceptions
23
3460/2.3.2008/jan (Ref.: http://www.oreilly.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html )

24. Example: ZEF evaluation methodology
ZEF is an advanced interactive evaluation methodology and tool that is originated from strong core
competences of evaluation technique and human interface technology.
Origin:
• ZEF methodology was developed in Finland at Oulu University as Z-scored Electronic
Feedback (= ZEF) tool. Effectiveness of its two dimensional evaluation structure was
validated by University of Lapland.
• Commercial ZEF solution is a product of Oulu based company ZEF Solutions Inc.
Usage:
• ZEF is an all-purpose tool that is suitable for all kinds of organizations.
• ZEF evaluation services are web-based solutions operating as SaaS (Software as a Service).
With ZEF on-line tool one may collect, compare and evaluate data effectively and efficiently
from even a big group of people.
• ZEF tool has been used by a lot of organizations for business information and hundreds
thousands of ordinary citizens in several survey-cases of general societal interest.
24
3336/.20.3.2008/jan (Ref.: http://zefsolutions.com/www/english , http://qiblog.blogspot.com/2007/06/carry-out-appropriate-surveys.html )

25. ZEF evaluations
ZEF tool is applicable for:
• All kinds of quality-related performance evaluation, feedback and survey
applications
!E.g. 3-In-1 assessment scheme that combines Malcolm Baldrige model,
EFQM model and ISO 9000 and is easy to tailor to specific business aspects
or organizational strategic priorities. Examples include modifications for
!Health services quality management (CEN/TS 15224)
!Food safety management (ISO 22000)
!Information technology service management (ISO/IEC 20000)
!Process performance evaluations
!Customer / workforce satisfaction
• Other business related assessments needs including
!Innovations evaluation
!Risk, vulnerability and threat analyses
!Strategy analyses
!Customer lead surveys
• Other general e.g. social, entertainment, etc. applications including
!National innovation policy surveys
!Parliamentary / presidential election comparisons
!Education and school evaluations
25 !Idols rating
3476/20.3.2008/jan

26. Improving effectiveness / efficiency of self-assessments
with an advanced tool (ZEF)
26
3288/20.3.2008/jan (*) ZEF Solutions Inc., Finland: www.zef.fi

27. Performance evaluation for excellence
through a multi-faceted and consistent approach
Overall business performance (e.g. 3-In-1 criteria)
2. Strategic 5. Human
management resource focus
7. Business
1. Leadership results
3. Customer and 6. Process
market focus management
4. Measurement, analysis,and knowledge management
Leadership performance Customer satisfaction Information security
Social responsibility Market communication Product
Innovations performance
IT Governance
Board operations
Strategic performance Process performance
Corporate security People satisfaction Project performance
Risk management 360 feedback Supplier performance
27 Networking capability OHS assessment Quality Assurance:
ISO 9001, AQAP
3325/26.3.2008/jan Environmental management

28. Results of ZEF evaluations (“3-in-1”)
Assessesment scheme “3in1”: EFQM, Malcom
Baldrige and ISO9000 combined in one scheme
Category 4: Measurement, analysis, and
knowledge management
1. We follow up fulfilment of targets and
development of performance on company and
process levels with adequate measures and
indicators.
2. We follow up effectiveness and efficiency of
our business processes.
3. We have easily available analyzed and
current information on performance of our
company and its development against targets
and other references.
4. Our real-time internal accounting operates
in advance.
5. Our employees measure, analyze and
evaluate quality of their own work in order to
make decisions and initiate improvement
actions for their work in line with our
28 company-wide plans. …
… Etc. other assessment items
3475/20.3.2008/jan

29. Results of ZEF evaluations
29
3334/.20.8.2007/jan

30. Statistical inferences from ZEF evaluations
Detailed business information is obtained from ZEF raw data through statistical treatments.
All individual respondent answers to the query items in the ZEF evaluations are recorded as
separate data pieces that are then used for statistical analyses as needed:
! The basic ZEF report shows the means and standard deviations of all respondents’
answers per query items in one or two dimensions as needed.
! The result data may be normalized according Z-scored transformation that is useful to
emphasize relative differences among query items.
! Answers from respondents may be grouped in arbitrary way or presented as
individual answers.
! Results of different ZEF evaluations, e.g. of different points in time, may be compared
by using ZEF’s “comparison engine” function.
! Raw data of evaluations may be presented in the formats of Excel (.xls) or
"Concurrent Versioning System" (.cvs) in order to facilitate the use of more
sophisticated statistical tools, e.g. SPSS, for data analyses.
ZEF tool’s interactive user-interface enhances user’s deliberation when he/she is responding
to the query items and also enhances understanding the relative differences among query
items. That makes answering more motivated and justified.
30
3471/.20.3.2008/jan

31. Z-score transformation
Z-score for an item indicates how far and in what direction that item deviates from its distribution's
mean expressed in units of its distribution's standard deviation. The z-score transformation is such
that if every item in a distribution is converted to its z score, the transformed scores will have a
mean of zero and a standard deviation of one.
Z-score transformation is especially useful when seeking to compare the relative standings of
items from distributions with different means or different standard deviations.
31
3478/20.3.2008/jan (Ref.: http://www.sysurvey.com/tips/statistics/zscore.htm )

32. Z-scored normalized evaluation results (ZEF)
32
3477/.20.3.2008/jan

33. Summary and conclusions
1. Competitiveness and success of all organizations are based on right business related
information and knowledge on time.
2. Impressive business phenomena and their nature including stochastic features should be
taken into account in planning business performance measurements.
3. Business related measurements and information may have many different purposes.
4. One should recognize vital few valid subjects for business performance measurements.
5. Measurements should be based on sound methodologies and effective measuring tools.
6. Observed data should be understood in the business context and also data’s statistical
viewpoints should be noted.
7. Tacit knowledge and explicit information should be brought together in making
conclusions and activating managerial measures.
8. Interactive information technology gives remarkable advantage and serves as an
extension of our body and senses both in making observations and in using information
within business communities.
9. Cooperation of quality and statistics experts is obviously useful.
33
3479/.20.3.2008/jan

34. j
Overall quality of a network
Win / Win in network
Overall quality of a network Qvm:
Degree to which the network fulfils needs and
S S expectations of all network members:
i - m members in the network
A A - Each member gets benefits (Si) but also
loses something (Ai).
Network Network
member i member j
Number of knots with k links
Knots with
few links
Power law of a scale-free network:
Knots with
many links
A topical problem:
How to measure in practice
the overall quality of a network?
34
3464/15.2.2008/jan Number of links (k)

35. Please, evaluate my presentation
and material and
give your feedback with ZEF:
http://www.zef.fi/service/user/?q=597
Thank you!
35
xx/15.2.2008/jan