All the presentations that were held at the Data Vault automation conference (October 6th 2011). Organized by Ronald Damhof, Tom Breur and Simone Molenaar (DIKW).
We had 110 attendees, 8 sponsors.
4. To Push or To Pull, That is the question
Ronald Damhof
woensdag 21 september 11 1
!"#$%&
!"#$%&
Taken from ‘Out of the Crisis’, Dr. W.Edwards Deming
'!&()*+%',)-
!"#$%!&'(%)*+!,+(-#./('!,+(.*$$
woensdag 21 september 11 2
7. Push characteristics
! Mass production
! Known specifications, operational definitions, standards
! Repeatable, predictable, & even better; uniform process
! Part of the system that needs statistical control
! Inventory allowed/necessary
! Supply driven
! Reliability over flexibility
Pull characteristics
! Just in time
! Demand driven
! Build to order
! Preferably no inventory
! Flexibility over Reliability
woensdag 21 september 11 7
D&%4-4&*/
?@'A&*&$)/&'F*1"$()+"*-$"8#%/2
0'1(+2&/('!3*456*+7!,+(.*$$
7*89#2&$':;"%)<=
.)/)'0'1#*%+"*'2&$34%&
? ? ? ? ?
>@'7*$4%B')*8'%<&)*2&'8)/)
> > > > >
5@'D&,42/&$'0'!/)*8)$84E&
5 5 5 5 5
6@'A&/'/B&'$)C':#*%#/='8)/) 6 6 6 6 6
8*'*+5.!90!,+(.*$!:;*'%+&4<
.)/)'2"#$%&2
0'1(+2&/('!3*456*+7!=+(.*$
woensdag 21 september 11 8
10. Remember the Push characteristics
! Mass production Data Vault
! Known specifications, operational definitions, standards Data Vault
! Repeatable, predictable, & even better; uniform process Data Vault
! Part of the system that needs statistical control Data Vault
! Inventory allowed/necessary Data Vault
! Mainly supply driven Data Vault
! Reliability over flexibility Data Vault
Automation of a Data Vault production system is just common sense
woensdag 21 september 11 13
WS'R"O'
IT"#/':.)/)'O)#</=')#/"()+"*'Y""<4*,
! A&*&$)+"*'42')*')48J'*"/')',")<'4*'4/2&<1
– '."'*"/')%%"("8)/&'/B&'-$4*%4-<&2'/"'Z/'/B&'/""<@@@@
– ';""V'1"$'8&%"#-<4*,
! Y$#<S'#*8&$2/)*8'/B&'(&%B)*4%2'9'B)*8%$)['4/'Z$2/
– 'F*3&2/'4*'-$"-&$'&8#%)+"*')*8'<&)$*4*,
– 'F*3&2/'4*'K,&]*,'$&)8SL'+(&
– 'F*3"<3&'S"#$'K%#2/"(&$2L'1$"('/B&'2/)$/
! R"UJ'R"UJ'R"U
! .&<43&$J'.&<43&$J'.&<43&$
woensdag 21 september 11 14
26. • BI strategy development • Maintenance & support
• Information analysis • Data vault technology
• (E)DW architecture • Quipu development
• Project management
• Adapttm training
27. QUIPU: Open Source DW generation
• Open Source Data Warehouse Generation System, based on
Data Vault principles
• First public release July 1st 2010
• QOSQO takes a leading role in continuous development and
support
28.
29. Fast implementation
of DV based EDWH
Removal of
repetitive tasks
Reduction of risk
of modeling errors
Source:
31. QUIPU - Key IT benefits
• Automated data warehouse data model design
and implementation
• Fully repository based metadata driven data
model and load code generation
• Supports most common database platforms using
ANSI-SQL over JDBC
– Template based platform support
• Integration with ETL and scheduling tools
• Lower total cost of ownership using open source
licensing model
36. Business model
• Development of new functionality
– Paid customer assignments
– QOSQO roadmap priority
• Support
– Quick start consultancy
– Proof of Concepts
– Flexible support model
• On site
• Remote
• Training
• Quipu Model Manager
– Paid software
– Hosting
37. Quipu products
Community Model
Edition Manager Powered by
New DWH Management &
developments Maintenance
- Open source - Closed source - Embedded in BI
- Generate models - Manage models solutions
- Single user - Delta changes
- Continuous - Multi-user - CaseWise Modeler
developments and solution
improvements
- New: Data mart - New
New product - New solutions
DM product
generation roadmap
generation roadmap
assistance
38. Data Mart assistance
• In cooperation with BinckBank
• Logical layer on top of DataVault
• Basic Starschema or snowflake generation
39. Quipu Model Manager
• Version control of data models
• Multiple users, projects, versions
• Quipu Community Edition as client
• Check in / Check out
• Migration of run time DW data
• Central repository of models and code
Quipu
CE
Quipu
Quipu MM
CE
Quipu
CE
40.
41. • Download and evaluate Quipu (it’s free!)
• Share your experience and feature wishes
• Hire us
43. QOSQO, the DataVault Our sister company
Karel Doormanlaan 1b
specialist, is the leading Nippur assists in
5688 BP OIRSCHOT company behind Quipu executing business
The Netherlands intelligence projects
E: info@QOSQO.nl
T: +31 ( 0499 ) 577 562 www.QOSQO.nl www.nippur.nl
F: +31 ( 0499 ) 577 059
open source data warehousing
47. 26-09-2011
!"#$%&'(#)*+&%)&,#%'
9/:.'-,77.#&'5%&"'0"&"'6%7.4+&.#8'
6%7.#&4'
0"&"' 3,4+/%44' 0"&"'
-.,#)%' -&"$+/$'
1",2&' 1",2&' 5"#&'
(/"284+4'
Source Back End Front End Reporting &
Systems Systems Systems Analysis
9/:.'-,77.#&';./&#.2'</=+#./>%/&'
!"#$%&'(#)*+&%)&,#%'
9/:.'-,77.#&'5%&"'0"&"'6%7.4+&.#8'
6%7.#&4'
0"&"' 3,4+/%44' 0"&"'
-.,#)%' -&"$+/$'
1",2&' 1",2&' 5"#&'
(/"284+4'
Source Back End Front End Reporting &
Systems Systems Systems Analysis
9/:.'-,77.#&';./&#.2'</=+#./>%/&'
3
66. !"#$%&'()*+,%)-*./0/-&%
90% Centennium 70% Customer 100% Customer
100% Centennium 30% Customer 40% Centennium 10% Centennium
!"#$%&'& !"#$%&(& !"#$%&)& !"#$%&*&
+,-.%/%,0&
1-8-2011 31-12-2011
!"#1 "+&+%3+4,&%
7.+)-)-2%+-8%!6+*9)-2%6-1&9/1:6;% <4((6.&)-2%*4=&60/.%
&.+)-)-2% !/.&)5)*+&)6-%
Typical increment ranges from 2 to 6 months
Centennium role changes from LEAD to FOLLOW
Customer is fully CDM-aware at the end of the increment
Centennium continues supporting customers through
knowledge partnership
>?@A7%!BC7BCCDA#%
28
88. “Our highest priority is to satisfy the
customer through early and continuous
delivery of valuable software”
Agile Manifesto, 2001
Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham,
Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern,
Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas
www.xlntconsulting.com 2
89. Counter intuitive Agile practices
! People are more productive if nobody tells
them what to do
! Pair programming leads to more (effective)
production code
! Business partners must be full-time
engaged (co-located) with the
development team
www.xlntconsulting.com 3
90. Counter intuitive Agile practices
! Only the business has the right to choose
what gets done
! An efficient team must have “slack”, must
have people sitting idle, with nothing
productive to do, on a regular basis
! Etc.
www.xlntconsulting.com 4
91. Software ‘inventory’
“Work-in-Progress is a liability
– not an asset”
Tom Breur, 2011
www.xlntconsulting.com 5
92. Simplified development
Error Reports
Idea Develop Test Working Code
www.xlntconsulting.com 6
93. (More) realistic development
Idea Analysis Design Code
Error Error Error
Working Acceptance System Unit
Code Test Test Test
www.xlntconsulting.com 7
95. Throughput Accounting metrics
THROUGHPUT INVENTORY
Rate of cash* generated through Quantity of ideas for client-valued
delivery of working code into functionality queing for input to, in-
production, not merely code process through, or waiting for
complete output, from the system
*Assuming a constant level of Investment
INVESTMENT OPERATIONAL EXPENSE
The sum of money invested in the The sum of money spent in the
system of software production plus system to produce working code from
the sum spent to obtain the ideas for ideas for client-valued functionality
client-valued functionality input to the (marginal expense to create
system (gathering requirements) production code)
www.xlntconsulting.com 9
96. ROI in Throughput Accounting
Unknown (T) – Pretty hard to guess (OE)
ROI =
Didn’t bother to measure (I)
www.xlntconsulting.com 10
97. NP in Throughput Accounting
(more) Net Profit (NP) = T – (less) OE
www.xlntconsulting.com 11
98. ROI in Throughput Accounting
Throughput (T) – Operating Expense (OE)
(more) ROI =
(less) Investment in Inventory
www.xlntconsulting.com 12
99. ROI in Throughput Accounting
(more) Net Profit (NP) = (more) T - OE
(more) Throughput (T) – Operating Expense (OE)
(more) ROI =
Investment
www.xlntconsulting.com 13
100. Focus on Throughput
! Focus on T, I, or OE?
! Throughput is unlimited, it can grow
forever
! Focusing on cost has a logical (yet
unattainable) lower bound – namely zero
! Throughput focuses on the customer –
externally
! Cost focuses on the team – internally
www.xlntconsulting.com 14
101. Investment
! Minimizing Investment (I) drives ROI up
! Minimizing Investment also reduces OE,
by reducing carrying cost of capital
! And, most importantly
! Lower I means lower inventory,
which leads to reduced Lead Times,
hence earlier delivery of value
(Agile Manifesto principle #1)
www.xlntconsulting.com 15
102. Cost vs Throughput Accounting
Cost Accounting Throughput Accounting
! Inventory is an asset ! Inventory is a liability
! Efficiency = function/ ! Efficiency = function/
dollar (hours) " labor is direct costs (idle or not)
a “variable” cost " labor is a “fixed” cost
! People sitting idle are ! People sitting idle are a
discarded! part of the system!
www.xlntconsulting.com 16
103. Cost vs Throughput Accounting
Cost Accounting
Operating
Inventory Production
Least Focus
Expense
Most Focus
Throughput Operating
Inventory
(Production) Expense
Throughput Accounting
www.xlntconsulting.com 17
104. Agile & Data Vault
! (very) few other architectures allow
incremental build at such low marginal
cost
! Deliver early – in (very) small increments
! (very) few other architectures allow
‘mistakes’ in your model, that you can
recover from inexpensively
! Deliver early – (long) before you have settled
on “the” final business model
www.xlntconsulting.com 18
105. Conclusion
! By providing appropriate metrics
(=Throughput Accounting), complex
adaptive systems (Agile projects) will
display the desired emergent properties
! Agile BI is not about delivering faster (or
cheaper) – efficiency
! Agile BI is about delivering in arbitrarily
smaller increments to end-users – hence
gathering feedback about effectiveness
www.xlntconsulting.com 19