Overview of Agile BI with maturity model to assess your current state and determine steps to move forward

2. Agenda
◊ Agile BI Definition
◊ Assess Your Current State
◊ Then…
–
–
–
Select an Agile Methodology
Have a Kickoff
Inspect and Adapt

3. BI
BI encompasses all aspects of a system needed to
produce meaningful information to drive data driven
decision making
–
–
–
–
Data Processing (Cleansing, Transforming, Loading)
Data Architecture and Warehousing
Data Analysis and Visualization Tools
System Architecture

4. Agile

5. Agile BI
Applying an Agile mindset to business intelligence
◊ Using an iterative, incremental, evolutionary
approach
◊ Focusing on value-driven development
◊ Delivering production quality applications
◊ Using barely sufficient processes
◊ Automating everything
◊ Collaborating with the customer
◊ Encouraging self-organizing and self managing
teams

6. Why Agile?
The people who need
the data see the data
You need different
data? Sure!
The data source
changed? We’re on it!
Business Intelligence,
enough said
See Visibility (plus lots
of testing!)
Source: http://www.versionone.com/Agile101/Agile-Software-Development-Benefits/

7. Agile BI Maturity Model
Level 3
*All team members can independently
complete any task from database
design to report creation
*It’s not about getting your job done
it’s about getting the job done
Level 2
*Increased collaboration
Example: ETL developers work with
data modelers to come up with a
database design that balances the
tradeoffs between reporting and
loading
*Decreased formality in interactions
across skill sets
Level 1
Team Roles / Skill Sets
*Collaboration among people with the
same skill set - Example: data modelers
work with other data modelers
*Official transitions and likely
disagreement across skill sets Example: ETL developers are given
source to target mappings when the
data modelers complete the database
design and are upset that the design is
hard to load
Technical Architecture
Engineering Practices
*Clear understanding of data’s business
value
*Clear understanding of the purpose
for each component of the technical
architecture
*End-to-end use of optimal engineering
practices
*Team self-enforces usage through
criteria for completing work
*Active effort to clarify understanding
of data’s business value
*Streamlined architecture where
possible
*Process to deprecate unused
components
*Numerous (possibly) redundant layers
(staging, ODS, EDW, data marts, etc.)
*Inclusion of data with no clear
business value
*Lingering tables, reports, ETL scripts,
with no known purpose
*Some configuration management
(SQL scripts to create all db objects are
under CM, but not ETL and report
information)
*Some automation is in place (perhaps
to promote new objects or code to
another environment or to test ETL)
*Building blocks of the system (db
create scripts, ETL packages, report
files, etc.) are not maintained in any
central location nor are they under
configuration management
*Files are manually copied from one
environment to another

8. Assess Your Current State
◊ How well is your team setup for collaboration and
change?

9. Agile BI Maturity Model
Level 3
*All team members can independently complete any task from database
design to report creation
*It’s not about getting your job done it’s about getting the job done
Level 2
*Increased collaboration
Example: ETL developers work with data modelers to come up with a
database design that balances the tradeoffs between reporting and loading
*Decreased formality in interactions across skill sets
Level 1
Team Roles / Skill Sets
*Collaboration among people with the same skill set - Example: data
modelers work with other data modelers
*Official transitions and likely disagreement across skill sets - Example: ETL
developers are given source to target mappings when the data modelers
complete the database design and are upset that the design is hard to load

10. Agile BI Maturity Model
Level 3
Frequently and consistently support the team’s self organized culture
- let the team define their own success criteria
- don’t overly proscribe HOW things must be done, for example, a source
to target mapping may be required, but let the team decide how that
information gets captured
Level 2
Determine where there are skill set gaps
- obtain training to fill the gaps and/or
- pair up team members to cross train each other
Have informal lunch and learns to share knowledge and perspectives,
building team communication and trust
Level 1
Team Roles / Skill Sets
Create a 5 – 7 person team with a mix of all the skill sets needed to take
data from a source system and get it into the hands of end users that need
it to make decisions

11. Assess Your Current State
◊ What is your current technical architecture? What
aspects present the biggest challenges to
incremental evolution and change?

12. Change Is…
Avoidable Inevitable

13. Change Is…
◊
◊
◊
◊
◊
◊
◊
◊
◊
◊
◊
◊
◊
Grain of fact table
New type 2 attribute
Change from type 1 to type 2
Multi-purpose column or table
Redundant data
Tables with too many columns or rows
“Smart” columns
Complex ETL objects
Large SQL modules
Unconformed Dimensions
Indiscriminate use of materialized views
Underutilization of materialized views
Overreliance on documentation

14. Agile BI Maturity Model
Level 3
*Clear understanding of data’s business value
*Clear understanding of the purpose for each component of the
technical architecture
Level 2
*Active effort to clarify understanding of data’s business value
*Streamlined architecture where possible
*Process to deprecate unused components
Level 1
Technical Architecture
*Numerous (possibly) redundant layers (staging, ODS, EDW,
data marts, etc.)
*Inclusion of data with no clear business value
*Lingering tables, reports, ETL scripts, with no known purpose

15. Agile BI Maturity Model
Level 3
Keep it up! Don’t let complexity creep in. If there is a choice
between refactoring an existing column and adding a new one
that is almost identical – choose refactoring!
Level 2
*Create a central repository for the entire organization with
clear, consistent definitions of all major business data
*Get rid of things that are no longer being used
Level 1
Technical Architecture
*Identify redundancy – are multiple tables capturing very similar
information? Do multiple databases get populated with minor
transformations between them? Is there a way to combine or
streamline things?

16. Assess Your Current State
◊ Do you follow technical practices that can enable
agility?

17. Agile BI Maturity Model
Level 3
*End-to-end use of optimal engineering practices
*Team self-enforces usage through criteria for completing work
Level 2
*Some configuration management (SQL scripts to create all db
objects are under CM, but not ETL and report information)
*Some automation is in place (perhaps to promote new objects
or code to another environment or to test ETL)
Level 1
Engineering Practices
*Building blocks of the system (db create scripts, ETL packages,
report files, etc.) are not maintained in any central location nor
are they under configuration management
*Files are manually copied from one environment to another

18. Agile BI Maturity Model
Level 3
*Hold your team to adding automated tests for everything new
and work to reduce the technical debt incurred for existing things
that do not have automated testing
Level 2
*Start creating automated tests. Create test data for an existing
or new ETL process and have scripts that check the load for
expected results. Add this for all ETL over time.
Level 1
Engineering Practices
*Start putting files into a configuration management system
*Work out the kinks of deploying new and changed tables, ETL
code, etc. to another environment

19. Select an Agile Methodology
Scrum
Kanban
BDD
RUP
XP
And so on…

20. Have a Kick Off
PEANUTS © 2013 PEANUTS Worldwide LLC Image from http://whodeyfans.com/2011/08/01/tis-the-season-for-charlie-brown-kicking-the-football/

21. Inspect and Adapt

22. Upcoming Meetups
◊ Wednesday, December 11, 2013
– Anne Marie Macek, Senior Manager – eCommerce, Marriott
– Ecommerce Data and Web Analytics
◊ Wednesday, January 15, 2014
– Eric Nelson, Managing Principal, Synaptitude
– Facilitated discussion on up front data behavior
understanding and cleansing using agile
approach/methodologies
◊ What are you interested in hearing more about?

23. Contact Information