More Related Content
Similar to Endasol Streamlining Bi Implementation, Euci March 2008 (20)
Endasol Streamlining Bi Implementation, Euci March 2008
- 2. Brief Introduction - Endasol (www.endasol.com)
Focus Areas
Data Warehouses/ BI Efforts
Lead Roles in PJM Enterprise-Wide Data Warehouse prior to forming Endasol
Called back to PJM to help develop a separate Market Monitoring Data Mart
Just underway with helping to identify business requirements and early design for
a major data mart at an ISO.
Business Analysts for a couple different energy companiesâ major data
consolidation & reporting projects
Developed Comprehensive Solution for Wholesale Power P&L + Other KPIs
Systems Integration â ETRM, Scheduling, Settlements, Generator
Management, Fuel Inventory
ISO Operations & Interfaces - PJM, MISO, NYISO, ISO-NE
Developed Streamlined Interfaces to many of the PJM eSuite Applications
Direct Experience in estimating Major Settlement Line Items
Business Process Development/ Refinement
© 2008 Endasol, Inc. All Rights Reserved.
- 3. Streamlining BI Implementation - Topics
Components of Business Intelligence
Methodologies for Streamlining a BI Implementation
Relationship to Business Process Management
© 2008 Endasol, Inc. All Rights Reserved.
- 5. BI System Integration & Information Layers
Data Warehouse
Data movement from Sources to Target Tables
Usually scheduled in batched ETL processes
Staging Tables or Operational Data Store often an intermediate step
Data Model promotes business user reporting & querying â Dimensional
Data Propagation
Data movement among Systems
Usually in Real Time or near Real Time
Use of Enterprise Application Integration (EAI) tools to publish & subscribe to data
Data Warehouse can be a Subscriber
Data Federation
No data movement
Pre-established Views are often constructed to relate data across systems
More suitable for highly decentralized organizations
A sensible approach may involve elements of each method
© 2008 Endasol, Inc. All Rights Reserved.
- 6. BI System Integration Layer
Pre-established Reports
Routine reports designed & developed in advance â often scheduled
Typically âlocked downâ and controlled by a couple of individuals - often IT
May build intermediate tables, views and/or calculated fields to feed reports
Self Service Reports and Queries
Business users should be able to do reporting & querying without the need for IT
Business users do not always know exactly what they want until they start
using the data in ways not previously possible
Requires a sound foundation (typically dimensional data model), data
dictionary or other metadata, and training
OLAP Cubes
OLAP provides interactive analysis and reporting using multidimensional views of
data
Users can slice, dice, drill-down and roll-up data â more than Excel pivots
Cubes are often pre-built and refreshed right after batch ETL processes so that they
are immediately available to users
© 2008 Endasol, Inc. All Rights Reserved.
- 7. BI System Intelligence Layer
The value of the BI System is often not realized until deploying this
Layer
Advanced Visualization, including Dashboards & Scorecards
Enable rapid assessment of Business Metrics or Key Performance
Indicators
Dashboards display information in tables, charts, gauges, and other
indicators
Scorecards display metrics against pre-established thresholds grouped into
red, yellow, green or numerical weightings
Ultimately, should have the capability to drill down into information
feeding the dashboard and scorecard items
© 2008 Endasol, Inc. All Rights Reserved.
- 8. BI System Intelligence Layer
Advanced Analytics
Exploratory Data Mining â seeking new insights through examining the
information with statistical analysis or other techniques
Predictive Analytics â formulating rule-based models to predict what
will happen under various scenarios with high degree of confidence
Many BI efforts are abandoned before reaching this Layer
Why? Because implementation of the other layers have been overly
costly with mediocre results
© 2008 Endasol, Inc. All Rights Reserved.
- 10. Streamlining BI Implementation
Employ methods that get to the higher value BI layers sooner:
Establish Fundamentals Early
Consider using a Packaged Solution or Starter Model
Consider Agile Development Approach
Consider Hybrid Approach
Considerations for the Interactive Layer
Considerations for the Intelligence Layer
© 2008 Endasol, Inc. All Rights Reserved.
- 11. Establish Fundamentals Early for Any Phase
Implement a Data Governance Program Upfront
Assign Data Stewards from the Business responsible for quality and use of data
Establish Monitoring and Methods for Correcting Issues
Consider a Master Data Management Program
Managing your Reference Data, such as customer, generators
Ensuring consistency and dynamic mapping among systems
Some examples of the need:
How many systems or databases house customers (individuals or organizations) and how
many different identifications, abbreviations, or aliasâs are used?
How are you tracking who owns whom? Frequently changes - if not careful, you may
have excess credit exposure to entities or entities may be exercising unfair market power.
Various technologies are available
Ensuring consistency among systems regarding Master Data will streamline BI
implementation, but not absolutely needed since the data warehouse will ultimately
solve this issue for users of the data warehouse
© 2008 Endasol, Inc. All Rights Reserved.
- 12. Establish Fundamentals Early for Any Phase
Set realistic expectations from both business & IT perspectives
Define precise & measurable objectives
Ensure that the BI system will be scalable to accommodate other corporate
initiatives in the future
More importantly, align the project with overall corporate BI objectives
Minimize scope creep
Well-defined business requirements
Establish Change Control policies upfront â know how changes will be
processed and who needs to approve these
If you donât have the expertise in-house, then hire professionals
Ensure they have experience in the phase you are embarking on
Ensure those on the team have experience with your business
Be involved â you still need to âownâ the program & business must be involved
BI systems are not static â you will need skills to advance & maintain your system
If not prepared to adopt these & other fundamentals, then donât start!
© 2008 Endasol, Inc. All Rights Reserved.
- 13. Use of Packaged BI Solutions & Starter Models
Packaged BI Solutions: 3rd party products fulfilling a slice of your BI
needs
A Packaged BI Solution should:
Provide the business with a clear definition of the end results â what you
will get is visible from the start!
Integrate and merge data across all appropriate systems
Enable easy access to underlying data and the information it provides
Serve as a starting point for other analysis and reports
Build on an existing data warehouse as appropriate
© 2008 Endasol, Inc. All Rights Reserved.
- 14. Use of Packaged BI Solutions & Starter Models
A Packaged BI Solution is typically not:
A front line application, such as an Energy Trade & Risk Management
System (ETRM), or a Customer Information System (CIS) - too many
try to elevate these transactional systems to be an enterprise data mart
for reports & analysis
Are there Packaged BI Solutions available in our industry?
For wholesale power, including financial transactions, we have our
TM
PREMIR Solution
There may be solutions available for the Retail, and
Transmission/Distribution Sectors
Consulting Companies focused in your business may offer at least
Starter Models
© 2008 Endasol, Inc. All Rights Reserved.
- 15. Endasolâs Flow Diagram
premir Solutions
TM
Our present focus is the Wholesale Power & Financial Transactions Data Models and Reporting
© 2008 Endasol, Inc. All Rights Reserved.
- 16. Use of Packaged BI Solutions & Starter Models
Advantages:
Requirements/Vision already defined â you see what the end result looks like
Data Model provided and can be built upon
Key Reports may be provided and others can be added
KPIâs & Dashboard may be provided and can be expanded
In any case, they can provide a head start enabling quicker deployment
Potential Pitfalls:
Donât expect a packaged solution or even starter model to meet all of your
objectives â expect that some customization will be needed
Compatibility with existing data warehouse, data marts must be considered
Compatibility of technology used â database, reporting tools â must be
considered
Capability to expand the solution to other areas of the enterprise must exist
Conclusion: A packaged solution or starter model can significantly expedite
your BI implementation, but do your homework first
© 2008 Endasol, Inc. All Rights Reserved.
- 17. Consider Agile Development for BI - Background
Waterfall Methodology â Traditional Approach
From Wikipedia: âThe waterfall model is a sequential software
development model (a process for the creation of software) in which
development is seen as flowing steadily downwards (like a waterfall)
through the phases of requirements analysis, design, implementation, testing
(validation), integration, and maintenance.â
Often characterized by lengthy requirements, design and signoff before any
development is allowed to begin.
Agile Methodology â Currently gaining acceptance
From Ralph Kimball: ââŠIn general, it (Agile) refers to a group of
methodologies, including Extreme Programming, SCRUM, Adaptive
Software Development and others, which share a common focus on iterative
development and minimizing risk by delivering new functionality in short
timeframes, often measured in weeks.â
Less structured with emphasis on deliverables, not documents & signoffs
© 2008 Endasol, Inc. All Rights Reserved.
- 18. Consider Agile Development for BI
Agile Development is gaining interest - Agile is now included in training
classes and conferences
Scrum is one structured methodology for Agile Development
Product/Project broken into deliverables, called âSprintsâ of 2 to 4 weeks.
Sprints established to bring priority, but achievable results within time frame.
Sprints typically locked down with high resistance to changing scope once started.
Product Owner - The Product Owner represents the voice of the client. He/She ensures
that the Scrum Team works with the right things from a business perspective.
ScrumMaster (or Facilitator) - Scrum is facilitated by a ScrumMaster, whose primary
job is to remove impediments to the ability of the team to deliver the sprint goal. The
ScrumMaster is not the leader of the team (as they are self-organizing) but acts as a
buffer between the team and any distracting influences. Empowered teams without
traditional Project Management.
Scrum Team - A small team of 2+ people with cross-functional skills to do the actual
work (designer, developer etc.).
Daily meetings among players â IT & Business.
Development proceeds immediately and documentation often lags.
© 2008 Endasol, Inc. All Rights Reserved.
- 19. Consider Agile Development for BI Implementation
Biggest Issue with many DW/BI projects
Changes during the project!
Source system changes
Requirements
Data quality issues â some sources are just not very good.
Change in business sponsor / business owner
Market changes
ISO rule changes
Subscription services
Business Priorities
© 2008 Endasol, Inc. All Rights Reserved.
- 20. Consider Agile Development for BI Implementation
Advantages of Agile Development
Design at the beginning of each sprint - Learn from each sprint
Phased / iterative approach â Useable tools delivered quicker
Determine next sprint based on priorities.
Working closely with the business users during data cleansing and
development of reports, cubes and dashboards
Increase user satisfaction and âownershipâ
Decrease âdo-oversâ
Building what the business needs overrides building just to meet
documentation
© 2008 Endasol, Inc. All Rights Reserved.
- 21. Consider Agile Development for BI Implementation
Disadvantages of Agile Development
Budgeting difficulties due to minimal upfront analysis & design
Hardware needs
Staffing â how long?
When is the project considered finished?
How does the present sprint fit into the solution?
Some rework may be needed
Have a risk of delivering isolated silos of data which meet individual releases, but
donât integrate well with each other
New to the organization and not well understood by IT and Business.
Users are confused about their role.
Users may not have sufficient time.
Minimize risk by using company standards & following corporate SDLC process
Documentation â Agile methodology minimizes documentation.
What about support and future enhancements?
© 2008 Endasol, Inc. All Rights Reserved.
- 22. Hybrid Approach for BI Implementation
Optimum Approach may be a Hybrid between Waterfall & Agile
Take time upfront to develop overall BI plan/ roadmap
Spend some time on how the overall BI solution fits into the enterprise picture
Identify subject areas for potential phases
Develop framework of data model
Set standards
Select hardware / software
Break into focused phases
Phase 1 may be 4 â 6 months due to infrastructure, overall requirements, design
& documentation
Focus mainly on current phase subject area but know how it fits with the other
areas.
Other phases may decrease in length as more and more subject areas are added.
Take each phase all the way through user reports, OLAP cubes, dashboard
& scorecard
Include support with the delivery of each phase
© 2008 Endasol, Inc. All Rights Reserved.
- 23. Hybrid Approach for BI Implementation
Establish Project High Level Design Infrastructure
âąSubject Area Data âąHardware
âąBI System Objectives
âąSubject Area Relationships âąSoftware/ Licenses
âąBusiness Requirements
âąHigh Level Data Model âąSet Up Environments
âąPrioritize Phases
âąSet Standards
âąProject Plan
Detailed
Requirements
Phase
1
Detailed
Phase
Design
2
Phase
ETL &
3
Test
Repeat Each
Phase
for Phase
each may get 4
User
Phase shorter
Reports
OLAP
Cubes
Dashboard/
Scorecard
© 2008 Endasol, Inc. All Rights Reserved.
- 24. Streamlining Interactive Layer
Choosing Reporting and Querying Tools
Business Ease of Use
Deploying a mixture of tools
What about Excel?
MS Excel will continue to be the tool of choice for many business users
Even with well managed data warehouses, end users still resort to Excel
But this may not be bad as long as the source is the data warehouse and
not direct pulls from source systems
How important is OLAP?
© 2008 Endasol, Inc. All Rights Reserved.
- 25. Streamlining Interactive Layer
Give Business Early Access to Data
Standard reports should be pre-established using the corporate reporting
tool & âLocked Downâ
Underlying structure/ data model should be such that business users can
easily understand
Allow them to develop their own reports and analysis - Stress using
corporate reporting tools, but be flexible as long as they are sourcing
from the data mart
© 2008 Endasol, Inc. All Rights Reserved.
- 26. Streamlining Intelligence Layer - KPIs
Restrict Key Performance Indicators (KPIs) to a small number (6 â
10) specific to the organizationâs goals
Develop different set of KPIs for each business unit, sub-unit
The business unitâs KPIs should relate to corporate KPIs
The business unitâs KPIs should also include whatâs important to bonuses
Most effective use is often a mixture of financial & non-financial
KPIs as in the Kaplan and Norton Balanced Scorecard approach
KPIs should follow the SMART test:
Specific, Measurable, Achievable (Actionable), Realistic, Timely
Once implemented, continue to refine KPIs
Revise as corporate and business unit objectives change
Monitor who is accessing and how often âretire KPIs not being used
Ensure individuals can easily access KPIs and understand what they are
viewing
© 2008 Endasol, Inc. All Rights Reserved.
- 27. Streamlining Intelligence Layer â Visualizing
Choosing Technology
Numerous products available with nice dashboard & scorecard features
Some are limited to specific technologies, such as using MS Analysis
Services (cubes) to drive dashboard components
Most effective is ability to drill down â start with the picture & drill to
underlying information
Donât need to go overboard on dashboard/ scorecard technology
If canât justify expense upfront, then consider publishing output of certain
reports & OLAP such as graphs or charts in a web-portal â this can be
automated
Once base functionality is working, can always buy advanced technology
later
© 2008 Endasol, Inc. All Rights Reserved.
- 28. Streamlining Intelligence Layer â Visualizing
Implementing Dashboards & Scorecards
Keep it simple â donât clutter the viewing screen
Consider allowing viewers to construct their own âmyBIâ page
Some items may be required viewing such as important metrics
Others may be selected from a list of available dashboard reports,
graphs, charts
Continue to monitor usage and effectiveness & continue to refine
© 2008 Endasol, Inc. All Rights Reserved.
- 29. Streamlining Intelligence Layer â Data Mining
Determine Data Mining or Advanced Analytical Methodology
Early
Exploratory â looking for new insights and/ or
Predictive â developing models to predict behavior/ market
opportunities
Different methodologies are available for each focus
Methodology may require very specific data model or structure
Data model used for advanced analytics may conflict with that optimal
for reporting and user-queries
If you know this upfront, you may be able to build a hybrid model that
can serve both purposes, or
You may be able to implement a model structure that can be more easily
transformed into that needed for the advanced analytic application
© 2008 Endasol, Inc. All Rights Reserved.
- 30. Align Your BI System with Corporate Objectives
Ultimately, your BI System should be aligned to corporate objectives
and provide monitoring & feedback for continued alignment
Corporate KPIs should trickle down to corresponding KPIs pertinent
to individual Business Units
Business Unit KPIs should link to corporate & adjusted as corporate
objectives change
The above is part of the broader initiative referred to as Business
Performance Management (BPM), Corporate Performance
Management (CPM), or Enterprise Performance Management (EPM)
There are advancement in software tools which support
implementation & monitoring
© 2008 Endasol, Inc. All Rights Reserved.
- 31. Streamlining BI Implementation - Summary
Align your BI System with Corporate Objectives
Value of your BI System is often not realized until deployment of
Visualization and Advanced Analytics
Dashboards & Scorecards to track KPIs & other important measures
Drill-down into information driving displays is an important capability
Data Mining for discovering new insights
Predictive Analytics to gain edge in market
© 2008 Endasol, Inc. All Rights Reserved.
- 32. Streamlining BI Implementation - Summary
Objective is to get to these value BI Layers as soon as possible:
Use Packaged BI Solutions or Starter Models if available and if
applicable
Use a Hybrid Approach of Agile & traditional Waterfall Development
Do some detailed requirements & design upfront
Break Project into short phases of 1 â 6 months covering selected subject
areas
Each Phase should include all BI Layers - Data Model, User Reports,
Cubes, Dashboard & Scorecard items
Technology â decide whatâs important to you from a budget and
capability perspective, be flexible and donât get stuck in analysis
paralysis.
© 2008 Endasol, Inc. All Rights Reserved.