White Paper - Data Warehouse Governance

Data Management & Warehousing

WHITE PAPER

Data Warehouse Governance
DAVID M WALKER
Version: 1.1
Date: 06/04/2007

Data Management & Warehousing

138 Finchampstead Road, Wokingham, Berkshire, RG41 2NU, United Kingdom

http://www.datamgmt.com

White Paper - Data Warehouse Governance

Table of Contents

Table of Contents ...................................................................................................................... 2
Synopsis .................................................................................................................................... 3
Intended Audience .................................................................................................................... 3
About Data Management & Warehousing ................................................................................. 3
Introduction................................................................................................................................ 4
What is governance?................................................................................................................. 5
What affects the data warehouse? ............................................................................................ 6
What is affected within the data warehouse? ............................................................................ 7
How often will change occur?.................................................................................................... 9
What organisational structures are needed?........................................................................... 10
Executive Sponsor(s) .......................................................................................................... 10
Steering Committee ............................................................................................................ 11
Programme Management ................................................................................................... 11
Implementation Teams........................................................................................................ 12
Exploitation Teams.............................................................................................................. 12
User Forums ....................................................................................................................... 12
Certification Committee....................................................................................................... 13
Project Development Methodologies....................................................................................... 14
Waterfall .............................................................................................................................. 14
Iterative ............................................................................................................................... 14
Agile .................................................................................................................................... 15
Cowboy Coding................................................................................................................... 16
Best practices .......................................................................................................................... 17
Momentum .......................................................................................................................... 17
Monitor and Measure .......................................................................................................... 17
Prioritise .............................................................................................................................. 17
Light touch........................................................................................................................... 17
Communication ................................................................................................................... 17
Standards............................................................................................................................ 18
Track processes.................................................................................................................. 18
Understand cost and value ................................................................................................. 18
Continuous Learning and Improvement .............................................................................. 18
Pilot & Prototype ................................................................................................................. 18
Authority to act .................................................................................................................... 18
Plan for the long term.......................................................................................................... 19
Governance and Success ....................................................................................................... 19
Summary ................................................................................................................................. 20
Appendices.............................................................................................................................. 21
Appendix 1 - Programme or Project?.................................................................................. 21
Appendix 2 – The "Declaration of Interdependence" .......................................................... 22
Appendix 3 - Team Values and Principles .......................................................................... 23
References .............................................................................................................................. 24
Web resources .................................................................................................................... 24
Acknowledgements ................................................................................................................. 24
Copyright ................................................................................................................................. 24

© 2007 Data Management & Warehousing Page 2


Synopsis
An organisation that is embarking on a data warehousing project is undertaking a long-term
development and maintenance programme of a computer system. This system will be critical
to the organisation and cost a significant amount of money, therefore control of the system is
vital. Governance defines the model the organisation will use to ensure optimal use and re-
use of the data warehouse and enforcement of corporate policies (e.g. business design,
technical design and application security) and ultimately derive value for money.

This paper has identified five sources of change to the system and the aspects of the system
that these sources of change will influence in order to assist the organisation to develop
standards and structures to support the development and maintenance of the solution. These
standards and structures must then evolve, as the programme develops to meet its changing
needs.
1
“Documentation is not understanding, process is not discipline, formality is not skill”

The best governance must only be an aid to the development and not an end in itself. Data
Warehouses are successful because of good understanding, discipline and the skill of those
involved. On the other hand systems built to a template without understanding, discipline and
skill will inevitably deliver a system that fails to meet the users’ needs and sooner rather than
later will be left on the shelf, or maintained at a very high cost but with little real use.

Intended Audience
Reader Recommended Reading
Executive Synopsis and Summary
Business Users Entire Document
IT Management Entire Document
IT Strategy Entire Document
IT Project Management Entire Document
IT Developers Entire Document

About Data Management & Warehousing
Data Management & Warehousing is a specialist consultancy in data warehousing based in
Wokingham, Berkshire in the United Kingdom. Founded in 1995 by David M Walker, our
consultants have worked for major corporations around the world including the US, Europe,
Africa and the Middle East. Our clients are invariably large organisations with a pressing need
for business intelligence. We have worked in many industry sectors but have specialists in
Telco’s, manufacturing, retail, financial and transport as well as technical expertise in many of
the leading technologies.

For further information visit our website at: http://www.datamgmt.com

1
Agile Software Development, Cockburn, A., Addison-Wesley, 2002



Introduction
Data warehouse governance defines the model the organisation will use to ensure optimal
use and re-use of the data warehouse and enforcement of corporate policies (e.g. business
design, technical design and application security).

Governance structures must be in place as early as possible, ideally before the project starts
and before development begins. They are implemented in parallel with the initial development
to ensure that they work together from the outset.

A data warehouse is, by definition, a system that interacts with large parts of the organisation.
At a technical level this involves the number of systems feeding data to and being fed data
from the data warehouse. At a human level it relates to the people who interact with the
system. These include those querying the system, those operating the technical environment
and those directing the business who may not use the system directly but will mandate that
others do so.

The published literature on the subject falls into three categories:

• Home grown – what a specific project did within a specific environment
• Vendor specific – what to do when using the vendors tool set
• System Integrator proprietary – how an SI say they will do it if they are given the
contract

The problem is that the data warehouse interacts with and across so much of the
organisation. A programme must deploy standards and processes that integrate into the
organisations’ existing policies and procedures.

This paper looks at the areas that must be covered by governance and discusses some of the
standards, options and issues. These ideas should be incorporated into the governance of the
data warehouse.



What is governance?
In the introduction governance is defined as ‘the model the organisation will use to ensure
optimal use and re-use of the data warehouse and enforcement of corporate policies’, but
what does it mean?

Data Warehouses are not a single short-term project. They are long-term programmes of
work that includes a number of projects and a significant amount of maintenance. The
decision to deploy a data warehouse commits the business to a programme that needs to be
controlled to ensure that it provides business benefit and value for money.

The role of governance is to provide the policies, processes and procedures necessary to
ensure that the programme of work is effective. These must be clearly communicated to
everyone involved. Governance covers the integrated management of the programme from its
initial development, through production running to end-of-life close down. The governance
structures must also be reviewed over the lifetime of the programme. This is to ensure that
the governance is neither too little to allow effective control or so much such that programme
work is stifled.

Different organisations also have different requirements for the level of detail and breadth of
scope that should be covered by Data Warehouse governance. For example, does
governance imply coding standards? For some people this is far too much detail, for others it
is essential that the governance of the programme ensures that specific standards are
developed and that staff adhered to them.

The key to designing a suitable level of governance within an organisation is in understanding
the following:

• What events affect the data warehouse environment?

• What is affected in the data warehouse by the impacting event?

• What type of organisation is needed to support the data warehouse?

• What are the policies and procedures needed to control the data warehouse?

From this it is possible to develop the standards, processes and communications that will both
drive the programme forward and culturally fit the specific organisation.

The processes, used in conjunction with the standards, are the way in which decisions are
made. Good processes enabling quick, effective and well-communicated decisions and
consequently effective management of the solution. The converse is also true as poor
standards and processes lead to delays and unresolved issues that discredit the data
warehouse and create significant cost or overheads that eventually destroy the programme.



What affects the data warehouse?
Data Management & Warehousing has identified five types of change that impact a data
warehouse:

• Demand

Demand is the change needed to the system to give the users what they want. This
may include loading new data, possibly from new sources or restructuring existing
information. It can also include adjusting system availability or the time at which data
is loaded or any other user requirement.

• External Change

A data warehouse has a large number of systems feeding it. The systems operate in
a complex and integrated environment. For example, a data warehouse might have
three source systems and each source system is only allowed to perform upgrades
and patches once a quarter. As a result the data warehouse is faced with twelve
changes a year or one a month. In an ideal environment these changes will be
developed and regression tested in line with the change in the source system. In
practice most data warehouses have many more sources and more frequent
changes. When these changes are not communicated they become issues.
Managing external change prevents issues and therefore reduces ‘emergency fixes’
and ‘tactical solutions’

• Maintenance

All computer system have to manage routine maintenance issues such as software
patches, upgrades to software or hardware, special backups for year end, etc. It is
easy to ignore upgrades - the system is currently working so why bother? There
comes a point when the software or hardware vendor moves the product out of
support. At that point any system failure can have critical consequences. Routine
maintenance should be regarded as a critical part of “business as usual” and
allowance made for its impact.

• Risks

Risks to the data warehouse can be identified by those in control. These risks need
to be addressed before they affect the system. Over recent years typical risks have
included changes in regulatory policies, Year 2000 issues, currency changes (e.g.
2 3
revaluations, moving to the Euro or removing trailing zeros ). There are also risks
from corporate mergers and acquisitions and from the bankruptcy of other
companies that provide data to supplement the data warehouse (e.g. with market
share information). These risks will create significant work with tight deadlines.

2
Currently 13 countries are in the Euro zone
(http://en.wikipedia.org/wiki/Eurozone#Official_members)
3
49 countries including Turkey, Greece, Israel, Brazil and Argentina
(http://www.allaboutturkey.com/ytl.htm)



• Issues

In an ideal world all of the above would ensure that the data warehouse would never
have any issues. The reality is a continuous flow of low-level problems and
occasionally critical issues that need to be handled. In some ways this can be a
measure of the success of the solution that people care enough to report issues and
4
need support. There are cost and resource implications in handling these issues.

What is affected within the data warehouse?5
Once a data warehouse development has begun these areas can be affected:

• Requirements Catalogue

All data warehouse projects should have a comprehensive catalogue of
requirements. These will be developed at the start of the system and maintained
when new or changed requirements emerge. Requirements are not the same as the
functionality of the system. This is because not all requirements may be met,
however they provide a catalogue of work that the business needs to be delivered.
This also provides a check that new demands have not already been met by another
means.

• Technical Infrastructure

The technical infrastructure is the hardware, software and network used to build the
system. It can be affected by many different factors including upgrades for better
performance, platform re-hosting or a re-architecting exercise (e.g. moving from
desktop to web-based clients)

• Configuration Management

Configuration management covers all aspect of managing the developed code and
any system configuration files that exist outside the code. Suitable tools must be
used to ensure that the people responsible for the maintenance of the system can
look at all the different versions of code over time. This enables the deployment of
new code. It also can help when there is a need to roll back changes, for example
reverting to older code after a failed upgrade or to read an old format source file. It is
also important that the code can be related to data models for the data warehouse
and to the code versions and data models of the external systems that either supply
or use data from the data warehouse.

4
One Data Management & Warehousing client organisation with 2000 users and a 20Tb
system raised 10,000 issues in one year post implementation – or about 40 per day
5
This section refers to various documents such as requirements etc. Depending on who has
developed the system they will have their own names and templates for these documents.
For consistency this document refers to the templates and tools used in the Data
Management & Warehousing (http://www.datamgmt.com) Documentation Roadmap which
can be found at: http://www.datamgmt.com/index.php?module=article&view=77



• Test Management

Test management is about asking questions to ensure that the impact of any change
is well understood and that preventable errors are avoided. Questions like:

o After a change is made, how is it tested?
o Is there sufficient data to be considered a representative sample?
o Is the whole system regression tested, or are changes isolated and therefore
separately testable?
o How can changes or testing be modified so that tests can be isolated?
o How long does it take to test a change and will this impact emergency fixes to
the system?
o Does the change affect the performance or the data quality of the solution?
o Can the figures be calculated by an alternative independent method to
ensure that the numbers produced are correct?
o Does the project have sufficient resources (e.g. size of machine, people,
time, etc.) to perform the tests?

• Data Stewardship

Data stewardship is about understanding who is responsible for the data within the
system. Again the role is about asking questions:

o Who decides which data quality rules should be applied?
o Where is data cleansed?
o Who is responsible for cleaning the data?
o Can systems or processes be changed to improve the quality of the data
before it is sent to the data warehouse?
o Is data cleansed in the source system and if so is this work also done on
historical data?
o What is the impact if historical data that is already in the data warehouse is
cleansed in the source?
o What is the relationship between time and data e.g. is 95% of the data
available after one day good enough or must 100% of the data be there
before it can be used?

• Service Level Agreement

The service level agreement defines much about the performance and availability of
the system. It will include the times set aside for tasks such as backups and
maintenance. It also defines the times that the system will be available to the users.
It should also include agreements on how to respond in case of emergencies and
how to manage systems failures. The Service Level Agreement also defines how any
change that affects the system (either positively or negatively) needs to be
communicated to the user community.

• Support Model

The support model describes the processes and escalation paths for any user
support request. This model must be updated to reflect any changes to the system or
any changes in the organisation. The support model should be modified to improve
the responsiveness of support by understanding the frequently asked questions and
improving the response time explicitly on these items.



• Training Plans

Users and operators will need to be trained to use the system when it is first
deployed. There is need for on-going training as the system changes, when new staff
arrive or when parts of the solution are either commissioned or de-commissioned.
Users will also need refresher courses to reduce the number and cost of calls to the
support desk.

• Schedules

The system will have an operational schedule that will determine not only when
users can be on the system (also covered in part by the service level agreement).
The schedule defines which jobs to extract, transform or load data can be run, when
they are run and the dependencies between them.

• Monitoring

The system must be monitored and alerts directed to some function that can
prioritise them, act and if necessary escalate appropriately. The process by which
this is managed and its efficiency is critical to providing a viable service to the users.
Monitoring also ensures that service level agreements are met. Analysing the system
events allows technical support to improve the system in the same way that
analysing support calls assists the support team improve the support. The analysis
must look for ways in which to bring about significant improvement to the system by
either changing the system or improving the response to frequently occurring events.

How often will change occur?
The ten areas affected by change and the five types of change described above result in the
table below along with the likely frequency of any given type of change on a specific area of
the data warehouse:
External Change

Maintenance
Demand

Issues
Risks

Requirements Catalogue 3 1 1 2 1
Technical Infrastructure 2 1 2 1 1
Configuration Management 1 3 3 1 3
Test Management 2 3 3 1 3
Data Stewardship 2 3 1 1 3
Service Level Agreement 2 2 2 2 1
Support Model 1 1 2 1 3
Training Plans 1 1 1 1 3
Schedules 2 3 2 1 2
Monitoring 1 3 3 1 1

1: Infrequently 2: Occasionally 3: Frequently

This table can be used throughout the life cycle of the programme to prioritise governance
development to ensure that the most frequent changes are under the tightest controls.



What organisational structures are needed?
This paper has identified a large amount of potential change from different sources on
different processes on a long-term programme. It is necessary to have sufficient
organisational structure in place no control the changes and the communication of that
change. The basic model organisation for this can be described as follows:

Figure 1 - Data Warehouse Organisational Structure

Executive Sponsor(s)
An executive sponsor is a senior (executive) member of the organisation with an active
interest and an understanding of the business uses of the data warehouse. A system of
this magnitude and with such cross-functional reach needs to be sponsored at the very
highest level. It is likely that the executive sponsor for the solution as a whole will be
committed to the idea of developing better strategic information. A number of executive
sponsors for individual functional developments will assist this person. The sponsors of
individual functional developments are responsible for the delivery of the benefits
promised for any given development.

The executive sponsor(s) also need to ensure that the steering committee is directing
the development in line with the vision, strategic direction and priorities for the
organisation.



Steering Committee
The steering committee is the hub of the on-going data warehouse solution from a
management perspective. It is here that the development is aligned with the business
objectives. Monitoring ensures that the programme is delivering the right projects at the
right time and at fair value.

By setting the principles and policies the steering committee can control the direction
that the development goes in. This maintains an enterprise wide business perspective
for the data warehouse.

The steering committee is also the centre of communication. It takes input from the
user forums and the certification committee as to what is needed. In return the
committee manages the expectations of both the business and IT departments as to
what is possible. The steering committee is most effective when it is empowered to
make quick, effective, cross-functional decisions and communicate to all the
stakeholders.

Programme Management
Programme management is the co-ordinated management of a portfolio of projects to
achieve a set of business objectives. It delivers the co-ordinated support, planning,
prioritisation and monitoring of projects to meet changing business needs.

To achieve the business objectives the programme manager defines a series of
projects with quantifiable benefits that together will meet the long-term objectives of the
organisation. These projects may run simultaneously or at least overlap with each other
and they may share resources. Such projects might have some resources devoted to
the project for a period of time. In addition projects will require a range of specialists
available from the programme team whose services are used for shorter periods of
time. Projects within the programme can also be linked. Delays within one project will
then cause knock on effects in other projects due to logical links between tasks in both
of them. Resource and timing conflict resolution is also an integral part of the function
of programme management.

The programme is usually reflected in the management structure with a programme
manager to whom the project managers will report. The programme manager will be
concerned with recruiting and maintaining their project management teams, allocation
of key, shared resources and on the integration of the deliverables from each project
into the overall programme.

In this environment every project plays its part towards the organisation’s ultimate aims
and objectives. Often, as projects are completed, this translates back into a revised set
of corporate objectives.



Implementation Teams
The implementation teams are those that will actually do the work. They will consist of
a team of people (either for some or the entire project) that will fulfil the following roles:

• Project Manager
• Technical Architect
• Systems Administrator
• Network Administrator
• Database Administrator
• Metadata Administrator
• Data Modeller
• ETL Developer
• Front End Tool / Report Developer
• Product Specialist
• Test Manager
6
Many organisations may have outsourced some or all of this functionality . The
resources may have the skills and the time to fulfil more than one role, or for some
projects more than one resource may be required to perform a given role. The
management and resource levels must be determined by the project scope.

Exploitation Teams
Whilst the implementation teams focus on building the system the exploitation team are
trying to ensure that the business is extracting the most value from the solution.
Exploitation teams work on the current version of the system to help the business use
the current system and develop new requirements to exploit the system further. Typical
roles for the teams will include:

• Exploitation Team Leader
• Business Analyst
• Business Requirements Specialist
• Technical Author / Documentation Specialist
• Trainer
• End User Support Specialist
• Communications Specialist
• Helpdesk Support Specialist

As with the implementation teams the resource level is determined by the project
scope.

User Forums
The programme should also consider setting up a number of user forums that involve
end users, subject matter specialists and staff from the exploitation teams. These
forums are useful to allow various teams to express their issues and aspirations for the
system and expand on the art of the possible. These forums should also appoint
representatives on the steering committee to represent their perspective on the system.

6
Data Warehouses benefit from people with detailed knowledge of the subject and
outsourced development to coding shops with no experience is often more costly than the
initial perceived savings. When choosing resources Data Management & Warehousing
recommend that organisations look at the skills of named individuals on the project



Certification Committee
A number of groups within the organisation will also assess the data warehouse to
ensure that it is fit for purpose. These groups can either be consulted individually or
brought together as a committee to advise the programme. They include:

• Audit

Most organisations will have some internal audit function. The audit function
will need to ensure that the system meets the required standards and has
sufficient checks and balances especially if the system is used for any form or
statutory or fiscal reporting. The internal audit function should also examine
how the system is managed. Where there is no (required) internal audit
function an individual or team representing those with a stake in the quality of
the information should perform the role of auditor.

• Regulatory Compliance

Depending on the organisation and industry there may be a need to comply
with requirements from government, local administration or industry regulators
over the data that is held and who has access to that data. Current examples
7
include data protection law , financial management laws such as Sarbanes-
8 9 10
Oxley Act and Mifid and telecoms regulators such as Ofcom .

• IT Strategy & Architecture

Many large organisations have a team responsible for the strategy and
architecture of all IT systems. This team ensures inter-operability, re-use and
cost reduction of IT systems. The team will need to review and assess any
technical infrastructure and changes to the solution that are proposed.

• Security

Security of information is a growing concern for many organisations. Either a
security team, if one exists in the organisation, or a nominated individual
should review the system periodically to ensure its security.

7
DPA Law website: http://www.dpalaw.info/
8
Sarbanes Oxley Act: http://en.wikipedia.org/wiki/Sarbanes-Oxley_Act
9
Mifid: http://www.fsa.gov.uk/Pages/About/What/International/EU/fsap/mifid/index.shtml
10
Ofcom website: http://www.ofcom.org.uk/



Project Development Methodologies
There are four main methodologies for the development itself:

Waterfall
The waterfall model has the following phases that are followed in order:

• Requirements specification
• Design
• Build
• Integration
• Testing
• Deployment
• Maintenance

To follow the waterfall model, one proceeds from one phase to the next in a purely
sequential manner, starting each phase once the previous one has been completed.
Phases of development in the waterfall model are thus discrete and there is no jumping
back and forth or overlap between them, however there are practical variations on this
model.

Iterative11
The basic idea behind iterative development is to develop a software system
incrementally. This allows the developer to take advantage of what was being learned
during the development of earlier, incremental, deliverable versions of the system.

Learning comes from both the development and use of the system where possible. Key
steps in the process are to start with a simple implementation of a subset of the
software requirements. The system is then iteratively enhanced in a sequence of
evolving versions until fully implemented. At each iteration design modifications are
made and new functional capabilities are added.

The process itself consists of the initialization step, the iteration step and the project
control list. The initialization step creates a base version of the system. The goal for this
initial implementation is to create a product to which the user can react. It should offer a
sample of the key aspects of the problem and provide a solution that is simple enough
to understand and implement easily.

To guide the iteration process, a project control list is created that contains a record of
all tasks that need to be performed. It includes such items as new features to be
implemented and areas of redesign of the existing solution. The project control list is
constantly being revised because of the analysis phase.

The iteration step involves the redesign and implementation of a task from the project
control list and the analysis of the current version of the system. The goal for the design
and implementation of any iteration is to be simple, straightforward and modular,
supporting redesign at that stage or as a task added to the project control list.

11
Description is an edited form of the text from Wikipedia:
http://en.wikipedia.org/wiki/Iterative_development



The code can represent the major source of documentation of the system. The analysis
of an iteration is based upon user feedback and the programme analysis facilities
available. It involves analysis of the structure, modularity, usability, reliability, efficiency
and achievement of goals. The project control list is modified in light of the analysis
results.

Agile12
Agile methods are a family of development processes that build on iterative
development, not a single approach to software development. Agile evolved in the mid
1990s as part of a reaction against "heavyweight" methods such as the waterfall model
of development that often became heavily regulated, regimented and micro-managed.
The processes originating from this use of the waterfall model were seen as
bureaucratic, slow, demeaning and inconsistent with the ways that software engineers
13
actually perform effective work.

In 2001 seventeen prominent figures in the field of agile development (then called
"light-weight methodologies") came together to discuss ways of creating software in a
lighter, faster, more people-centric way. They created the Agile Manifesto and
14
accompanying agile principles:

• The highest priority is to satisfy the customer through early and continuous
delivery of valuable software.

• Welcome changing requirements, even late in development. Agile processes
harness change for the customer's competitive advantage.

• Deliver working software frequently, from a couple of weeks to a couple of
months, with a preference to the shorter timescale.

• Business people and developers must work together daily throughout the
project.

• Build projects around motivated individuals. Give them the environment and
support they need and trust them to get the job done.

• The most efficient and effective method of conveying information to and within
a development team is face-to-face conversation.

• Working software is the primary measure of progress.

• Agile processes promote sustainable development. The sponsors, developers
and users should be able to maintain a constant pace indefinitely.

• Continuous attention to technical excellence and good design enhances agility.

• Simplicity--the art of maximizing the amount of work not done--is essential.

• The best architectures, requirements and designs emerge from self-organising
teams.

• At regular intervals, the team reflects on how to become more effective, then
tunes and adjusts its behaviour accordingly

12
Taken from The Agile Manifesto (http://agilemanifesto.org/)
13
Source: Wikipedia - http://en.wikipedia.org/wiki/Agile_software_development
14
Source: Wikipedia - http://en.wikipedia.org/wiki/Agile_software_development



Cowboy Coding15
Cowboy coding is a form of software development method without an actual defined
method – team members do whatever they feel is right. Typical cowboy coding will
involve no initial definition of the purpose or scope of the project, no formal description
16
of the project and will often involve only one developer . The developer will often be
working from his own idea of what the software should do. It is often characterised by a
lack of any documentation for either the requirements of the project or the design of the
software overall.

Within a data warehouse programme it is strongly advised never to allow the cowboy
methodology to appear, however any of the other three can be used. There are two deciding
factors as to the approach that will be used. The first is the approach that is already being
used within the organisation and this may dominate over all other factors. The second is the
point in the lifecycle of the warehouse. In the early part of the programme projects using agile
methodologies offer fast development and quick delivery for new applications built by expert
teams that engage the users. Towards the end of the lifecycle a more waterfall like approach
to maintenance and de-commissioning operations run by systems management teams is
likely to be more successful for individual projects. This highlights the need for the
governance itself to be adaptive to the changing environment.

All formal methodologies (lightweight and heavyweight) can still lead to failure as the
methodology team members attempt to operate within the social/political environments of the
organisation. The probability of such a failure is related to the degree of processes inhibiting
the user(s) from deviating from the organisation standard on one hand and the cost in terms
of lost efficiency and lost creativity of implementing such processes on the other. Success is
therefore reliant on the management choosing appropriate processes that find the correct
balance between these two competing aspects.

15
Cowboy coding definition: Wikipedia: http://en.wikipedia.org/wiki/Cowboy_coding
16
Cowboy coding can occur even within projects that are using more appropriate
development methods. Warning signals that cowboy coding may be occurring include:
• Secrecy about what a developer is working on.
• The inability to describe the functionality of current development.
• The tendency to do lots of quick fixes.
• Working hard and not working clever, working without prioritisation.



Best practices
The following items are some of the key best practices that should be implemented:

Momentum
Get the project going fast and then keep it moving by keeping the scope and deadlines
for delivery very tight. This avoids analysis paralysis and ensures that there is space to
take corrective actions if necessary.

Monitor and Measure17
Large long-term projects are difficult but if the management team cannot see what is
happening then they do not know when things are going wrong. Ask questions about
whether teams deliver on time, to budget and scope. Track the number of issues raised
in development, test and production, monitor the service level agreements, design
metrics for the quality of the code and the user satisfaction, etc. Much of this is built in
to approaches such as agile.

Prioritise
Everyone will want their component now, but it cannot all be done at once. Have a
clear, transparent process for deciding the priories and then stick to them. One useful
technique is to say that (provided the project uses small scopes) once started a phase
cannot be stopped but that before the next phase is started all priorities can be re-
assessed. This forces low value priorities to the back but rapidly brings higher priorities
to the fore. The steering committee should be responsible for these priorities and the
project methodology should support the prioritisation process.

Light touch
This document has covered many aspects of governance however the programme
should implement the minimum necessary because otherwise the project will become
about governance and not delivery. Review the processes regularly to add process
where required and do not be afraid to remove un-necessary process.

Communication
The data warehouse is a long-term project for which the perception will shift over time
from all parts of the business. Clear communications will help people understand what
is available for them, what the priorities are going forward and how to access the
information. Publish these widely to encourage involvement and interaction with the
system.

17
A useful book on this subject is “Controlling Software Projects: Management, Measurement
and Estimates” by Tom DeMarco



Standards
Develop and enforce clear standards for all aspects of the warehouse such as naming
conventions for code and documents, data ownership, data cleaning, access to data,
18
service levels, performance etc.

Track processes
Ensure that there are clear formal processes for handling change requests, risks and
issues. These are the aspects of the project that cause divergence from the baseline
and therefore it is critical to know what is happening, when and why. It allows scarce
resource to be focused on the correct actions. Use key design decision templates to
ensure that design topics are not constantly revisited. Ensure that all change related
processes have powerful change management routings to back them up.

Understand cost and value
Define methods for determining the cost and value of any action, some apparently low
cost (often called quick-win or tactical) solutions deliver little value and their true cost is
disproportionately higher as they have to be backed out after a short life. Do not forget
to include end of life cost in the calculations and make budget holders responsible for
demonstrating the benefit that has been derived because of the effort.

Continuous Learning and Improvement
Carry out stage point audits and post-mortems on all aspects of the system, use the
lessons learnt from these to improve the processes to ensure that the next
development will not make the same mistakes.

Pilot & Prototype
Pilot and prototype high risk or complex aspects of the development using
methodologies like ‘Agile’ but ensure that the pilots and prototypes do not go into
production until the normal quality thresholds have been met.

Authority to act
Ensure that the steering committee has the authority to act cross functionally using
sufficient information to make good decisions. A steering committee that wants a
change to a source system that has a critical effect on the data quality is ineffective if
they are told that it will be put in a queue and should be delivered in two years time. In
the mean time is it acceptable that the data warehouse will just have to make do with
poor data? Conversely a lack of sufficient information might be making the same
demand of a system that is about to be de-commissioned or for data for which the
business user cannot articulate a use.

18
See the Data Management & Warehousing Documentation Roadmap at
http://www.datamgmt.com



Plan for the long term
Ensure that the business and users understand that whilst aspects of the data
warehouse will be available quickly the system is a long-term commitment. It will need
funding and maintenance across its entire life span. During that lifetime the system will
also grow in size and complexity, increasing the financial burden.

Governance and Success
Governance is the control aspect of management and should promote a situation where
senior management are in control of the situation much like the maestro conductor of a fine
orchestra. They need to understand what is going on and where to go for the next action and
how to respond to a changing situation.

Sometimes the reality is that managers are working in a suppressed panic, not believing what
people are telling them or what they themselves are communicating to their management.
When managers know that people have to work outside the process to get things done and
not knowing what is actually going on then governance has failed.

Governance is therefore not the documentation, the processes or the formality, but is about
developing a culture where understanding, discipline and skill are regarded as virtues in
teams that have leaders with strong technical skills, initiative, communications skills and
personal authority.

Organisations that ‘buy the book’, be it from a bookshop or as a formal methodology from a
vendor and follow it rigidly are guaranteed to achieve the lowest common denominator
solution, one that checks all the boxes but underwhelms the management and disappoints the
users. Those organisations that use governance and methodologies as enablers and allow
systems to fulfil their potential by meeting the constantly changing requirements of the users
succeed.

Creating effective governance for an organisation requires imagination:

“What we need is imagination, but imagination in a terrible strait-jacket. We have to find a new
view of the world that has to agree with everything that is known, but disagree in its
predictions somewhere .... And in that disagreement it must agree with nature. If you can find
any other view of the world which agrees over the entire range where things have already
been observed, but disagrees somewhere else, you have made a great discovery. ...A new
19
idea is extremely difficult to think of. It takes a fantastic imagination.”

19
Richard Feynman, The Character of Physical Law, 1965, Chapter 7, "Seeking New Laws.”



Summary
An organisation that is embarking on a data warehousing project is undertaking a long-term
development and maintenance programme of a computer system. This system will cost a
significant amount of money and should be well controlled. Governance defines the model
that the organisation will use to ensure optimal use and re-use of the data warehouse. It will
also enforce corporate policies (e.g. business design, technical design and application
security) that ultimately derive value for money.

This paper has identified five sources of change to the system:

• Demand
• External Change
• Maintenance
• Risks
• Issues

It has also looked at the aspects of the system that they will affect:

• Requirements Catalogue
• Technical Infrastructure
• Configuration Management
• Test Management
• Data Stewardship
• Service Level Agreement
• Support Model
• Training Plans
• Schedules
• Monitoring

Using these it has been able to highlight standards and best practices that should be
employed by the organisation to deliver effective governance. The paper has also highlighted
the fact that because governance is about fitting into the existing organisation. It also
highlights the fact that the system grows significantly over time. Consequently there is no
single right solution for governance but instead there are ways of working and adapting over
time that will ensure success.



Appendices
Appendix 1 - Programme or Project?
This document has several times differentiated between programmes and projects. The
20
table below outlines the differing characteristics of the two:

Programmes: Projects:
Address the entire business change Deliver a specific change component
Focus on strategic goals Focus on tactical delivery
May have imprecise definition Have a precise objective
May have uncertain timing Are defined with a specific timeline
and budget
Evolve over a period of time to derive Try to avoid change to the defined
optimum benefit for the organisation scope in order to ensure delivery
Require much senior management Require management communication
attention, often including strategic and primarily at an operational level
political debate across organisational concerning operational details
boundaries
Produce an overall improvement in the Produce specific pre-defined
business that may be multi-faceted and deliverables
not fully defined at the outset of the
programme
Require a manager who is high- Require a manager who pays attention
powered, high-level, visionary, strategic, to detail, has good team leadership,
political, sales-oriented and works with plans in detail, follows a disciplined
people at the top and across the approach and delivers the goods.
organisation

20
The ePMBook by Simon Wallace (http://www.epmbook.com/)



Appendix 2 – The "Declaration of Interdependence"
Throughout this document reference has been made to the agile development
methodology. Because of the development of agile, a set of principles characterised by
statements “We accomplish X by doing Y.” was developed for project managers. These
principles are of value outside the agile methodology itself and should be used by
project managers in any data warehouse programme.

The “Declaration of Interdependence” for modern (agile/adaptive) (product/project)
21
management

"We ...

• increase return on investment by making continuous flow of value our focus.
• deliver reliable results by engaging customers in frequent interactions and
shared ownership.
• expect uncertainty and manage for it through iterations, anticipation and
adaptation.
• unleash creativity and innovation by recognizing that individuals are the
ultimate source of value and creating an environment where they can make a
difference.
• boost performance through group accountability for results and shared
responsibility for team effectiveness.
• improve effectiveness and reliability through situationally specific strategies,
processes and practices."

21
2005 David Anderson, Sanjiv Augustine, Christopher Avery, Alistair Cockburn, Mike
Cohn, Doug DeCarlo, Donna Fitzgerald, Jim Highsmith, Ole Jepsen, Lowell Lindstrom,
Todd Little, Kent McDonald, Pollyanna Pixton, Preston Smith and Robert Wysocki:
http://alistair.cockburn.us/index.php/The_declaration_of_interdependence_for_modern
_management



Appendix 3 - Team Values and Principles
The author of this white paper used to work at Sequent Computer Systems. The
company had a set of values and principles that the author feels have always proved
useful to teams building data warehouses by creating a culture in which governance
can work.

Our Values:
• Easy to Do Business With
o The Customer ALWAYS Comes First
o Do All We Can to Ensure Customer Success - Does Not Always
Mean Saying Yes
o Three Keys -- Listen, Consider and Act
o Go the "Extra Mile" for the Customer

• Profitability
o Required for Our Success
o Allows for Growth & Investments for Future
o Everyone is Responsible for Profitability
o Follow the "Spend Smart and Invest Smart" Concept

• Teamwork
o Strong & Balanced Teams
o Put Group Goals & Interests Ahead of Personal Reward
o Open & Honest Communication
o You're Accountable for Your Commitments
o Nobody Wins Unless the Team Wins

• Quality
o Exceed the Customer Expectations
o Strive for Continuous Improvement in All You Do
o Know Who Your Customers Are
o Establish Clear Mutually Acceptable Expectations
o Managed Expectations, Consistently Achieved

• People
o Assimilating & Integrating New Members is Critical
o Open, Honest & Timely Communication
o Treat Each Other With Respect
o Take Time to Say "Thank You" for a Good Job
o Strive for Win-Win Relationships

Our Principles:
• Act With Honesty & Uncompromising Integrity
• Take Responsibility for Our Customers' Success
• Strive to be the Best
• Have a "Can Do" Attitude
• Respect Each Other
• Exhibit Team Pride
• Take Calculated Risks
• Be Active Community Citizens
• Urgently Do the Right Thing
• Make Consultative Decisions



References
The section below represents some useful resources for those considering building a data
warehouse solution.

Web resources

Organisation Website
Data Management & Warehousing http://www.datamgmt.com

DM Review – http://www.dmreview.com/
Data Warehouse Project Management
The Data Warehouse Institute – http://www.tdwi.org/
Data Warehouse Governance
Data warehouse governance – http://www.amazon.com/
Best practices at Blue Cross & Blue Shield of North Carolina
Data Warehouse Project Management http://www.amazon.com/
The Seven Pillars of Data Warehouse Governance http://www.itweb.co.za/
The ePM Book http://www.epmbook.com/
Software Testing Resources http://www.testingfaqs.org/
From Here to Agility: The Physics of Speed http://www.stickyminds.com/
Agile Manifesto http://agilemanifesto.org/
Agile Alliance http://www.agilealliance.org/
e-Programme http://www.e-programme.com/
Data Protection Law http://www.dpalaw.info/
Alistair Cockburn http://alistair.cockburn.us/

Wikipedia http://en.wikipedia.com

Acknowledgements
The author and Data Management & Warehousing would like to acknowledge and thank all
the clients and colleagues who have taken the time to review and comment on this document
prior to publication. Special thanks to Ron Ballard for the most detailed of reviews and proof
reading.

Copyright
© 2007 Data Management & Warehousing. All rights reserved. Reproduction not permitted
without written authorisation. References to other companies and their products use
trademarks owned by the respective companies and are for reference purposes only.


White Paper - Data Warehouse Governance

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (18)

Ähnlich wie White Paper - Data Warehouse Governance

Ähnlich wie White Paper - Data Warehouse Governance (20)

Mehr von David Walker

Mehr von David Walker (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

White Paper - Data Warehouse Governance