How CGS collaborates with clients and stakeholders to deliver great learning solutions.

1. Lean & Agile Learning: The CGS Approach
to Delivering Effective Learning Solutions
Elizabeth Woodward
Learning Strategist and Agile Transformation Consultant
ewoodward@cgsinc.ca
Micah White
Director of Research and Development
pwhite@cgsinc.ca
August 17, 2015

2. 2
Contents
L&A Learning: The CGS Agile Approach to Delivering Effective Learning Solutions................1
Abstract..............................................................................................................................................3
Agile and Lean Learning Innovation................................................................................................3
A Foundation of Scrum.................................................................................................................4
Lean Principles..............................................................................................................................5
Efficient and Effective Instructional Design.................................................................................8
Harnessing the Rapid Evolution of Technology .............................................................................9
CGS History of Continuous Innovation ...............................................................................................11
Speed to Value................................................................................................................................13
Agile Value Delivery.......................................................................................................................14
Iterative Delivery .........................................................................................................................14
ADDIE Delivery...............................................................................................................................15
Embracing Change with Adaptability and Flexibility ....................................................................16
Agile Adaptation of Requirements and Design.........................................................................16
Flexibly Meeting Client Schedule Changes ..............................................................................17
Demonstrating Progress of Development and Delivery...........................................................17
Commitment to Quality...................................................................................................................19
Measuring Success ........................................................................................................................19
Measure #1: Cost Effectiveness of Solutions...........................................................................19
Measure #2: Delighting Clients..................................................................................................20
Helping Clients Measure Success.............................................................................................21
Top-Down Approach to Measuring Learning Solutions...........................................................21
Summary .........................................................................................................................................23
Author Biographies .........................................................................................................................24
Elizabeth Woodward...................................................................................................................24
Micah White.................................................................................................................................24

3. 3
Abstract
Developing and delivering effective learning solutions is more challenging than ever. Enterprises
are continuously asking their learning organizations to do more with less. Total corporate
spending on training and the amount spent per employee has been declining since 2010. Chief
Learning Officers (CLOs) and their learning organizations are arguably experiencing the most
dramatic pace of technology change and convergence of technologies ever experienced.
Additionally, emerging technologies, business competition, and learner expectations are
increasing the pressure on learning organizations to reduce time-to-value—to deliver value
more quickly than ever. Learning organizations that can harness the rapid evolution of
technology to improve learning outcomes and demonstrate business impact are more likely to
receive investment in the future. Computer Generated Solutions (CGS) has been a leading
provider of learning solutions for over 30 years. They have managed to thrive, innovate, and
help clients to deliver high-value learning solutions in a climate where many learning
organizations have closed their doors. CGS’s bold success is the result of passion for
continuously innovating within the field of enablement. Their approach to creating learning
solutions is deeply rooted in a collaborative culture of delighting clients using a strong
foundation of lean and agile principles and values.
Agile and Lean Learning Innovation
Learning professionals who have developed a proposal or designed, developed, or delivered a
learning solution using traditional methods are likely to have experienced challenging projects
throughout their careers where:
 Learning solutions may have taken much longer to deliver than originally anticipated.
 Cost estimates were exceeded just to meet minimum expectations.
 Delivery was delayed—solutions took much longer than they had originally expected.
 The true progress (percent completion) was difficult to assess.
 Rich media versions of learning solutions didn’t meet the learning organization’s or the
learner’s expectations.
 The enterprise may have questioned the business impact of the investment in learning.
 Budget cuts may have left the learning organization to deliver less effective learning
solutions or meet fewer of the learner’s needs.
Similar problems have plagued software development organizations. In 2011, a group of
software development thought leaders met to discuss the essential principles and values that
lead to successfully delivering software solutions. Through those discussions, they created the
Agile Manifesto—a set of principles and values that describe agile software development
methods.
Today, more than 52% of organizations are using agile approaches to manage the majority of
their software development projects. Scrum and Scrum variants are the most popular agile
methodologies in use at 73%. 75% of companies using agile are doing so to accelerate time to

4. 4
market, 74% to increase productivity, and 81% to improve their adaptability and to better
manage change.1
In 2004, CGS first began embracing elements of Scrum, the leading agile software development
framework today as a result of analyzing their own successes and challenges with delivering
learning solutions. The Practical Guide to Distributed Scrum shares some of CGS’s early
experiences with implementing Scrum in a complex environment with distributed team members
and clients.
CGS combined core agile principles, Scrum, best practices for instructional design, and lean
principles championed by Tom and Mary Poppendieck to create a lightweight approach to
continuous improvement.2 3
CGS continues to evolve the approach and has most recently
extended it with DevOps concepts of further optimizing the whole. CGS refers to their lean and
agile approach as L&A Learning.
A Foundation of Scrum
Although CGS can and will accommodate most learning development methods and deliverable
approaches requested by clients, the CGS team will internally leverage agile methods to the
extent possible to help ensure the success of projects. To provide a project heartbeat,
transparency, reduced time-to-value, and greater alignment with client expectations over the
course of a project, L&A Learning uses Scrum.
Scrum is a framework within which people can address complex adaptive problems, while
productively and creatively delivering products of the highest possible value. Scrum employs an
iterative, highly transparent, and adaptive approach to delivery. (See Figure 1.)
1
“8th
Annual State of Agile Survey,” Version One, 2014, http://www.versionone.com/pdf/2013-state-of-agile-
survey.pdf
2
“Manifesto for Agile Software Development,” http://agilemanifesto.org/
3
See Implementing Lean Software Development by Tom and Mary Poppendieck

5. 5
Figure 1. Scrum
Using Scrum we have an ordered dynamic list of all of the work that might be needed for the
learning solution in a Product Backlog. Every Sprint--one week for CGS--a CGS cross-functional
team (instructional designers, artists, writers, editors, and others) discusses the highest ranked
work for the week based on client insight and agrees to the approach for delivering the week’s
learning deliverables. The team commits to the work for the week and that becomes the Sprint
Backlog. The team identifies the primary focus of the week and that becomes the Sprint Goal.
The team consists of team members with all of the skills required to complete the learning
project, including instructional designers, artists, writers, editors, and others. Each day, the team
participates in a 15-minute meeting to discuss what they accomplished the previous day, what
will be accomplished that day, and any issues. At the end of the week, the team engages in
Sprint Reviews—demonstrations of what was accomplished that week and how it might impact
the next week’s work. Note that the resulting learning deliverable is expected to be DONE—in
some usable condition that clients can touch, interact with, and evaluate.
Next, the team conducts a Sprint Retrospective where they discuss how they can improve in the
following week. This is one of the critical events prescribed by Scrum. The Sprint Retrospective
is a chance for the team to inspect how the last Sprint went with regards to people,
relationships, process, and tools and to create a plan for implementing improvements in the next
week.4
By inspecting and adapting, CGS weaves continuous improvements into the fabric of
their delivery system. For more information on Scrum see the Scrum Guide on Scrum.org.5
Lean Principles
L&A Learning uses Scrum as a foundational framework for managing projects in an adaptable,
value-driven way. Additionally, L&A Learning embraces the seven principles of lean software
4
https://www.scrum.org/Portals/0/Documents/Scrum%20Guides/2013/Scrum-Guide.pdf
5
https://www.scrum.org/Portals/0/Documents/Scrum%20Guides/2013/Scrum-Guide.pdf#zoom=100

6. 6
development in order to improve efficiency of the learning development and delivery process
wherever possible.6
The principles of lean software development are:
 Eliminate waste. Anything that does not add value in some way to the learning
solutions delivered to clients is considered a waste. By ruthlessly targeting and
eliminating waste in processes, CGS is able to focus more intensely on learning
capabilities that matter—advancing instructional design, leveraging the latest
technologies to improve the learner’s experience, etc.
 Build in quality. Processes are in place to prevent defects and issues from entering the
process.
 Amplify knowledge. Planning is useful, but both the client and the team create critical
knowledge by experiencing the solution—even if the experience involves a fraction of
the ultimate deliverable.
 Decide as late as possible. Development and delivery of learning solutions is almost
always associated with some uncertainty. By taking an options-based approach and
delaying decisions until they can be made based on facts learned through the
development process rather than uncertain assumptions and predictions, teams are able
to deliver better results.
 Deliver as quickly as possible. The sooner a solution (or part of a solution) is delivered
with quality, the sooner clients are able to provide feedback that the team can
incorporate into the work for the next week.
 Empower the team. People are not resources.7
An empowered team attracts bright,
creative people who collaborate to deliver wildly successful deliverables.
 Optimize the whole. Optimizing the whole means observing the end-to-end value
stream that CGS clients experience. From initial contact through to delivery of the
solution and subsequent follow-up, CGS aggressively analyzes opportunities to improve
the client experience.
Fundamental to lean software development is the notion of actively seeking out many
opportunities to eliminate waste—activities that deliver no value.
Inventory: Unfinished goods. For learning organizations, unfinished goods are partially
completed or undelivered courseware. Lean principles of eliminating waste promote delivery of
smaller modules, learning modules, or other enablement artifacts as they become available
during the process of completing larger courses that remain undelivered for a longer period.
Overproduction: Producing more than required. Enabling learners to efficiently gain
knowledge, skills, and behaviors is essential to delivering value to the enterprise.
Overproduction, delivering more materials than required, can result in additional and
6
Mary Poppendieck and Tom Poppendieck, Lean Software Development: An Agile Toolkit (Boston, MA: Pearson
Education, Inc., 2003).
7
Mary Poppendieck and Tom Poppendieck, The Lean Mindset (Upper Saddle River, NJ: Pearson Education, Inc.,
2014).

7. 7
unnecessary time spent by learners. The CGS team collaborates with clients to determine the
most effective approach for a given learning and business objective.
Extra processing: Additional, unnecessary steps in processes. An example of extra
processing is individually producing separate learning deliverables for mobile devices rather
than using responsive design to create one design for multiple devices. This waste increases
costs and opportunities for errors.
Transportation: Shipping goods from place to place. Applying lean principles to learning
solutions, transportation involves both movement between systems and movement between
team members and handoffs.
Waiting: Delays between process steps. Examples of waiting include delays in delivery of
working software to customers, bureaucracy—decisions or approvals that create delays,
blockers (inefficient tools, inputs, or reviews), and waiting for feedback. Daily planning meetings
(also known as Daily Scrums) help to address blockers and inefficiencies. During the 15-minute
meeting each day, the team answers three questions:
1. What work did you complete yesterday?
2. What will you complete today?
3. Do you have any impediments that prevent you from meeting commitments?
Impediments/blockers are taken seriously by the team, and the team works together with
management, clients, and others as needed to address the blocker within hours.
Motion: Movement within the process. An example of movement within a process would be
the case where an organization has no central repository and poor version control systems. In
this case, team members and clients email files from person to person, which can result in
confusion over which version is the most recent and where the most recent files are.
Additionally, time is wasted as team members and clients seek and access files in different
locations. Task switching is another example of movement within processes. Even brief mental
blocks created by shifting between tasks can cost as much as 40% of someone's productive
time.8
Defects: Flaws in deliverables. Costs of defects increase over time as the development team
moves farther away from the point of injection. A team member may have to set up the
development and testing environment from scratch and re-familiarize with the materials before
troubleshooting and correcting the problem. Additionally, moving from one module or learning
deliverable to another without completing all testing can result in defects accruing and “technical
debt” increasing. By building quality into the process, L&A Learning not only improves quality
but also reduces the costs of developing and delivering learning solutions.
8
David Meyer, as cited in Iain Gillespie, “Multitasking Makes You Stupid, Studies Find,” The Sydney Morning
Herald, May 27, 2015, http://www.smh.com.au/digital-life/digital-life-news/multitasking-makes-you-stupid-
studies-find-20150520-gh5ouq.html

8. 8
Efficient and Effective Instructional Design
Instructional design is the practice of creating "instructional experiences which make the
acquisition of knowledge and skill more efficient, effective, and appealing."9
Quality instructional
design is essential to developing and delivering impactful learning experiences. Constructivist
theory is used to create learning experiences that approximate real-world interactions and
enable learners to construct their own knowledge.10
In the 1990s, CGS was an early adopter of
remote lab environments to enable learners to gain hands-on experience with complex software
as part of their learning experience. In the 2000s, as instructional designers recognized the
need to integrate e-learning into the creation of learning objects and curricula, CGS
implemented one of the earliest learning management systems, applying IEEE Learning
Technology Standards Committee (LTSC) Learning Object Metadata (LOM) and ADL Sharable
Content Object Reference Model (SCORM) standards. Throughout the 2000s, as cognitive load
theory advanced, the CGS team embraced the new knowledge to create more effective learning
experiences.11
CGS pursues new knowledge related to learning and digital experiences to
continuously improve and evolve instructional design expertise.
That said, CGS quickly recognized that ADDIE (a process for creating instructional materials),
and most popular ISD models that are variants of ADDIE, suffered from the same challenges as
waterfall software development.12
ADDIE provides a guideline for developing training and
performance support tools in five linear phases: Analysis, Design, Development,
Implementation, and Evaluation. The intent was for each phase to be completed before moving
on to the next phase. This approach in the world of software development is referred to as a
“waterfall” approach.
9
M. David Merrill, Leston Drake, Mark J. Lacy, Jean Pratt, and the ID2 Research Group. “Reclaiming instructional
design.” Educational Technology 36, no. 5 (1996): 5–7. http://mdavidmerrill.com/Papers/Reclaiming.PDF
10
Robert A. Reiser. "A History of Instructional Design and Technology: Part II: A History of Instructional Design."
Educational Technology Research and Development 49, no. 2 (2001): 57–67.
https://files.nyu.edu/jpd247/public/2251/readings/Reiser_2001_History_of_ID.pdf
11
Fred Paas, Alexander Renkl, and John Sweller. "Cognitive Load Theory: Instructional Implications of the
Interaction between Information Structures and Cognitive Architecture". Instructional Science 32, (2004): 1–8.
12
George M. Piskurich. Rapid Instructional Design: Learning ID Fast and Right (San Francisco, CA: John Wiley &
Sons, 2006).

9. 9
Figure 2. ADDIE and Waterfall Software Development
Today, there are many variations of the ADDIE model that take an iterative approach to
development. However, most focus on iterative delivery and evaluation of phases as well as
rapid prototyping after significant analysis.
Figure 3. More Recent ADDIE Model
The Scrum and agile approach to iterative development differs in that small slices of full-
featured capabilities are delivered throughout the project in weekly time-boxed iterations. Early
in the process, clients and learners are able to experience and provide feedback on the learning
deliverables. In small bites, the CGS team works through full cycles to demonstrate functional
courseware with reduced risk and more rapid time-to-value.
Harnessing the Rapid Evolution of Technology
CLOs and their learning organizations are arguably experiencing the most dramatic pace of
technology change and convergence of technologies ever experienced. The ability to quickly
analyze the trends and rapidly harness key evolving technologies to deliver more effective and
efficient learning experiences is essential. Six trends today demonstrate the impact of changing
technology on both the development and delivery of learning programs:

10. 10
 Digital experience delivery
 Digital convergence
 Data and analytics
 Mobile technology
 Cloud computing
 And even 3D printing
The continuous improvement aspect of agile development means that harnessing the rapid
evolution of technology is integral to the learning development process. Every Sprint, teams
look for opportunities to innovate in both learning deliverables and learning development.
Digital experience delivery makes or breaks firms13
today. Consumers, in general, have come
to expect a high-quality digital experience from service and product providers across
industries—retail, health care, entertainment, and others. Part of that expectation is set by the
increasing budgets and influence of polished marketing organizations on all aspects of the
digital presence of the companies they support.
Consumer expectations spread to other online experiences, including online learning
experiences, where learners expect high-quality graphic design, interactivity, fast-paced video,
and integrated audio elements. Learning organizations that can take advantage of available
tooling and rapidly evolve new techniques to create a more robust digital experience are able to
deliver more engaging learning experiences.
The digital convergence of physical and digital worlds is also changing how learners expect to
experience learning. In many situations, learning is embedded in software solutions, rather than
being part of a completely separate website or service. Learners can learn how to perform tasks
without leaving the context of the application. The evolving field of wearables (miniature
electronic devices that are worn) provides an ability to analyze the environment and provide
learning based on contextual data.14
Additionally, digital convergence is slowly changing how
learning is developed. As an example, a subject matter expert (SME) can record the execution
of tasks using wearable technology, which can be used for development of new learning
objects.
Data and analytics are enabling the delivery of more personalized online experiences. Today’s
software applications blend data from systems of record (HR, CRM, consumer databases, data
warehouses, and others) with systems of engagement (email, collaboration, social, and learning
systems) to provide more opportunities than ever before to create custom, tailored, and targeted
learning solutions that efficiently meet an individual’s learning objectives.
Additionally, the proliferation of sensors and sensor data is enabling a personalized learning
experience that would have been impossible in the recent past. Global revenue for environment
13
“Forrester Top Technology Trends 2014 and Beyond,” Forbes, http://www.forbes.com/pictures/eimh45mkdi/2-
digital-experience-delivery-makes-or-breaks-firms/
14
“ELI 7 Things You Should Know About… Wearable Technology,” Educause,
https://net.educause.edu/ir/library/pdf/ELI7102.pdf

11. 11
and health monitoring sensors is expected to reach $400 million in 2017, up from less than $2
million in 2012.15
Sensors delivered in mobile devices have continued to increase in diversity
beyond accelerometers, electronic compasses, gyroscopes, ambient and proximity light
sensors, humidity and temperature sensors, blood glucose monitors, and many others. Sensor
data increasingly provides details of the learner’s environment, physical state, and needs that
can be used to deliver increasingly personalized learning experiences.
Mobile technology is growing as a learning platform. The proliferation of mobile creates many
new opportunities for learning experiences that are no longer tied to a classroom or desktop
machine. However, mobile also increases the complexity of learning development and delivery.
Not only are there now several types of devices to accommodate, but there are variations
across the devices. This creates significant challenges for learning developers—particularly
those delivering to Android devices that vary according to device manufacturer, telecom
provider, version of OS, configuration of the device, and applications included by default with
the device.
Furthermore, Bring Your Own Device (BYOD) policies are increasing the variations in devices
that access the learning organization’s online resources. Gartner reports that 38% of companies
expect to stop providing workplace devices to staff by 2016 and that 70% of mobile
professionals will conduct their work on personal smart devices by 2018.16
Cloud computing is changing the speed at which learning organizations can adopt new
technology and deliver solutions to their clients. Cloud computing provides access to a shared
pool of configurable resources, such as networks, servers, stories, applications, and services,
that can be rapidly provisioned and released with minimal IT management.17
Learning
organizations can quickly provision sophisticated, tailored software environments and
application environments to their learners for hands-on software experiences. Additionally,
learning organizations can take advantage of rapidly evolving learning development technology
by accessing learning development software solutions in the cloud. It is no longer necessary to
spend days acquiring hardware, software, and software licenses to take advantage of the latest
technology.
CGS History of Continuous Innovation
CGS is always moving forward with new technologies and innovations. One way they ensure
this development is by reserving a certain amount of capacity each Sprint for targeted
innovation development. For example, CGS steadily made progress over time on development
of tooling to enable more efficient delivery of custom courses. The tooling exceeded the
capability of other e-learning development tools on the market at the time and enabled CGS to
15
Jérémie Bouchaud, “Environment and Health Sensors in Smartphones Set for Mighty Growth,” IHS Technology,
August 21, 2013, https://technology.ihs.com/446281/environment-and-health-sensors-in-smartphones-set-for-
mighty-growth
16
David A. Willis, “Bring Your Own Device: The Facts and the Future,” Gartner., April 11, 2013,
https://www.gartner.com/doc/2422315
17
“Final Version of NIST Cloud Computing Definition Published,” NIST, October 25, 2011,
http://www.nist.gov/itl/csd/cloud-102511.cfm

12. 12
quickly implement foundational capabilities that increased the team’s capacity to deliver more
sophisticated e-learning capabilities. As a result of this innovation, CGS received the 2009
Brandon Hall Excellence in Learning Technology bronze award for Best Advance in Technology
for Rapid Authoring.18
Figure 4. CGS Innovations
Over the years, CGS has often implemented futuristic learning solutions while others considered
possibly implementing the technologies in the future.
 2004: Flash-based training featuring an advanced case-based scenario role play to
support the client’s sales methodology.
 2005: Video-based “right and wrong” vignettes that incorporated tropes and visual cues
of a television program, including feature interviews with executive stakeholders.
 2006: Illustrated characters serving as financial advisors—each with a defined backstory
(relevant to the scenario) and investment positions.
 2007: Video-based “learning nuggets” that incorporated television-styled advertising with
longer form infomercials to stimulate learners, generate interest, and issue a call to
action to take the training module.
 2008: Rapid authoring with a focus on inclusiveness: advanced closed captions,
keyboard access, and support for enterprise compliance as a competitive differentiator.
 2009: iPlus authoring tool received Brandon Hall Award for Best Advance in Rapid
Authoring, beating out major vendors like Sun Microsystems and Adobe Systems Inc.
18
“CGS e-Learning Software Wins 2009 Brandon Hall Excellence in Learning Technology Award,” prweb, May 13,
2010, http://www.prweb.com/releases/2010/05/prweb3986674.htm

13. 13
 2010: Development of a virtual data center (a highly private area that is not accessible to
outside people) to host a fully realized simulation of security and assessment practices
of the installation. The simulation was fully delivered in Unity3D.
 2011: Blackberry sales playbook developed in Adobe AIR middleware for Blackberry
Playbook, iPad, and Android tablets.
 2012: Telecom service course with game mechanics (drive persona home,
achievements, open-world).
 2013: QBE Town, a fully realized 3D city with 15 locations (including a sports stadium,
restaurant, and bowling alley), which uses dual outputs: Flash stage3D technology
and/or WebGL; interior and exterior rendering engine; and points, scoring, and hidden
objects, all part of simulation mechanics.
 2014: Fully responsive device agnostic portal for sales transformation. Course
completes increased from 100,000 to over 1 million. Sales enablement training used in
the field on iPhone, tablets, and legacy desktop computers. Learning nuggets used to
promote product sections.
As Yuan and Woodman’s research has indicated, one of the biggest deterrents to innovation
within organizations is the fact that employees are unaware that innovation is expected of
them.19
Where innovation is encouraged and made part of the fabric of operations, teams
innovate. CGS expects innovation from team members, and, as a result of that expectation,
CGS delivers award-winning innovations.
Speed to Value
Learning organizations that can deliver learning more quickly have a significant competitive
edge over others who follow more traditional delivery cycles. Delivering learning as smaller
objects can:
 Enable an enterprise to begin training their employees more quickly, so that those
employees can begin reaping the benefits of knowledge or skill improvement more
quickly
 Help customers to have greater success with products more quickly, leading to improved
customer satisfaction and future sales opportunities
 Provide potential customers with the education that they need to make positive
purchasing decisions
Delivering more quickly allows the enterprise to begin accruing business value earlier in the
process. Figure 5 demonstrates value to the enterprise when all modules are delivered at the
end of delivery of an entire curriculum—in this case, four modules. Notice that development
takes place from t1 to t4 of the entire curriculum, and it is not until t4 that the learners and
enterprise begin extracting value from the investment.
19
Feirong Yuan and Richard W. Woodman. “Innovative Behavior in the Workplace: The Role of Performance and
Image Outcome Expectations.” Academy of Management Journal 53, (2010): 323–342.

14. 14
Figure 5. Increased Value Delivery over Time
Agile Value Delivery
Figure 6 demonstrates taking an agile approach to the curriculum. In this example, the
development team works with stakeholders to deliver Module 1 to learners. While Module 2 is in
development, learners and the enterprise are gaining value from Module 1. While Module 3 is in
development, learners and the enterprise are gaining value from both Module 1 and Module 2.
Figure 6. Traditional Value Delivery
As an example, one CGS client had a mission of “delivering a world class curriculum for
software development methods.” By working with the client, CGS was able to develop smaller
learning modules (learning objects) that enabled the organization to begin delivering education
to improve their software testing capabilities much earlier than the client had originally
anticipated. Given the business costs of finding defects later in the software development
process, the client and their independent metrics consultant determined the approach resulted
in millions of U.S. dollars of increased productivity. Additionally, the client eventually received a
$3.4 million grant from the U.S. Department of Labor in collaboration with the State of Arizona to
deliver the courses as part of the university curricula to raise the skills of Arizona's technology
workers.20
Iterative Delivery
One of the key aspects of Scrum is that the agile team delivers a useable increment of the
solution to the client at the end of each fixed block of time. Typically, CGS teams deliver
20
“USDOL Announces $3.4 Million Grant to Increase Skills of Arizona IT Workers,” National Association of State
Workforce Agencies, October 20, 2005, http://www.naswa.org/assets/utilities/serve.cfm?path=azitgrant05.htm

15. 15
increments weekly for longer projects. This means that every week, clients are able to interact
with a tangible deliverable that demonstrates progress on their project to date. Tangible weekly
deliverables provide CGS clients and the CGS team with transparency regarding progress of
the project.
For the modules described in Figure 7, CGS takes the approach of completing all aspects of
development required for each delivery cycle or Sprint. For example, the CGS team engages in
design, writing graphics, coding, QA, and any other required tasks for a smaller module before
continuing to the next module. Courses are developed as a sequence of smaller chunks. A
module consisting of three lessons that could be completed within a Sprint is decomposed into
the three lessons, and the team makes an effort to complete each lesson before moving on to
the next. At the end of the week, the team is able to demonstrate the three lessons to the client
and obtain feedback that is used to inspect, adapt, and improve delivery in the following week.
Figure 7. Approach to Delivering Rapid Value
ADDIE Delivery
For organizations that follow an ADDIE model or prefer/require that a highly detailed design be
delivered first for an entire course, followed by a lightweight alpha process for the entire course,
then beta, and then gold release, CGS takes an approach as shown in Figure 8.

16. 16
Figure 8. Traditional Approach Delivered Iteratively
Embracing Change with Adaptability and Flexibility
One of the key principles of agile methods is that “the best architectures, requirements, and
designs emerge throughout a project.”21
Agile Adaptation of Requirements and Design
Traditional Big Design Up Front (BDUF) approaches advocate spending significant time in
requirements and design, expecting that enough time will enable the development team to get
every detail exactly right in the earliest stages of a project.
Some development organizations require a sign-off at each of these early stages to “lock in”
requirements and designs during a paper or text stage before clients and the development team
have an opportunity to experience those elements in a richer e-learning format.
Unfortunately, clients in this situation have difficulty
providing meaningful feedback until they (and their
learners) have seen graphics options or treatments,
tried an e-learning interactive that an instructional
designer described in text, or have seen the elements
laid out by a graphic artist or programmer in the actual
deliverable format. As the client experiences education
deliverables, they have more ideas and feedback
regarding their preferences and what they believe might
be most effective for their enterprise and their learners.
As discussed in the previous section, delivering early
and frequently enables learning organizations to inspect and adapt quickly.
21
“Principles behind the Agile Manifesto,” http://agilemanifesto.org/principles.html
“In preparing for battle I
have always found that
plans are useless, but
planning is indispensable.”
― Dwight D. Eisenhower

17. 17
Flexibly Meeting Client Schedule Changes
As most learning and development professionals will attest, no matter how carefully a team
plans, plans will change throughout a project.
Plans may change for any number of reasons. Some examples include:
 While working with their SMEs, clients may realize that certain content is more or less
important than they previously understood. They may want to emphasize or devote more
effort to one learning objective rather than another.
 Clients may determine that they or their SMEs are not available as previously expected.
SMEs may have to address issues with their own customers or unexpectedly meet with
executives.
 Urgent issues or changes in priority may adversely impact the client and their project
schedule.
Rarely does a project go exactly according to the initial schedule, and plans, schedules,
priorities, and staffing plans change quickly. In most cases, strictly adhering to the schedule and
initial requirements would result in unsatisfied customers. The challenge many organizations
have is to be able to adapt to continuously changing requirements at scale—welcoming
changes throughout the process in a sustainable way that enables the team to delight their
customers.
Demonstrating Progress of Development and Delivery
Although plans change over time, planning—and collaboration—is essential. Planning takes
place in four ways. Initially, as part of the Statement of Work, CGS works with clients to
establish a high-level working plan. That plan evolves over the course of the project to
accommodate clients and new learning that takes place throughout development.
Release Planning
During Release Planning, the CGS team (facilitated by the CGS client advocate) works with the
client to determine an approach to delivery that reduces risk early in the development process,
provides hands-on experience with all aspects of the learning solution early in the process, and
enables the client to begin receiving value from completed deliverables as quickly as possible.
This information helps the team to determine the best ways to decompose work over the course
of the project and prioritize deliverables for optimal delivery of business value to the client.
Sprint Planning
In Scrum, every iteration begins with a Sprint planning meeting. At this meeting, the client
advocate works with the learning development team to determine what work should be
addressed during the next week. This brief weekly planning process is most often a
checkpoint—an opportunity to discuss any changes in client plans, priorities, learning, and
availability and what the team and client learned during the previous week regarding
preferences and implementation details.

18. 18
Daily Planning
Daily, the CGS development team meets briefly to discuss the work completed the previous
day, the work that is scheduled to be completed that day, and any blockers or issues they have
encountered. CGS has client advocates who help to ensure that projects progress forward and
that any issues are addressed. (In a pure Scrum environment, the Scrum Master would work to
address any potential blockers for the team and the client’s deliverable.) As an example of how
the process works, if the development team determines that they have not yet received content
from a client SME, the team makes the client advocate aware right away or—worst case—
during the Daily Scrum. Within a short time, the client advocate follows up with the client to
obtain the necessary content. In this way, daily planning helps assure clients that
development is progressing at a high-performing pace.
Demonstrated Results
Every Sprint results in a demonstrable deliverable that meets the discussed criteria of
acceptance and the definition of “done” for that deliverable at that point in the project. As an
example, for a client who prefers the ADDIE model and expects to see an entire course
delivered in phases, the team may deliver a version of the alpha within a Sprint. For a client that
embraces agile methods, CGS may deliver a completed smaller module at the end of each
Sprint.
Clients can expect to view—and often interact with—the deliverable at the end of every Sprint.
In the Scrum framework, demonstrations are identified at the Sprint Review meeting. For most
CGS projects, clients can expect to see significant progress at least every week. Demonstrated
results are the key to clients’ confidence that learning solutions are truly progressing
according to expectations.
Figure 9. Sprint Reviews

19. 19
Commitment to Quality
L&A Learning builds quality into the learning solution development and delivery process.
Definition of done. The definition of done is a way of gaining shared understanding regarding
what “done” will look like for deliverables in a given Sprint. The definition applies to everything
delivered during that Sprint. Editing, QA, SCORM packaging, and other items may be part of the
definition of done checklist.
Acceptance criteria. Conditions of acceptance relate to a specific deliverable. It’s possible that
an early deliverable may not include a completed, sophisticated simulation, for example. What is
important is that the team and clients have a common understanding of “done” for the iteration.
Weekly reviews. Weekly demonstrations of progress with the client help to build quality into the
deliverable, as well.
Traceability. Traceability throughout a project is critical for building quality into the learning
solution. Client feedback is traceable to a specific version of the deliverable and the CGS team
responds with an action taken for each item of feedback.
Measuring Success
CGS measures success in two ways: cost effectiveness of solutions and delighting clients.
Measure #1: Cost Effectiveness of Solutions
By innovating for cost-effective development of high-quality, effective learning solutions, CGS
enables clients to deliver more value to their learners for less investment. Additionally, finding
ways to increase productivity of common, low-end tasks enables the team to invest more effort
in leveraging new technologies and methods for delivery of more effective learning.
Figure 10. Metric #1—Cost Effectiveness of Solutions
Revenue per Headcount
The revenue per headcount target is a fixed number that represents the team’s throughput. This
metric is only used in conjunction with customer satisfaction. The team is only successful if an
increase in revenue per headcount occurs in parallel with delighting customers.

20. 20
Expense/Revenue as a Percentage
The expense/revenue percentage is helpful in quantifying the success of L&A Learning. Again,
this metric is only viewed in parallel with client satisfaction.
Measure #2: Delighting Clients
Of course, cost-effectiveness is pointless if clients are not delighted by the results of their
learning solutions. As a result, CGS’s primary metric for determining success is the degree to
which deliverables, interactions with the team, and processes delight CGS clients.
Figure 11. Metric #2—Delighting Clients
CGS measures success in delighting clients in four ways.
Net Promoter Score
After each project completion, clients are asked to answer one simple question: “On a score of
0–10 with 10 being extremely likely and 0 being not likely at all, how likely is it that you would
recommend CGS Enterprise Learning to a friend or colleague?”
Growth in the Number of E-learning Projects
Delighted clients are more likely to return for additional help with learning initiatives and projects
in the future. CGS measures success on opportunities to subsequently support clients.
Number of New Clients
Delighted clients are more likely to recommend CGS to other departments within their enterprise
or to their peers in other enterprises.
Client References
CGS views client references in the form of case studies, white papers, co-presentations, video
testimonials, and others as being a significant testimony to CGS’s ability to delight clients.
Additionally, client references co-created with CGS can often be used by clients to build
awareness of their excellent learning programs.

21. 21
Helping Clients Measure Success
CGS’s position is that CGS is a partner in helping clients to develop and implement learning
solutions that drive a higher level of business value—increased revenue, reduced operating
costs, improved quality, etc. Throughout the project, CGS works with clients to determine what
their business goals are and to identify ways that they can support clients in measuring the
impact of their solutions.
Most learning organizations are familiar with Kirkpatrick’s Four Levels of Evaluation22
as a way
of determining the effectiveness of learning solutions:
 Level 1. Reaction
 Level 2. Learning
 Level 3. Behavior
 Level 4. Results
It can be more challenging to measure results at higher levels beyond the reaction “smile
sheet,” but learning organizations that are able to demonstrate value to the business are more
likely to receive future investment from the business. The corporate training market experienced
a low point between 2009–2010 and total spending on training, as well as amount spent per
employee, have been declining since then.23
Top-Down Approach to Measuring Learning Solutions
During the initial proposal process, CGS takes a top-down approach to collaboratively deriving
the measures for success. CGS asks important questions about the expected value of the
learning program and the problem(s) that the program is intended to solve. Understanding the
business impact enables CGS to propose a solution that helps clients to evaluate and
demonstrate the value of their learning deliverables in terms of business value.
 Level 4 Results. What is the expected business impact of this learning solution (e.g.,
employee retention, increased sales, greater customer satisfaction . . .)? This is often a
longer-term metric that will require input, baseline, and tracking from outside of the
learning organization.
 Level 3 Behavior. What are you expecting for people to do differently on the job or as a
customer in order to have the desired business impact?
 Level 2 Learning. What do you need to teach your audience in order for them to behave
differently? What is their current state and what should it be after they use your learning
solution?
 Level 1 Reaction. How will you know if they had a positive reaction to this deliverable
and what might help you to improve for next time?
22
Donald L. Kirkpatrick and James D. Kirkpatrick. Implementing the Four Levels (Oakland, CA: Berrett-Koehler
Publishers, 2008).
23
“E-Learning Market Trends & Forecast 2014 – 2016 Report,” March 2014,
http://www.docebo.com/landing/contactform/elearning-market-trends-and-forecast-2014-2016-docebo-
report.pdf

22. 22
Example 1. Measuring Success
As CGS was working with a client to create a proposal to deliver at a global virtual conference,
the team discussed the different types of functionality that would be required. The client focused
on hands-on virtual software labs, conference kiosks/pedestals, virtual help desk, branding of
the site, and much more. Eventually, the CGS facilitator turned the discussion to measures of
success and asked, “What would it take to make you a hero?” The client paused thoughtfully
and then responded, “We would have to have 1,500 people in specified parts of the organization
and certain regions attend . . .” With that information, CGS proposed a solution that included
active recruitment and follow-up with conference attendees to ensure that the client achieved
the target demographics with their learning program. While participation is a low-level metric, in
this client’s case the specifics of attendance were critical to achieving their goal of global
awareness.
Example #2: Tracking the Impact of Software Development Testing Modules
CGS provided a learning solution for an organization that was interested in reducing the number
of defects that were being identified by customers using their software. The expectation was
that the defects should have been caught earlier during development and that by teaching
software development team members new testing methods, the learning organization would
help to reduce software maintenance costs. The following is a high-level summary of the
measurement plan established for the learning solution. Note that only Level 1 and Level 2
metrics were solely in the purview of the client’s learning organization. The team collaborated
with the QA organization and Support organization for Level 3 and Level 4 metrics. The
successful measurement of this program was key to the client learning organization receiving
increased funding the following year.
The Level 4 metric was to reduce maintenance costs related to defect “escapes”—defects
detected by the customer. A second supporting metric was the severity of the defects that
customers were reporting to the Support organization. A “Severity 1” defect that shut down the
client was expected to be more costly in terms of the fix, the urgency of the fix, and impact to
other efforts in progress and long-term satisfaction of the client.
The Level 3 metric involved QA metrics—defects being detected earlier in their processes. The
client expected for the courseware to enable detection of potential defects earlier in the process.
By detecting defects earlier, fewer defects were expected to reach customers.
The Level 2 metric involved a survey sent out to teams a month after they had taken the
learning modules. They were asked to indicate the degree to which they were using what they
had learned in the course. Learners were also asked to identify the amount of time that they
believed the methods had saved them in identifying problems before the software was passed
on to production. Although the second metric is a “soft” metric, it was useful as an indicator of
the value that learners believed the modules to be delivering.
The Level 1 metric was gathered from a survey delivered immediately after modules were
taken. Learners were asked about the value of the courseware, whether they had learned
something new, whether they believed they would apply what they learned on the job, graphic
design, quality of writing, quality of the hands-on exercises, etc.

23. 23
Summary
CGS delivers measurable impact and value by taking a lean and agile approach to creating
learning solutions that delight clients. The L&A Learning approach has resulted in a remarkable
95% rate of repeat business. Clients are better able to quickly seize opportunities related to
market and technology changes and to deliver high-quality solutions that deliver the targeted
results. The L&A Learning approach embraces continuous improvement. For additional
information, contact CGS Enterprise Learning. CGS Enterprise Learning is an award-winning
provider of custom professional development solutions with more than 30 years of experience
helping successful businesses to continuously innovate and drive change through learning
programs with measurable business impact.

24. 24
Author Biographies
Elizabeth Woodward
Elizabeth Woodward is a Lean and Agile Consultant and Learning
Strategist with Computer Generated Solutions (CGS). She has more
than 20 years of experience improving efficiency and effectiveness of
learning organizations and learning deliverables. Elizabeth has held
roles as diverse as senior IT operations administrator, programmer,
architect, project manager, transformation consultant, instructional
designer, and learning strategist for some of the world’s largest
enterprises and small businesses. As a member of the IBM CHQ Agile Transformation Team
from 2006–2010, Elizabeth was responsible for enabling more than 65,000 development team
members, managers, and executives worldwide during the early years of IBM’s agile adoption.
She has coached diverse agile organizations and teams, including large-scale product
development, small-scale project, and systems/embedded. Elizabeth has been an active
member of the agile community since 2007. She co-authored the book A Practical Guide to
Distributed Scrum endorsed by Ken Schwaber (co-creator of the #1 agile approach worldwide)
and has presented numerous conference sessions and courses in the United States, Canada,
United Kingdom, China, Sweden, and Switzerland. Elizabeth is also an avid inventor and
innovator with more than 20 patents filed and a special interest in agile methods for research,
innovation, and learning programs. More information: www.linkedin.com/in/elizabethwoodward/
and twitter.com/liz_woodward
Micah White
Micah White is the Director of Research & Development for Computer
Generated Solutions (CGS), a large privately held technology
company with over 5,500 employees and with offices in four
continents. Micah is a career graphics programmer who has been
working in the e-learning industry since 1999. Holding a bachelor’s
degree in Mathematics and Computing Science, Micah works with the
world’s best IT and telecom companies to deliver training that can be
used on all devices. Micah has a deep understanding of intelligent
design, which he has used again and again in the e-learning field, which stems, in part, from his
early association with Paul Allen’s Wired World. Micah’s work was nominated for a Canadian
Gemini Award in 2003, and he won a Brandon Hall Award for Best Advance in Rapid Authoring
in 2009. Since 2009, Micah has been working in the multi-device world, delivering responsive e-
learning solutions to smartphones, tablets, and desktops.