Moving towards Innovation Experiment Systems

1. Teemu Karvonen, Lucy Ellen Lwakatare, Tanja Sauvola,
Pasi Kuvaja. Markku Oivo
MALMÖ UNIVERSITY
Helena Holmström Olsson
CHALMERS UNIVERSITY OF TECHNOLOGY
Jan Bosch
Hitting the Target: Practices for
Moving towards Innovation
Experiment Systems

2. AGENDA
BACKGROUND AND MOTIVATION
RESEARCH GOALS (RESEARCH QUESTIONS)
RESEARCH DESIGN (METHODS AND STUFF)
RESULTS
• CASE STUDY FINDINGS
• ”EXTENDED STAIRWAY TO HEAVEN”

3. Software business- & development challenges
SPEED &
EFFICIENCY
BUSINESS ACCURACY
”Hitting the Target”
SW company’s capability for
continuous delivery
SW company’s capability for
customer experimentations

4. Agile software development (Highsmith J., 2009)
• Agile Manifesto
• Scrum, XP, TDD, Continuous Integration…
Lean Software Development (Poppendieck M, 2007)
• Lean thinking, Toyota Way
• Continuous Improvement, Kaizen, Value stream mapping, Kanban…
Beyond agile methods
• Continuous delivery I.e build, test and deployment automation (Humble & Farley,
2010)
• Continuous experimentation (Fagerholm et al. 2014)
• Continuous innovation & Lean Startup-method (Ries E., 2011)
• Continuous software engineering Continuous * (Star) (Fitzerald et.al, 2014)
• Stairway to Heaven model, Innovation experiment systems &
HYPEX model (Holmström Olsson H., Bosch J. 2013)
BACKGROUND & RELATED STUDIES

5. Stairway to Heaven model
”EVOLUTION OF SOFTWARE DEVELOPMENT COMPANIES”
(Holmström Olsson H., Bosch J, Alahyari H. 2012)
Traditional
Development
Agile R&D
Continuous
Integration
Continuous
Deployment
R&D as
Innovation
Experiment
Systems
Step A:
Step C: Step E
Step D
R&D
Validation
R&D
Product Management
R&D
Validation
(Customer)
Product Management
R&D
Validation
Customer
Validation
R & D
Customer
Product Management
Step B:

6. RESEARCH
GOALS
To understand software development companies
current ways of working and their status in relation
to Stairway to Heaven model steps
To undertand software development companies
opinions about benefits and barriers in transition
towards Continuous Deployment and
Innnovation Experiment Systems
Research questions:
RQ1: What are the benefits and barriers that
software development companies experience
when moving towards Innovation Experiment
Systems
RQ2: What are the key practices that software
companies need to adopt in order to evolve their
software development practices according to the
Stairway to Heaven model?

7. Research design
• Replication/rerun of multiple-case study conducted 2012 in Sweden
(Holmstöm Olsson et al. 2012). Minor modifications to interview plan.
• Method: Multiple-case study
– Semi-structured interviews and qualitative data analysis (Data coding in Nvivo)
– Interpretive data analysis approach
• Interviews conducted during Oct-Dec 2014 in 5 software development companies
from Finland (Company A,B,C…)
• Five case companies from Finland
– 4 companies mainly operating in embedded systems domain (R&D services, telco, wireless
systems etc.)
– 1 company mainly operating in information technology (IT) services domain
– Products: Base stations, wireless device platforms, PC-client sofware, web-applications…
• 24 interviews, 4-5 interviews per company
• 1,5-2 hour individual semistructured interviews
• Interviewees from multiple roles (product managers, project managers, architecture
designers, UX designers, developers, testers, quality managers…)

8. RESULTS

9. CASE STUDY FINDINGS
• All 5 companies had adopted Agile development practices (Typically Scrum or Kanban or
”some mix of them”). 1 company (interviewed team) had just started using Scrum method.
• 4/5 had adopted continuous integration practices and was able to run automated builds
and test cases -> Developers were able to get fast feedback of how their code changes
integrates and works.
• Typical challenges in Continuous Integration
– Expenses and resources for CI, embedded systems hardware configurations (Special HW
accelators etc.…) Improvement of test automation coverage
• ”The challenge”: Moving towards continous deployment
– Improvement of software development capabilities for Continuous Deployment
– Increasing customer’s awereness of the benefits of Continuous Deployment? Finding
lead customers.
Traditional
Development
Continuous
Integration
Continuous
Deployment
R&D as
Innovation
Experiment
Systems
Step DStep B
Agile R&D

10. − “You actively seek the feedback and send interim versions of SW and prototype HW, to
collect that feedback. In the end, the result would be a product that the customer
actually want and not a product that they specified in the beginning, if those two things
are not the same anymore.”
− “Mainly the customer feedback, I think that's the most valuable information that you
get from deploying the early releases to customer.”
− “More visibility to the, how the development is proceeding”
− “It is also a way to show the customer that we are on schedule to deliver what we have
promised, to do”
− “We (and customer) are both saving money. We can keep our schedules better. You can
have those features that you really want and need, more.. faster than earlier. “
− “I think that, we benefit, and I benefit from not doing work that’s going to be trashed,
that we develop stuff that’s really required and we develop in an order that customer
needs it. “
Moving towards Continuous Deployment and IES
Key Benefits – Quotes

11. − “If we do handsets, deployment is fairly easy. We can do over the air updates, it's
very simple. But then, in some cases we have very complex systems where
multiple pieces of the system needs to be updated to specific versions. In those
cases, it is more challenging. It might take even, days to get the system updated
and back up and running again - It is almost impossible to do it continuously.“
− “In the early stages of the development it is risky. If the quality of your deliverables
is R&D (quality), which it is in the early stages.”
− “If you start deploying too early, to your customers, it might cause some negative
effects. Because sometimes customers are not as educated as us how, the
development cycle goes. They might freak out if they see the status of the
deliverables too early. So I would be cautious when starting the continuous
deployment.”
Moving towards Continuous Deployment and IES
Key Challenges – Quotes

12. − “I think the obstacle is the money. It is expensive but, doing it in a right manner
could make the next projects cost less money.”
− “Business is one limitation here. This level E (experimentation) is of course
interesting, and it would be very nice to work in project with this kind of a way of
working. But, I can't see, it’s not so near future for us. Maybe someday. “
− “Of course our customer they do have their own processes and way of working.
It’s quite hard for us to, tell them that please change your, processes. “
− “My first idea of (R&D as Innovation Experiment System) would be a nightmare.
I would not want to go there. We do this type of work when we develop,
demonstration, product for some customer. “
− “Customers need to get involved and they have to be continuously willing to check
things, which might be hard “
Moving towards Continuous Deployment and IES
Key Challenges – Quotes

13. WHAT ARE THE BENEFITS OF
• The innovation validation is fast
• Immediate feedback is obtained
• New business opportunities are identified
• Development resources are focused
Continuous
Integration
Continuous
Deployment
• Customers get fast and incremental
delivery of relevant functionality
• Customers can perform their own
testing and business activities on top
of deliveries
a) Moving from Continuous Integration to
Continuous Deployment
b) Moving from Continuous Deployment
to R&D as Innovation Experiment Systems
Continuous
Deployment
R&D as
Innovation
Experiment
Systems

14. WHAT ARE THE BARRIERS OF
Continuous
Integration
Continuous
Deployment
a) Moving from Continuous
Integration to Continuous
Deployment
b) Moving from
Continuous Deployment to
Innovation Experiment
Systems
Continuous
Deployment
R&D as
Innovation
Experiment
Systems
• The shortening of the validation and
verification cycle is complex and expensive
• Lack of trust in software quality
• Missing functions may cause a negative
impression – Traditional delivery
expectations
• Customer feedback integration into the
short development and business planning
cycle.
• Difficult to conduct experiments in safety
critical systems

15. PRACTICES FOR
Continuous
Integration
Continuous
Deployment
Moving from Continuous Integration to
Continuous Deployment
Moving from Continuous Deployment to
R&D as Innovation Experiment Systems
Continuous
Deployment
R&D as
Innovation
Experiment
Systems
• Identify and incorporate lead customers in
development process
• Renew business model, contracts, marketing and
sales strategies
• Provide architecture where software
functionalities can be deployed independently
• Improve automated system testing and adopt a
continuous release process
• Ensure that System/UX design and business
development work in short cycles and in
alignment with R&D
• Adopt data-driven strategic decision-making
model.
• Implement A/B testing with the customer
• Adopt product platform (e.g. virtualization,
cloud technologies) that enables flexible
experimentation.
• Establish a short customer feedback loop and
process for data-driven decision making
• Synchronize supplier and customer
organization in short development cycles

16. Extended
Stairway to
Heaven model
Practices for Moving towards Innovation
Experiment Systems

17. Extended Stairway to Heaven model (Practices)
20 practices in 4 categories:
– Business
– Architecture
– Process
– Organisation
4 adoption levels
Institutionalised
Product
Team
Not adopted
Business practices Architecture practices
Innovation experiment systems A/B testing Product Architecture supporting run-
time variation of
Product
Continuous Deployment Business model
transformation
Not adopted Componentisation for partial
release & rollback
Product
Continuous Integration Reduce cost of bad quality Institutionalised Modularisation to improve
ability to do unit &
Institutionalised
Agile R&D Product owner to represent
the customer
Institutionalised Lead architect to protect
architecture from erosion
Institutionalised
Traditional Development Customer validation at the
end of the project
Not adopted Early-on requirement freeze Not adopted
Process practices Organisation practices
Innovation experiment systems Customers real usage data
collection practices
Not adopted Product management &
business development
Not adopted
Continuous Deployment Continuous release process Team UX/system design integrated
in team
Product
Continuous Integration Test-driven development &
automated test scripts &
Institutionalised V&V function integrated in
agile team (continuous
Institutionalised
Agile R&D Sprints & daily standups Institutionalised Feature teams (cross-
functional)
Institutionalised
Traditional Development Milestone-driven
development
Not adopted Large development teams
divided into discipline
Not adopted

18. Extended Stairway to Heaven model (Example)
CASE COMPANY A
Business practices Architecture practices
Innovation experiment systems A/B testing Product Architecture supporting run-
time variation of
Product
Continuous Deployment Business model
transformation
Not adopted Componentisation for partial
release & rollback
Product
Continuous Integration Reduce cost of bad quality Institutionalised Modularisation to improve
ability to do unit &
Institutionalised
Agile R&D Product owner to represent
the customer
Institutionalised Lead architect to protect
architecture from erosion
Institutionalised
Traditional Development Customer validation at the
end of the project
Not adopted Early-on requirement freeze Not adopted
Process practices Organisation practices
Innovation experiment systems Customers real usage data
collection practices
Not adopted Product management &
business development
Not adopted
Continuous Deployment Continuous release process Team UX/system design integrated
in team
Product
Continuous Integration Test-driven development &
automated test scripts &
Institutionalised V&V function integrated in
agile team (continuous
Institutionalised
Agile R&D Sprints & daily standups Institutionalised Feature teams (cross-
functional)
Institutionalised
Traditional Development Milestone-driven
development
Not adopted Large development teams
divided into discipline
Not adopted

19. FUTURE STUDIES?
Ideas about directions of the future studies
Investigating different stakeholder’s point of view.
How customers perceive benefits and challenges
associated to continuous deployment /
experimentation?
How subcontractors and technology platform
providers perceive benefits and challenges
associated to continuous deployment /
experimentation?
Expanding view from company level to whole supply
chain or ecosystem level.
What are the challenges and benefits of continuous
deployment/experimentation in ”system level” e.g.
software ecosystem?

20. teemu.3.karvonen@oulu.fi
@ravennok
https://fi.linkedin.com/in/tkarvonen
Teemu Karvonen
University of Oulu

22. SUMMARY
Research
New ways of working -
Steps towards Innovation
Experiment Systems
Agile & CI – ”First two steps”
• Integrated and iterative Code & Test
-cycles
• Automated I&V, rapid feedback for
developers
• Sprint based planning, reviews and
repetitive retrospectives (continuous
improvement)
Continuous deployment ”3rd Step”
• Integrated and iterative Plan-Design-
Code-Test-Deploy -cycles
• Ability to deploy at will. Frequency
not deciding factor. Could be weekly
or daily deployments to production.
Innovation experiment systems
• Continuous customer feedback
• Data driven development decision
making
• Plan-Design-Implement-Test-Deploy-
Get Feedback
• Capability to learn from customer
real usage data

23. Hitting the Target: Practices for Moving towards Innovation Experiment
Systems. Karvonen T., Lwakatare L., Sauvola T., Bosch J., Holmström Olsson H.,
Kuvaja P., Oivo M.
RESEARH METHOD:
• A multiple-case study conducted in Finland (October-December 2014).
• 24 interviews, 5 software development companies, Qualitative data analysis
RESEARCH QUESTIONS:
• Moving towards Continuous Deployment & Innovation Experiment Systems.
RQ1: What are the improvement steps and key practices?
RQ2: What are the benefits and barriers?
MAIN CONTRIBUTION:
• Extension of the Stairway To Heaven model
– Introduction of steps and practices for moving towards Innovation Experiments Systems
STYDY IN A NUTSHELL:
COLLABORATION PARTNERS:
UNIVERSITY OF OULU,
MALMÖ UNIVERSITY,
CHALMERS UNIVERSITY OF TECHNOLOGY