The concept of “shifting testing left” in the software development lifecycle is not new. Shifting testing from manual to automated and then upstream into engineering is a driving factor in DevOps and agile software development. However, Michael Nauman wonders why test automation, DevOps, and agile software development still frequently fail to deliver on their promises? Aligning and hardening your DevOps and test automation—along with streamlining your agile processes—is critical to your project. Michael shares how AutoCAD’s shifting testing left enabled improvements within their engineering team. Learn how the team increased engineering reliability and velocity, and forced process changes upstream into design and research all the way through to product support. Leave knowing why the concept of separation of concerns with regards to quality is as fundamental as the separation of code quality from product quality. Learn how the AutoCAD web team used process dogma and ruthless prioritization to combat metric idolatry and the host of other evils that hold teams back from fully realizing their potential and going beyond agile.

1.
T7
Test
Management
10/5/17
11:15
Shift
Left
Testing:
Going
Beyond
Agile
Presented
by:
Michael
Nauman
Autodesk
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2. Michael
Nauman
Autodesk
Michael
Nauman
started
out
his
career
doing
computer
aided
design
but
after
a
decade
made
the
leap
from
an
AutoCAD
user
to
a
tester
on
the
test
development
team
for
Autodesk
Inc.’s
flagship
product
AutoCAD.
He
has
spent
the
past
twenty
years
testing
various
versions
of
AutoCAD.
Michael
started
out
as
a
black
box
tester,
dabbled
in
test
automation,
evolved
into
a
testing
lead,
managed
testers
as
a
QA
manager,
and
now
is
the
product
owner
of
AutoCAD
Web.
Michael
is
passionate
about
AutoCAD,
DevOps,
bicycles,
and
surfing.

3. 1
© 2017 Autodesk
Michael Nauman
Software QA Manager
STARWEST 2017
Shifted Left: Beyond Agile
§ Who I am
§ Where I work and what I do
§ Why is Shifted Left: Beyond Agile important
§ Why Autodesk Shifted Left
§ What we learned…
What you are going to learn today

4. 2
Me
§ 10 years AutoCAD User
§ 20 years Quality Assurance
Autodesk, Inc.
§ BA in Geography specializing
in Geographic Information
Systems and Remote Sensing
§ Quality Assurance Lead for
AutoCAD Web
§ 4th STAR Conference
What is Autodesk?

5. 3
What is AutoCAD?
THE
GROUND
FLOOR
DRAWING
OFFICE
12.64.
©
GM
ARCHIVE
Image courtesy Apple Inc.
© 2017 Autodesk
What is Shifted Left: Beyond Agile?

6. 4
Software Development Life Cycle (SDLC)
Design Development
Quality
Assurance
Research Release
Years - Months
Waterfall
Design Development
Quality
Assurance
Research Release
Years - Months

7. 5
Shifting Left
Design Development
Quality
Assurance
Research Release
Months - Weeks
Quality AssuranceQuality AssuranceQuality Assurance
Left Shifted
Design DevelopmentResearch Release
Weeks
Quality
Assurance
Quality
Assurance
Quality
Assurance
Quality
Assurance
Product Engineering

8. 6
Beyond…
Design Development ProductionResearch Support
DevOps
Prototyping
Surveys
Customer Input
Weeks - Days
QA QA QA QA QA
Product Engineering
Development
Production
Support
Build Measure Learn
Research
Prototyping
Design
QA
DevOps
QA
QA
Engineering
Product
Customer
SupportQA
QA

9. 7
© 2017 Autodesk
Why is Shift Left important to Autodesk?
Why is Shift Left important to Autodesk?
§ Business Model Transformation
§ Changing Landscape
§ Respond to Competition
§ Efficiency/Speed
§ Profit
Image courtesy: Gettyimages Dianne Manson

10. 8
© 2017 Autodesk
How Autodesk is Shifting Left
• Default to Open
• Global Engineering
• Cross Team commitment to Quality
• Senior Leadership Support
• Experience Design VP
• MVP vs. MLP
How Autodesk is Shifting Left

11. 9
Image courtesy blog.theworkinggroup.ca
How AutoCAD is Shifting Left...
Image courtesy Utkarsh Khare
Image courtesy https://www.123rf.com
© 2017 Autodesk
Beyond Agile

12. 10
Separation of Concerns
Engineering Team
§ Quality of Code
§ Performance Metrics
§ Infrastructure
§ Builds
§ Test Automation
§ DevOps
§ Monitoring
Product Team
§ Quality of Product
§ Business Metrics
§ Customer
§ Releases
§ Success Criteria
§ Analytics
§ Support
Process Dogma
§ Lean Startup vs. Agile Scrum
§ Release Cycles
§ 1 week Sprints
§ Planning Monday, Retro Friday, Standup Everyday
§ User Stories
§ Very small: 1-2 days
§ Success Criteria: Gherkin
§ Acceptance criteria fails, story rejected, no bug
§ Infrastructure & Test Automation…

13. 11
2.12.0
Develop
2.11.0
2.11.0.d7ec6b92.10.1.715f978 2.12.0.0c4283d
Master
2.10.1
2.12.0.715f978
TEST
http://nvie.com/posts/a-successful-git-branching-model/
✖
TEST ✔
TEST ✔
TEST ✔
TEST ✔ TEST ✔
TEST ✔
TEST ✔
DevOps
Master
Metric Idolatry
§ “Nothing is above the scrutiny of why” - Keith Klain
§ What problem are we trying to solve?
§ What value are we providing to our customer?
§ What metrics matter?
§ Business vs. Engineering
§ Analytics vs. Benchmarks
§ Fear Drives Metrics

14. 12
§ Build Measure Learn = Learning can be painful
§ UX, QA, & Tech, ALL debt matters
§ Increased Engineering efficency puts pressure on
other functional teams
§ Experience Design: Quality Injection
§ Success Criteria (Gherkin)
§ “Environment, Workflow, Dataset” – Hai Qing Wang
§ Merging & Promotion: Quality Validation
§ CI & DevOps
Learnings
Learnings, Part Deux
§ Right place, right time
§ Keep your axe sharp
§ “Always proud, never satisfied” - Amar Hanspal

15. 13
© 2017 Autodesk
Questions
Autodesk and the Autodesk logo are registered trademarks or trademarks of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders.
Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document.
© 2017 Autodesk. All rights reserved.