Presentation from Toronto's 2016 Canadian Executive Cloud & DevOps Summit on Friday, November 4th. Speaker: Shawn Gandhi, Head of Solutions Architecture, AWS Canada Title: Rogue Development: DevOps in the Amazon Warehouse

1. DevOps-ing the
Fulfillment Center
Shawn Gandhi
Head of Solutions Architecture
AWS Canada
@shawnagram

3. How to set up an IT organization (processes/people/tools)
where Customer Obsession, Agility, and Innovation
are primary components of its culture?

4. What type of organizations are doing this well?
Customer Obsession, Agility, Innovation

5. Question then becomes:
How can a large-scale organization
operate like a start-up?

6. Back to the Fulfillment Center…
Identify and break warehouse processes into fine-grained units

7. Assign those units to small development teams
Amazon Fulfilment Technologies

8. Process: Stow products into Bins
microservice
(software)
builds uses
Now let each team operate as a start-up
Autonomous, full ownership of their domain, freedom to innovate
Stow Team

10. Web of Loosely-Coupled Micro-services
Store products in the warehouse
Pick products for customers Report defects
Track shipments
Product Details

11. Amazon Software Development Team
4-8 developers (SDEs) + manager (SDM)
Teams should be small and hierarchically flat
2-Pizza Teams

12. Amazon Software Development Team
4-8 developers (SDEs) + manager (SDM)
Agile Roles
scrum master, developers, product owner
Teams adopt an Agile methodology (eg, Scrum, Kanban, etc.)
Agile Teams

13. Teams are flexible and can be quickly re-organized within the org
Eg, due to new demands, teams getting too big, maturing products
Team A Team B
Team C Team D
Team A Team B
Team C Team D Team E
new team
quick re-org

14. Discourage “silos” in the organization
It leads to lack of Agility / Ownership
Operations
Database
Team
Quality Assurance
Application
Security
Software Dev
Team
“We’re blocked waiting
the Dev environment setup”
“We’re blocked waiting
a new table/SQL script”
“We’re blocked waiting
performance tests”
”We’re blocked waiting source
code manual inspection”
”We’re blocked waiting
database security approvals”
”We’re blocked waiting
security tests to complete”
”We’re blocked waiting the
creation of security certificates”
“We’re blocked waiting the
setup of the database hosts”
“We’re blocked waiting new test hosts”

15. Instead teams should operate in a DevOps setting
Break the Silos!
Operations
Database
Administrator
Quality Assurance
(Testers)
Application
Security
Software Dev
Team
“Working together”

16. It’s not necessarily about putting
different professionals to
work together
At Amazon, it’s about empowering
developers and letting them fully
own the systems they build
What does “working together” mean?
The Anatomy of Amazon Dev Teams

17. Some Responsibilities of an Amazon Developer
• Write application code
• Review code of others
• Write unit/integration/performance tests
• Create database schemas/SQL queries
• Create and automate CI/CD pipelines
• Provision test and production hosts
• Use secure APIs and apply security policies
• Monitor production environments
• Be on-call on a rotation-basis and carry a pager

18. Amazon DevOps Model
Builder tools team Application security team
Tools, APIs, standards, best practices,
documentation, training, reviews, etc.
Empower developers and let them OWN the system they build!
Thousands of Dev teams
…

19. push code
CI/CD Pipeline
Code review check / Code standards check
Security tests / Unit tests check
Integ tests check / Acceptance tests check
…
deployment
monitoring production software
automated quality assurance and packing
Teams automate tasks as much as possible
Teams are encouraged to achieve full CD (continuous deployment)
continuous
feedback

20. Release processes levels
Source Build Test Production
Continuous integration
Continuous delivery
Continuous deployment

21. Benefits of Micro-services for Amazon
Build Test Release
Build Test Release
• 50 Million Deployments a Year (2014)
• Software enhancements delivered every second
Build Test Release
Build Test Release
Build Test Release
Build Test Release
… … … … … …
Gigantic Web of Micro-services at Amazon

22. Amazon Transformation Into a DevOps Shop

23. Continuous Improvement
Culture, processes, people, software, tooling
2001
BuildSystems
dashboarding
2009 Today
EmpowermentResizing

24. 2001
2001-2009: Microservices + 2-Pizza Teams
monolithic application + teams microservices + 2 pizza teams
2009

25. Things went much
better under this model
and teams were
developing features
faster than ever, but we
felt that we could still
improve.

26. In 2009, we ran
a study to find
out where
inefficiencies
might still exist

27. We were just waiting.
WaitWrite
Code WaitBuild
Code WaitDeploy
to Test
Deploy
to Prod

28. We were just waiting.
WaitWrite
Code WaitBuild
Code WaitDeploy
to Test
Deploy
to Prod
Mins Days Mins Days Mins Days Mins

29. We were just waiting.
WaitWrite
Code WaitBuild
Code WaitDeploy
to Test
Deploy
to Prod
Weeks
Mins Days Mins Days Mins Days Mins

30. We were just waiting.
WaitWrite
Code WaitBuild
Code WaitDeploy
to Test
Deploy
to Prod
Weeks
Mins Days Mins Days Mins Days Mins

31. 2009-2013: Empowerment of Teams + Agile
2009
microservices + 2 pizza teams
BuildSystems
dashboarding
DevOps Tools + Agile
2010-Today

32. We built tools to automate
our software release
process
https://secure.flickr.com/photos/lindseygee/5894617854/

33. Automated actions and
transitions; from check-in to
production
Development benefits:
• Faster
• Safer
• Consistent & Standardized
• Visualization of the process
Pipelines

34. This has continued to work out really well:
In 2014:
• Thousands of service teams across Amazon
• Building microservices
• Practicing continuous delivery
• Using Agile practices
• Many environments (staging, beta, production)
50 million deployments

35. We continue to survey our software
developers every year and in 2014
results found only one development
tool/service could be correlated
statistically with happier developers:
Our pipelines service!

36. continuous delivery
==
happier developers!
https://www.flickr.com/photos/cannnela/4614340819/

37. Amazon-style DevOps on AWS
BuildSystems
dashboarding

38. https://aws.amazon.com/devops/
Find more about how these work at

39. Thank you!