Join Perfecto & CloudBees for a presentation on how to drive mobile app quality feedback in every build, on real devices. Watch a demo featuring the CloudBees Jenkins Workflow showcasing automated testing with Perfecto's Continuous Quality Lab.
2. A few housekeeping items
1. Submit
questions using the
Q&A panel.
2. Slides and
recording sent
after the webinar.
3. Please complete
survey after the
presentation.
4. Distress call!
• Release late by 2.5 months (from committed baseline)
• Low quality confidence
• Coverage gap concerns
• High defect rate
• “Stabilization” exposes more issues
• Low customer ratings after the delayed release
• Is this “Agile” or “Fragile”???
8. In Real Agile – Specify Acceptance Tests FIRST
to DRIVE Design/Development
Elaborate
Requirements
Test Design
Technical
Design
Coding / Unit
Testing
Test
Execution +
Fixing
Done
Apply ATDD Thinking
Test scenarios guide code design
ATDD* = Build Quality Into Design to prevent defects rather than just finding them
http://www.agilesparks.com/test-first-reading-list
Test Automation
*ATDD: Acceptance Test Driven Design
Define acceptance tests before developers begin coding
Start earlier Done is Done
9. Testers create more detailed test
examples
Developers use them to guide technical
design/specification
Team pulls a few stories
10. Story is pulled into
development
Testers focus on detailed test plans,
automation preparation, data
preparation.
Developers code and do
developer/unit/small testing
11. Developer verifies the story passes the
team “ready for story testing” criteria:
• Code is “in the build”
• The acceptance tests for this story
pass in the dev env
• Good code coverage using
automated unit tests
• Sanity test passes
Result: Teams using ATDD
Thinking get much higher,
automated, test
coverage(some above 90%)
than the typical test-last
teams. Both due to discipline
as well as due to developers
awareness to the acceptance
tests
Developer marks story
as dev done, ready for
testing
Development done means the acceptance tests already
pass
12. Tester pulls story into
testing
• Tester finds significantly fewer defects
• Product Owner is happier with how the
story works & finds fewer surprises.
• Both as a result of deeper alignment
achieved in the specification workshops
with ATDD scenarios.
Developers meanwhile
pull more stories into
coding
13. UAT
Regression
Performance
Security
Functional
Progression
Exploratory
Achieve Agile Testing Flow at the Story AND
Feature/Epic/Iteration level
1. Add Feature/Epic
level testing
2. left-shift more and more testing through
automation, enabling teams using
environments/tools/knowhow and more.
The 3-level Test Strategy Pyramid -
Story+Feature/Epic/Iteration+Release
Across platforms and user conditions
14. Automation Ownership Poll
Who’s responsible for automation in your world?
1. Automation COE separate team/department
2. Automation COE engineers assigned to the Agile Teams
3. Test Engineers/Testers on Agile Teams with support by Automation COE experts
4. Developers/Testers on the Agile Team
5. Other
14
15. Continuous Integration/Delivery
Needs an Automation Engine
Complex Delivery Pipelines
Delivery of App and Config
Robust and Highly Available
Results Report
SOURCE
CODE
CONTROL
SYSTEM
Code
Commit
</>
Development Production
Continuous Integration/Delivery
Commit Build Test Stage Deploy
16. Development Production
Continuous Integration/Delivery
Commit Build Test Stage Deploy
Jenkins – #1 Automation
Platform for CI/CD
“Jenkins is for building, testing
and delivering software
projects continuously.”
10 Years of Open Source Innovation
http://www.jenkins-ci.org
17. • Pipeline-As-Code
• Version control, Code review, etc.
• Entire flow is one concise Groovy script using Workflow DSL
• For loops, try-finally, fork-join, etc
• Stages throttle concurrency of builds
• Allocate build nodes and workspaces
• As many as you want, when you want
• Human input/approval integrated into flow
Multi-Stage CI/CD: Enter Jenkins Workflow
18. Jenkins Workflow:
Powers Multi-Stage CI/CD Pipelines
Development Deploy/Publish
Commit Build Regression Release
? ?
Branching
Looping
Restarts
Checkpoints
Manual Input
Conditionals
Unit Test
Smoke Test
Jenkins Workflow
Continuous Feedback
19. CloudBees Jenkins
Workflow Stage View
Jenkins Workflow Stage View
SOURCE CODE
CONTROL
SYSTEM
Code
Commit
</>
Development Production
Jenkins Workflow
24. •Connect WIFI
•Use iPhone
•Skip that OS update
•Connect to WIF
•Check Facebook
•Login your app
•Watch YouTube videos
•Answer the phone
•Clear emails
•Return to your app – check
status
•Accept LinkedIn invite
•Use navigation app
•…
25. UI
Integration
Unit Testing
Performance Security
Mobile Requires Shifting the Plan …
Target
Devices
Target
Env.OS
OS versions
Form factors
CPU & Memory
Manufacturers
Wearables
IOT
Networks
Orientation
Location
Notifications
Events
Device events
Edge tests
Submit questions using the Q&A panel.
We try to answer all questions during and after the presentation.
Our experts do not have access to the chat interface
Participate in the polls
Your participation is critical to a relevant, meaningful discussion.
Watch your email for a copy of the recording and slides
Today’s presentation will be recorded and sent to you.
Complete the survey following the presentation
Your feedback helps us provide educational tools like this webinar.
http://www.flickr.com/photos/andreasnilsson1976/1307180278/sizes/l/in/photostream/
slipping release due to long time finalizing quality.
Continuous Delivery needs to be managed by a robust automation engine.
The Automation engine must orchestrate the full CD pipeline including execution of each stage of the pipeline and the best of breed DevOps tooling used within each stage.
Developers can simply focus on creating code. When they check in code to the Source Code Control System, the CD automation engine will see that check-in, pull the code out, run the application compile and build process, execute and monitor unit tests and functional tests, possibly deploy to a staging environment and then run user acceptance tests and then, if desired, the CD automation engine can even automatically deploy to production.
(Automatic deployment to production is optional as not every organization is ready for that. However, Continuous Delivery means that there will always be production-ready code available to push to production when appropriate.)
A enterprise-ready CD automation engine will need to have three characteristics:
1) It needs to be able to create and manage complex, real-world CD Pipelines. These are pipelines that may not be just linear sequential steps. There may be parallel branches for parallel testing, there may be loops and restarts, and there may be a need for human-intervention within the pipelines. The automation engine needs to support all of these.
2) Continuous Delivery often is applied to not just the application code but the actual environment that the application runs in as well. This means defining the application environment configuration (OS parameters, etc) “as code” and managing that “configuration-as-code” through the CD Pipeline along with the Application code. This helps insure that the application will perform exactly in production as it did in Dev and Test.
3) The last characteristic of an enterprise CD automation engine is that it needs to have the capabilities expected of enterprise software including the ability to run in an HA, or highly available, configuration. Since the CD Pipelines are touching production systems and are automating the path to production, the CD Automation engine needs to be HA.
So, how can we do CD?
Jenkins is open source software that was originally developed over 10 years ago.
Here are some really cool features in Jenkins Workflow.
SAs talk here about coolness of Workflow.
As we said, CD involves creating Pipelines for application code.
This means that for every code “check-in” by every developer, the new code needs to go through the Pipeline that will take it from Dev to Prod.
So, the CD Platform needs to manage the execution of many of these Pipelines.
And the pipelines can be complex. They may have decision points or branches. They may need to be paused or restarted.
Modeling the real-world process of application delivery requires that these pipelines be robust, flexible and powerful.
The CloudBees Jenkins Platform also provides a dashboard that shows the current state of the pipeline activity, such as this example, which is critical to providing a full understanding of how things are performing.
CloudBees Jenkins Platform Enterprise Edition:
For Enterprise DevOps or Infrastructure teams that want to:
Run Jenkins operations @scale to support a large number of developers and/or teams
Optimize the performance and efficiency of the CD platform
Monitor usage and share resources across their Jenkins implementation
Have access to Jenkins support experts for quick problem resolution
Ships with All Advanced Enterprise Features for both:
CloudBees Jenkins Enterprise
CloudBees Jenkins Operations Center
CloudBees Jenkins Platform Team Edition:
For small Dev teams or organizations with a limited number of developers that want to:
Improve developer productivity by leveraging advanced developer features for Jenkins
Easily get started with continuous integration (CI) and continuous delivery (CD)
Have access to Jenkins Support experts for problem resolution
Ships with a starter-pack of Enterprise Features focused on Developer Productivity:
Developer Productivity
Improved developer feedback loop and eliminate downtime with Validated Merge for Git and GitHub Pull Request
Team Management
Organize teams with folders
Recoverability from failures using backup
Security
Roles-based access control
Isolate team-sensitive slaves
Analytics
Build and performance dashboards
Support Module
Faster diagnosis for production issues