2. Agenda
Context
Background: Software Testing Concepts
Background: What is DevOps?
What is Continuous Quality (CQ)?
CQ Concept of Operations
Categories of CQ Tests
CQ Technologies
What Happened?
What did we Learn?
Lessons Learned
3. Context
Halfaker and Associates (Halfaker,
www.halfaker.com) is a midsize company,
headquartered in Arlington
Halfaker creates, modernizes, integrates, and
secures mission critical systems for Federal
Government organizations
Halfaker is a fast-growing organization that invests
in improving our Agile management and software
engineering maturity to better serve our customers
As Halfaker has scaled rapidly (100 employees in
2015) and begun managing large, more complex,
more visible programs, we’ve needed to improve
how we ensure quality in our technology solutions
We’ve been prioritizing investment at the
intersection of Software Testing/Quality and
DevOps
4. Background: Software Testing Domain
The Test Pyramid
Diagrams from https://martinfowler.com/bliki/TestPyramid.html and
https://martinfowler.com/articles/microservice-testing/#conclusion-test-pyramid
Similar to the Food Pyramid, a healthy Testing Program (“diet”)
should be balanced – Mike Cohn and Martin Fowler have written
about the concept of the Test Pyramid many times
Some teams focus too much on end-to-end testing and
exploratory (ad-hoc) testing, without building the solid foundation
of unit and integration tests
Unit tests are small, in-code tests, which developers should write
consistently throughout the development and maintenance of
software – they are fast to run (because they are often automated)
and provide a solid foundation by ensuring that micro-level
assumptions are met
Moving up the test pyramid, there should be fewer tests created at
each level, with a focus more and more at the macro level of the
system
Across the layers, think about: (1) Does it do what it’s supposed to
do? (2) Is it intuitive to use correctly? (3) Can I break the system?
5. Background: What is DevOps?
Development Operations (DevOps) is a set of practices focused on improving
the intersection/relationship between software development and IT operations
teams
DevOps is based on a culture of collaboration, instead of traditional silos
DevOps is not a just technology – it is more of a philosophy, similar to how
Agile is much more than just Scrum ceremonies/tools/roles
DevOps is enabled through the use of technologies that automate activities
including building, testing, and delivering software to create smaller, lower
risk feedback loops related to software engineering
Atlassian.com/devops and The Phoenix Project are great introductions to the
topic
6. What is Continuous Quality (CQ)?
Viewing software testing and quality management as an event done by people
with “test” in their job titles doesn’t scale with the velocity and Agility needed for
today’s engineering work
Instead, organizations should view Continuous Quality (CQ) as the goal – with a
focus on:
Shift Left – Focus on measuring/assessing quality as early as possible,
continually “shifting left” to reduce the time a defect/misunderstanding exists
Automate Quality – The more we can automate quality, the more quickly and
comprehensively we can assess quality, freeing up time to focus on doing a
faster and better job of the quality work that can’t be automated
(If teams don’t have a mature, comprehensive testing program, they certainly
need to build that foundation BEFORE they can shift left and automate)
7. Concept of CQ Gates
We enforce quality in a Continuous Quality environment by
defining automated gates that does not let code/software pass
without passing the test
These gates enforce rules, such as:
Unit Test Coverage (Goal is 100%, but that isn’t always reasonable, so we
set an enforced threshold, such as >= 80% of lines are covered by unit tests;
see https://martinfowler.com/bliki/TestCoverage.html)
Unit Test Pass Rate (e.g. 100% of unit tests must pass)
Static Code Analysis Thresholds (e.g. SonarQube tests re: code complexity)
Accessibility Tests (e.g. Section 508,
https://www.deque.com/tools/worldspace-attest/)
Source code vulnerability scan (Look for security weaknesses)
Performance Testing (Build the application and test it for responsiveness)
Capitol One has a blog post about their 16 Gates
https://medium.com/capital-one-tech/focusing-on-the-devops-
pipeline-topo-pal-833d15edf0bd
8. Visualizing Continuous Quality
DevOps Teams should select the gates that provide the best return
(improving software quality) on investment of configuration and
maintenance
9. What Happened?
Halfaker established automated Continuous Integration/Continuous Deployment
(CI/CD) pipelines across several programs, using the open source Jenkins automation
technology
Halfaker then added Continuous Quality gates, which has led to dramatically
improved efficiency (less rework and surprises late in the process) and improved
quality
We initially focused on a few key gates (Unit Tests, Static Code Analysis (Code
Complexity) and Source Code Security Scans), because they were our biggest
drivers in ROI
One of our Federal Government customers had an Production Readiness Review
(PRR) security audit done by a IV&V team on one of our programs, and they were
shocked that we had zero identified security vulnerabilities – something that is very
uncommon for this Federal agency
We also standardizing on a source code branching strategy (GitFlow,
https://nvie.com/posts/a-successful-git-branching-model/), which improved the quality
of our build process
10. Some Recommended Technologies
These are some of the technologies we’ve found success with
Test Type CQ Approach
User Acceptance Testing
(UAT)
N/A (Humans presenting to customers)
Exploratory N/A (Humans exploring)
End to End (Functional) Many options such as Cucumber, Gherkin, Protractor
Performance Apache JMeter™
Security Fortify or SonarQube static code analysis
Code Complexity SonarQube Quality Gate
Component (Functional) N/A (Humans exploring parts of the system)
Integration Jenkins Build/Delivery Validation
Unit SonarQube Quality Gate: Unit Test Coverage >= 80%
11. Jenkins Automation Server Technology
Jenkins, an open source technology, provides the central orchestration required to
define the CI/CD pipeline, which collects, builds, validates quality gates, and
deploys the software
12. SonarQube Continuous Code Quality Tool
SonarQube, an open source technology, provides a platform to analyzing code
quality to identify areas for developers to review – below are screenshots showing
code quality (left) and quality gates (right)
13. Example of Unit Test Coverage Analysis
Image from https://blog.lgohlke.de/java/sonarqube/coverage/maven/2015/02/05/sonarqube-
code-coverage-unittests-integrationtests.html
14. Example of Static Code Analysis (Code Complexity) Failure
xxxx
Image from https://www.sonarqube.org/features/issues-tracking/
15. Example of Source Code Security Scan Failure
xxxx
Image from https://www.sonarqube.org/features/issues-tracking/
16. What did we Learn?
Make it Easy to See Continuous Quality
Make information convenient
(information radiators) – we used
open source tools including Capital
One’s Hygieia (see right) and
Grafana for several of our projects
to visualize data related to
Continuous Quality and DevOps
Many teams used shared TVs to
keep this information very visible
for the team, and created some
healthy peer pressure not to be the
person who broke the build
17. Example DevOps and Continuous Quality Technologies
Planning and Rqmts.
(PP, RD, REQM)
• Atlassian Jira
• Zephyr
• CA Agile (Rally)
Configuration
Management (CM)
• Source Repository
• GitHub
• Bitbucket
• Infrastructure
Configuration:
• Chef
• Ansible
• Terraform
Product Integration
(PI) and Release
Management
• Jenkins CI/CD
Pipeline
• CA Automic
• IBM UrbanCode
Testing (VAL, VER)
• Unit Testing (JUnit,
HtmlUnit, Microsoft
Unit Test)
• Behavior Driven Dev
(Cucumber, Gherkin,
RSpec)
• Functional Tests
(Selenium, Katalon,
Appium, Protractor,
SmartBear
TestComplete)
• Accessibility (Deque,
aXe, SortSite)
• Performance
(LoadUI, JMeter, MS
VSTS, LoadRunner)
18. Recap
Invest time in improving to get out of reactive firefighting
1. Shift Left! Try to enforce and test as early as you can, as it only gets
more expensive the farther requirements, designs, code, etc. make it in
the process
2. Automate everything you can and then tune to the right amount of
automation – you can’t, and shouldn’t, automate everything; but it’s wise
to try and see what sticks!
3. Enforce Continuous Quality gates that require developers to meet or
exceed your requirements, which reduces rework and clarifies
engineering expectations at the beginning
4. Maximizing the value produced (functional code that comes through the
CI/CD pipeline), instead of trying to keep everyone on your teams busy
(see The Goal)