Kevin Brockhoff presented on accelerating software delivery through test-driven development (TDD) and behavior-driven development (BDD). TDD and BDD help identify incorrect behavior early, reduce rework, increase code confidence through automated tests, and improve code flexibility and extensibility. Brockhoff discussed key software delivery metrics like deployment frequency and change failure rate. He also covered testing best practices like the testing pyramid, unit testing frameworks, test source layout, and refactoring for testability.
3. Accelerate
❖ Software Delivery Performance Metrics That Matter
❖ Deployment Frequency
❖ Lead Time for Changes
❖ Mean Time To Restore (MTTR)
❖ Change Failure Rate
Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
By Nicole Forsgren PhD, Jez Humble, and Gene Kim
9. Deliver Valuable Software Faster
Problem How TDD/BDD Can Help
Long QA Test Cycles Identify incorrect behavior before release to QA
Rework
Identify incorrect behavior during initial
development so can fix when most familiar with the
code
Disproportionate time spent studying code before
making small change
Confidence tests will identify if change will break
other behavior
Slow and difficult client development Verify API usability and tests as documentation
Slow and difficult to extend and/or add features
Testability strongly correlated with flexibility and
extensibility
10. –Chinese proverb
“To be uncertain is to be uncomfortable, but to be
certain is to be ridiculous.”
12. End-to-End Tests
❖ Developers like it because it offloads
most, if not all, of the testing to others.
❖ Managers and decision-makers like it
because tests that simulate real user
scenarios can help them easily
determine how a failing test would
impact the user.
❖ Testers like it because they often
worry about missing a bug or writing
a test that does not verify real-world
behavior; writing tests from the user's
perspective often avoids both
problems and gives the tester a greater
sense of accomplishment.
Days Left Pass % Notes
1 5%
Everything is broken! Signing in to the service is
broken. Almost all tests sign in a user, so almost
all tests failed.
0 4%
A partner team we rely on deployed a bad build to
their testing environment yesterday.
-1 54%
A dev broke the save scenario yesterday (or the
day before?). Half the tests save a document at
some point in time. Devs spent most of the day
determining if it's a frontend bug or a backend
-2 54%
It's a frontend bug, devs spent half of today
figuring out where.
-3 54%
A bad fix was checked in yesterday. The mistake
was pretty easy to spot, though, and a correct fix
was checked in today.
-4 1%
Hardware failures occurred in the lab for our
testing environment.
-5 84%
Many small bugs hiding behind the big bugs (e.g.,
sign-in broken, save broken). Still working on the
small bugs.
-6 87%
We should be above 90%, but are not for some
reason.
-7 89.54%
(Rounds up to 90%, close enough.) No fixes were
checked in yesterday, so the tests must have
been flaky yesterday.
In Theory In Practice
Source: Mike Wacker on Google Testing Blog
14. –Fred Brooks
“The hardest part of building a software system is
deciding precisely what to build... No other part of
the work so cripples the resulting system if done
wrong. No other part is more difficult to rectify
later”
15. Java Unit Test Frameworks
Strong Points Migrating from JUnit 4
JUnit 5
• Leverages Java 8 lambdas and
functional interfaces
• Modularity, powerful
extensions mechanism
• No support for Runners and
Rules
• Spring tests based on
ThreadLocal usage break
TestNG
• Flexible grouping and running
of tests
• Different ordering to
assertion parameters than
JUnit
Spock
• Data tables with multi-line
strings
• No need for mocking
framework
• Getting used to Groovy-
based DSL
• Additional test source
directory
16. Other Java Test Frameworks
Type Highlights
Mockito Mock Interface injection, Data control, Verification
Spring Test Dependency Inject Spring config doc, Transaction rollback
Arquillian JEE Server Stub Embedded JEE server functionality
Cucumber BDD Gherkin-based communication with SME
Wiremock Service Virtual Lightweight service stubs
Spring Cloud
Contract
Consumer Driven Uses Wiremock under the hood
Hoverfly Service Virtual Full system service virtualization
ArchUnit Architecture Consistent -ilities w/o bureaucracy
18. Test Source Layout
❖ Maven
❖ All in src/test/java or src/test/groovy
❖ Use surefire and failsafe include / exclude to control test execution (Unit = *Test, Integration = *IT)
❖ Define non-default profile for integration tests
❖ Use jacoco-maven-plugin to merge test coverage
❖ Gradle
❖ Utilize gradle-testsets-plugin and don’t link into default build
❖ Separate directories (Unit = src/test/java, Integration = src/integrationTest/java)
❖ Use Jacoco plugin to merge test coverage
❖ Sonar
❖ Automatically merges all test coverage in one metrics as of v6.x
21. When to Unit Test
- Questionable
- Code obvious at first glance
- Coordinating logic
✓ Absolutely Mandatory
✓ Algorithmic logic
✓ Complex code with many
dependencies
Vikas Hazrati on Knoldus blog
22. What to Test
Category Test Category Test
DTO
Serialization, equals,
hashCode
DAO/Repository
Integration tests against
actual datastore
Converters, Parsers,
Serializers, Validators
Source data from test data
parameters or files on class
path
Utility Methods Every branch, edge cases
Routers, Coordinators
Decision logic if any
extracted into methods
Business Rules
BDD specifications, All pairs
testing
JavaEE EJBs, JCAs Use Arquillian REST Controllers
Handling of optional
parameters, Use REST
framework-provided mocks
APIs
Integration tests verifying
wiring and configuration of
framework AOP
UIs Basic wiring, algorithms
Docker Images
Wiring, startup, health
check
Third-party Libraries
Integration of generated
code, Behavior assumptions
23. Data Transfer Object
public class ExampleEntityPKTest {
private final Logger logger = LoggerFactory.getLogger(ExampleEntityPKTest.class);
@Test
public void shouldSerializeAndDeserializeRetainingEquality() {
String employeeId = "99377";
String postingId = "DEC18";
ExampleEntityPK original = new ExampleEntityPK();
original.setPostingId(postingId);
original.setEmployeeId(employeeId);
ExampleEntityPK copy = (ExampleEntityPK)
SerializationUtils.deserialize(SerializationUtils.serialize(original));
assertEquals(original, copy);
assertEquals(original.hashCode(), copy.hashCode());
assertEquals(postingId, copy.getPostingId());
assertEquals(employeeId, copy.getEmployeeId());
}
@Test
public void shouldTreatObjectsWithDifferentValuesAsNotEqual() {
ExampleEntityPK pk1 = new ExampleEntityPK("99377", "JUL19TEST");
ExampleEntityPK pk2 = new ExampleEntityPK("11", "JUL19TEST");
assertEquals(pk1, pk1);
assertFalse(pk1.equals(pk2));
assertFalse(pk1.hashCode() == pk2.hashCode());
assertFalse(pk1.equals(null));
assertFalse(pk1.equals(pk1.getPostingId()));
}
@Test
public void shouldSortByPostingIdAndThenEmployeeId() {
ExampleEntityPK pk1 = new ExampleEntityPK("99377", "JUL19TEST");
ExampleEntityPK pk2 = new ExampleEntityPK(“11", "JUL19TEST");
assertTrue(pk1.compareTo(pk2) > 0);
}
}
24. Data Access Object
@RunWith(SpringRunner.class)
@ContextConfiguration(classes = { DatabaseConfig.class })
@TestPropertySource("/test-application.properties")
public class EmployeeRepositoryITest {
@Autowired
private EmployeeRepository repository;
@Autowired
private DataSource dataSource;
@Test
public void shouldRetrievePageOfEmployees() {
PageRequest pageable = PageRequest.of(0, 32, Sort.Direction.ASC, "employeeId");
Page<Employee> results = repository.findAll(pageable);
assertFalse(results.getContent().isEmpty());
}
@Test
public void shouldRetrieveEmployeeByEmployeeId() {
PageRequest pageable = PageRequest.of(4, 4, Sort.Direction.ASC, "employeeId");
Page<Employee> results = repository.findAll(pageable);
assertFalse(results.getContent().isEmpty());
for (Employee em : results.getContent()) {
Optional<Employee> result = repository.findByEmployeeId(em.getEmployeeId());
assertTrue(result.isPresent());
}
}
@Test
public void shouldRetrieveValuesFromPersonnel() {
Optional<Employee> result = repository.findById(retrieveValidEmployeeId());
assertTrue(result.isPresent());
assertNotNull(result.get().getSeniorityDate());
}
private Long retrieveValidEmployeeId() {
Long result = null;
try (Connection conn = dataSource();
PreparedStatement ps = conn.prepareStatement("SELECT MAX(employee_id) FROM employee")) {
try (ResultSet rs = ps.executeQuery()) {
rs.next();
result = rs.getLong(1);
}
} catch (SQLException cause) {
throw new IllegalStateException(cause);
}
return result;
}
}
27. Business Rules
public class RpnCalculatorStepdefs implements En {
private RpnCalculator calc;
public RpnCalculatorStepdefs() {
Given("a calculator I just turned on", () -> {
calc = new RpnCalculator();
});
When("I add {int} and {int}", (Integer arg1, Integer arg2) -> {
calc.push(arg1);
calc.push(arg2);
calc.push("+");
});
Given("I press (.+)", (String what) -> calc.push(what));
Then("the result is {double}", (Integer expected) -> assertEquals(expected, calc.value()));
Then("the result is {int}", (Integer expected) -> assertEquals(expected.doubleValue(), calc.value()));
Before(new String[]{"not @foo"}, (Scenario scenario) -> {
scenario.write("Runs before scenarios *not* tagged with @foo");
});
After((Scenario scenario) -> {
// result.write("HELLLLOO");
});
Given("the previous entries:", (DataTable dataTable) -> {
List<Entry> entries = dataTable.asList(Entry.class);
for (Entry entry : entries) {
calc.push(entry.first);
calc.push(entry.second);
calc.push(entry.operation);
}
});
}
static final class Entry {
private final Integer first;
private final Integer second;
private final String operation;
Entry(Integer first, Integer second, String operation) {
this.first = first;
this.second = second;
this.operation = operation;
}
}
}
28. All-Pairs Testing
❖ For each pair of input parameters, tests all possible
discrete combinations of those parameters
❖ The most common bugs in a program are generally
triggered by either a single input parameter or an
interaction between pairs of parameters
❖ Software available to generate optimum test data
29. EJB
@RunWith(Arquillian.class)
public class MemberRegistrationTest {
@Deployment
public static Archive<?> createTestArchive() {
return ShrinkWrap.create(WebArchive.class, "test.war")
.addClasses(Member.class, MemberRegistration.class, Resources.class)
.addAsResource("META-INF/test-persistence.xml", "META-INF/persistence.xml")
.addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml")
// Deploy our test datasource
.addAsWebInfResource("test-ds.xml", "test-ds.xml");
}
@Inject
MemberRegistration memberRegistration;
@Inject
Logger log;
@Test
public void testRegister() throws Exception {
Member newMember = new Member();
newMember.setName("Jane Doe");
newMember.setEmail("jane@mailinator.com");
newMember.setPhoneNumber("2125551234");
memberRegistration.register(newMember);
assertNotNull(newMember.getId());
log.info(newMember.getName() + " was persisted with id " + newMember.getId());
}
}
30. Concurrency
public class MicrometerTestApplicationTests {
private Logger logger = LoggerFactory.getLogger(MicrometerTestApplicationTests.class);
@Autowired
private WebTestClient webClient;
@Test
public void threadPoolResponseDistribution() throws BrokenBarrierException, InterruptedException {
int clients = 8;
int runs = 2048;
CyclicBarrier barrier = new CyclicBarrier(clients + 1);
ExecutorService executorService = Executors.newCachedThreadPool();
for (int i = 0; i < clients; i++) {
executorService.submit(new ThreadPoolTestRunner(barrier, runs));
}
barrier.await();
barrier.await();
webClient.get().uri("/actuator/mappings").exchange().expectStatus().isOk();
webClient.get().uri("/actuator/prometheus").exchange().expectStatus().isOk();
FluxExchangeResult<String> stats = webClient.get().uri("/stats/tpool").exchange().returnResult(String.class);
logger.info("Thread Pool Test Result: {}", new String(stats.getResponseBodyContent()));
}
FluxExchangeResult<String> stats = webClient.get().uri("/stats/hystrix").exchange().returnResult(String.class);
logger.info("Hystrix Test Result: {}", new String(stats.getResponseBodyContent()));
}
class ThreadPoolTestRunner implements Runnable {
private CyclicBarrier barrier;
private int runs;
ThreadPoolTestRunner(CyclicBarrier barrier, int runs) {
this.barrier = barrier;
this.runs = runs;
}
@Override
public void run() {
try {
barrier.await();
for (int i = 0; i < runs; i++) {
webClient.get().uri("/tpool/{id}", UUID.randomUUID().toString()).exchange().expectStatus().isOk();
}
barrier.await();
} catch (InterruptedException | BrokenBarrierException ignore) {
fail("improperly configured test");
}
}
}
32. –Andrew Hunt and Dave Thomas
“It’s not at all important to get it right the first
time. It’s vitally important to get it right the last
time.”
33. Testability Aphorisms
❖ Static getInstance involving a remote connection is EVIL
❖ PowerMock provides powerful functionality which should
NEVER be used
❖ PowerMock required == Redesign required
❖ God objects need the Single Responsibility Principle
applied
❖ In method section comments should be sub-method names
❖ Minimize leaky abstractions
34. –powermock.github.io
“Please note that PowerMock is mainly intended
for people with expert knowledge in unit testing.
Putting it in the hands of junior developers may
cause more harm than good.”
37. Test Automation Anti-Patterns
1. Mushrooming
• Stage 1: Small and Localized
• Stage 2: Generalization
• Stage 3: Staffing
• Stage 4: Development of Non-Core Features
• Stage 5: Overload
2. The Competition
• Variety 1: Different teams use different frameworks
• Variety 2: Different teams feed requirements to enterprise effort and one team gets favored
• Variety 3: Mushrooming at stage 5 so team starts new solution at stage 1
3. The Night Time Fallacy
• Test Automation Truism: Machines create work for more testers
• The Tragedy of the Commons: Five teams each assume 12 hours to test time: 5 teams * 12 hours = 60 hours
4. Going for the Numbers
• Point and click creation to up numbers but poorly engineered for resiliency
5. The Sorcerer’s Apprentice Syndrome
• Mimicking human actions sound straight forward, but is sometimes hard to accomplish programmatically and is frequently
inefficient.
The Pathologies of Failed Test Automation Projects - Michael Stahl, July 2013
39. Test Data Strategies
❖ Use actual production data
❖ PII, PCI, HIPAA data must be masked
❖ Use queries to get valid test values
❖ Use automatic rollback for data changing tests
❖ Delete and refresh on a regular basis
❖ Generate fake data
❖ Domain specific generators
❖ Generate per test
❖ Import reference data
❖ Periodic cleanup
41. Future of Enterprise QA
❖ Embedded in delivery team
❖ Gherkin specification development
❖ Exploratory testing
❖ Risk-based test prioritization
❖ Independent group
❖ AI-assisted intelligence
❖ Customer experience optimization and insights
❖ Robotic process automation
42. –Jerry Weinberg
“You can be a great tester if you have
programming skills. You can also be a great tester
if you have no programming skills at all. And, you
can be a lousy tester with or without programming
skills. A great tester will learn what skills she needs
to continue to be great, in her own style.”
43. Testing in Production Methodologies
❖ A/B Testing (aka Online Controlled Experimentation)
❖ Some percent of users of a website or service are unbeknownst to them given an alternate
experience (a new version of the service). Data is then collected on how users act in the old versus
new service, which can be analyzed to determine whether the new proposed change is good or not.
❖ Ramped Deployment
❖ Using Exposure Control, a new deployment of a website or service is gradually rolled out. First to a
small percentage of users, and then ultimately to all of them. At each stage the software is
monitored, and if any critical issues are uncovered that cannot be remedied, the deployment is
rolled back.
❖ Shadowing
❖ The system under test is deployed and uses real production data in real-time, but the results are not
exposed to the end user.
Source: https://blogs.msdn.microsoft.com/seliot/2011/06/07/testing-in-production-tip-it-really-happensexamples-from-facebook-amazon-google-and-microsoft/
44. –Isaac Asimov
“The most exciting phrase to hear in science, the
one that heralds discoveries, is not ‘Eureka!’ but
‘Now that’s funny…”