SlideShare ist ein Scribd-Unternehmen logo
1 von 34
Refactoring Away from 
Test Hell 
@alastairs 
alastair@alastairsmith.me.uk 
http://www.codebork.com/
The Graduate
The Junior
The Software Engineer
Tests are your parachute
•Getting to Green 
•Use Mocks Wisely 
•Document the System
Getting to Green
Do what you need to get there 
1. Invest in the infrastructure 
2. Fake the infrastructure 
3. Isolate yourself from the infrastructure
Invest in the infrastructure 
• Add more build agents 
• Run tests in parallel 
• Beefier hardware 
• Fix infrastructure issues
Fake the infrastructure 
• Introduce a seam 
• A place where you can alter behaviour without editing in that place 
• Use Value Objects for your inputs and outputs 
• Try Record-Replay
Record-Replay 
Short-term hack! 
• Fake the seam interface 
• Save requests to disk 
• Respond immediately if a known request 
• Compare artefacts against a known-good version
Isolate yourself from the infrastructure 
• Refactor your seams 
• Introduce mocks
Use Mocks Wisely
Mocking Rules of Thumb 
• Don’t mix mocks and stubs 
• Don’t nest mocks 
• Use strict mocks 
• Only mock types you own
It’s a Unit of Behaviour, Dammit! 
• Test your interface, not your implementation 
• Use Collaboration and Contract tests (http://j.mp/10UFgOS)
[Test] 
public void Moq_Example_With_A_Mock() 
{ 
// Arrange 
var mock = new Mock<IProductRepository>(MockBehavior.Strict); 
mock.Setup(r => r.Save(pen)); // Test fails if not included 
var sut = new AddProductsCommand(mock.Object); 
// Act 
sut.Add(pen); 
// Assert 
mock.Verify(r => r.Save(pen)); // Behaviour Verification 
}
[Test] 
public void Moq_Example_With_A_Stub() 
{ 
// Arrange 
var stub = new Mock<IProductRepository>(MockBehavior.Strict); 
stub.Setup(r => r.GetProduct(1)).Returns(pen); // Pre-condition 
var sut = new AddProductsCommand(stub.Object); 
// Act 
var actual = sut.Get(1); 
// Assert 
var expected = pen; 
Assert.That(actual, Is.EqualTo(expected)); 
}
Document the System
General tips 
• Use Descriptive And Meaningful Phrases 
• Use a defined structure with conventions 
• Make anonymous data obviously anonymous 
var anonymousText = "Anonymous Text"; 
• Use randomised data for large input spaces 
• Keep to a Cyclomatic Complexity of 1
SUT Factory 
• Decouples tests from construction logic 
• Protects you from changes to the constructor signature 
• Consider the Fixture Object pattern 
SUT: System Under Test
Test Builders 
• A Domain-Specific Language in your tests 
• Clarify intent 
• Use for anything with 
• Non-trivial construction 
• Varied setup 
• Test Data 
• The SUT
[Fact] 
public void Placing_An_Order_Adds_The_Order_To_The_Customers_Account() 
{ 
// Arrange 
var customer = new Customer( 
1, // Id 
"Joe", "Bloggs", // Name 
"10 City Road", "Staines", 
"Middlesex", "AB1 2CD" // Address 
); 
var order = new Order(1, customer); 
// Act, Assert: not interesting for this example 
}
[Fact] 
public void Placing_An_Order_Adds_The_Order_To_The_Customers_Account() 
{ 
// Arrange 
var customer = ACustomer() 
.WithGivenName("Joe") 
.WithFamilyName("Bloggs") 
.Build(); 
var order = AnOrder().PlacedBy(customer).Build(); 
// Act, Assert: not interesting for this example 
}
Bob the Builder 
• Maintains Fluent syntax 
• Reduces boilerplate 
• Install-Package BobTheBuilder
using BobTheBuilder; 
[Fact] 
public void Placing_An_Order_Adds_The_Order_To_The_Customers_Account() 
{ 
// Arrange 
var customer = A.BuilderFor<Customer>() 
.WithGivenName("Joe") 
.WithFamilyName("Bloggs") 
.Build(); 
var order = new Order(1, customer); 
// Act, Assert: not interesting for this example 
}
using BobTheBuilder; 
[Fact] 
public void Placing_An_Order_Adds_The_Order_To_The_Customers_Account() 
{ 
// Arrange 
Customer customer = A.BuilderFor<Customer>() 
.With(givenName: "Joe", 
familyName: "Bloggs"); 
var order = new Order(1, customer); 
// Act, Assert: not interesting for this example 
}
Assertions 
• Keep to one logical assertion per test 
• Aim for Equality assertions 
• Write custom constraints / matchers 
• Consider the Fixture Object pattern
Test-Specific Equality 
• IEqualityComparer<T> 
• Resemblance 
• Likeness
AutoFixture 
• Automates many of the techniques described here 
• Install-Package AutoFixture 
• Support for all popular mocking frameworks
[Test, AutoData] 
public void IntroductoryTest(int expectedNumber, 
MyClass sut) 
{ 
int result = sut.Echo(expectedNumber); 
Assert.Equal(expectedNumber, result); 
}
• Getting to Green 
• Do what you need to get there 
• Use Mocks Wisely 
• Don’t nest mocks 
• Use strict mocks 
• Write Contract and Collaboration Tests 
• Document the System 
• SUT Factory 
• Test Builders 
• Assertions
References and Resources 
http://j.mp/refactoringTestHell 
xUnit Patterns (Gerard Meszaros): http://j.mp/11bV67X 
Integration Tests are a Scam (J B Rainsberger): http://j.mp/10UFgOS 
Boundaries (Gary Berhardt): http://j.mp/1o18XbT 
Advanced Unit Testing (Mark Seemann, Pluralsight): http://j.mp/psAdvUT
Any questions? 
@alastairs 
alastair@alastairsmith.me.uk 
http://www.codebork.com/

Weitere ähnliche Inhalte

Was ist angesagt? (8)

Racing car katas Ⅲ - Static Cling
Racing car katas Ⅲ - Static ClingRacing car katas Ⅲ - Static Cling
Racing car katas Ⅲ - Static Cling
 
RSpec: What, How and Why
RSpec: What, How and WhyRSpec: What, How and Why
RSpec: What, How and Why
 
Validation for APIs in Laravel 4
Validation for APIs in Laravel 4Validation for APIs in Laravel 4
Validation for APIs in Laravel 4
 
Unit testing js
Unit testing jsUnit testing js
Unit testing js
 
Unit tests and TDD
Unit tests and TDDUnit tests and TDD
Unit tests and TDD
 
Effective Readable unit testing with junit5
Effective Readable unit testing with junit5Effective Readable unit testing with junit5
Effective Readable unit testing with junit5
 
Rspec presentation
Rspec presentationRspec presentation
Rspec presentation
 
Rspec 101
Rspec 101Rspec 101
Rspec 101
 

Ähnlich wie Refactoring Away from Test Hell

Grails Spock Testing
Grails Spock TestingGrails Spock Testing
Grails Spock Testing
TO THE NEW | Technology
 

Ähnlich wie Refactoring Away from Test Hell (20)

Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014
Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014
Breaking Dependencies To Allow Unit Testing - Steve Smith | FalafelCON 2014
 
Breaking Dependencies to Allow Unit Testing
Breaking Dependencies to Allow Unit TestingBreaking Dependencies to Allow Unit Testing
Breaking Dependencies to Allow Unit Testing
 
CBDW2014 - MockBox, get ready to mock your socks off!
CBDW2014 - MockBox, get ready to mock your socks off!CBDW2014 - MockBox, get ready to mock your socks off!
CBDW2014 - MockBox, get ready to mock your socks off!
 
Unit testing and mocking in Python - PyCon 2018 - Kenya
Unit testing and mocking in Python - PyCon 2018 - KenyaUnit testing and mocking in Python - PyCon 2018 - Kenya
Unit testing and mocking in Python - PyCon 2018 - Kenya
 
Java EE Revisits Design Patterns
Java EE Revisits Design PatternsJava EE Revisits Design Patterns
Java EE Revisits Design Patterns
 
Java EE revisits design patterns
Java EE revisits design patternsJava EE revisits design patterns
Java EE revisits design patterns
 
Java EE revisits design patterns
Java EE revisits design patterns Java EE revisits design patterns
Java EE revisits design patterns
 
SE2016 - Java EE revisits design patterns 2016
SE2016 - Java EE revisits design patterns 2016SE2016 - Java EE revisits design patterns 2016
SE2016 - Java EE revisits design patterns 2016
 
Just Do It! ColdBox Integration Testing
Just Do It! ColdBox Integration TestingJust Do It! ColdBox Integration Testing
Just Do It! ColdBox Integration Testing
 
VT.NET 20160411: An Intro to Test Driven Development (TDD)
VT.NET 20160411: An Intro to Test Driven Development (TDD)VT.NET 20160411: An Intro to Test Driven Development (TDD)
VT.NET 20160411: An Intro to Test Driven Development (TDD)
 
Chef Cookbook Workflow
Chef Cookbook WorkflowChef Cookbook Workflow
Chef Cookbook Workflow
 
Quick tour to front end unit testing using jasmine
Quick tour to front end unit testing using jasmineQuick tour to front end unit testing using jasmine
Quick tour to front end unit testing using jasmine
 
Prototype design patterns
Prototype design patternsPrototype design patterns
Prototype design patterns
 
Ch11lect1 ud
Ch11lect1 udCh11lect1 ud
Ch11lect1 ud
 
Agile Software Testing the Agilogy Way
Agile Software Testing the Agilogy WayAgile Software Testing the Agilogy Way
Agile Software Testing the Agilogy Way
 
Building unit tests correctly
Building unit tests correctlyBuilding unit tests correctly
Building unit tests correctly
 
Grails Spock Testing
Grails Spock TestingGrails Spock Testing
Grails Spock Testing
 
Advanced Java Testing @ POSS 2019
Advanced Java Testing @ POSS 2019Advanced Java Testing @ POSS 2019
Advanced Java Testing @ POSS 2019
 
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017
North Virginia Coldfusion User Group Meetup - Testbox - July 19th 2017
 
New types of tests for Java projects
New types of tests for Java projectsNew types of tests for Java projects
New types of tests for Java projects
 

Kürzlich hochgeladen

Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
masabamasaba
 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
VictoriaMetrics
 
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
masabamasaba
 
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
masabamasaba
 
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
masabamasaba
 

Kürzlich hochgeladen (20)

Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
 
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
 
%in Harare+277-882-255-28 abortion pills for sale in Harare
%in Harare+277-882-255-28 abortion pills for sale in Harare%in Harare+277-882-255-28 abortion pills for sale in Harare
%in Harare+277-882-255-28 abortion pills for sale in Harare
 
Announcing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK SoftwareAnnouncing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK Software
 
VTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnVTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learn
 
tonesoftg
tonesoftgtonesoftg
tonesoftg
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
 
%in kempton park+277-882-255-28 abortion pills for sale in kempton park
%in kempton park+277-882-255-28 abortion pills for sale in kempton park %in kempton park+277-882-255-28 abortion pills for sale in kempton park
%in kempton park+277-882-255-28 abortion pills for sale in kempton park
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
 
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
 
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems -The Print[A] Example- A Comprehension AidDirect Style Effect Systems -The Print[A] Example- A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
 
%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand
 
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
 
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
 
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
 
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdfMicrosoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
 
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
 
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
 

Refactoring Away from Test Hell

  • 1. Refactoring Away from Test Hell @alastairs alastair@alastairsmith.me.uk http://www.codebork.com/
  • 5. Tests are your parachute
  • 6.
  • 7. •Getting to Green •Use Mocks Wisely •Document the System
  • 9. Do what you need to get there 1. Invest in the infrastructure 2. Fake the infrastructure 3. Isolate yourself from the infrastructure
  • 10. Invest in the infrastructure • Add more build agents • Run tests in parallel • Beefier hardware • Fix infrastructure issues
  • 11. Fake the infrastructure • Introduce a seam • A place where you can alter behaviour without editing in that place • Use Value Objects for your inputs and outputs • Try Record-Replay
  • 12. Record-Replay Short-term hack! • Fake the seam interface • Save requests to disk • Respond immediately if a known request • Compare artefacts against a known-good version
  • 13. Isolate yourself from the infrastructure • Refactor your seams • Introduce mocks
  • 15. Mocking Rules of Thumb • Don’t mix mocks and stubs • Don’t nest mocks • Use strict mocks • Only mock types you own
  • 16. It’s a Unit of Behaviour, Dammit! • Test your interface, not your implementation • Use Collaboration and Contract tests (http://j.mp/10UFgOS)
  • 17. [Test] public void Moq_Example_With_A_Mock() { // Arrange var mock = new Mock<IProductRepository>(MockBehavior.Strict); mock.Setup(r => r.Save(pen)); // Test fails if not included var sut = new AddProductsCommand(mock.Object); // Act sut.Add(pen); // Assert mock.Verify(r => r.Save(pen)); // Behaviour Verification }
  • 18. [Test] public void Moq_Example_With_A_Stub() { // Arrange var stub = new Mock<IProductRepository>(MockBehavior.Strict); stub.Setup(r => r.GetProduct(1)).Returns(pen); // Pre-condition var sut = new AddProductsCommand(stub.Object); // Act var actual = sut.Get(1); // Assert var expected = pen; Assert.That(actual, Is.EqualTo(expected)); }
  • 20. General tips • Use Descriptive And Meaningful Phrases • Use a defined structure with conventions • Make anonymous data obviously anonymous var anonymousText = "Anonymous Text"; • Use randomised data for large input spaces • Keep to a Cyclomatic Complexity of 1
  • 21. SUT Factory • Decouples tests from construction logic • Protects you from changes to the constructor signature • Consider the Fixture Object pattern SUT: System Under Test
  • 22. Test Builders • A Domain-Specific Language in your tests • Clarify intent • Use for anything with • Non-trivial construction • Varied setup • Test Data • The SUT
  • 23. [Fact] public void Placing_An_Order_Adds_The_Order_To_The_Customers_Account() { // Arrange var customer = new Customer( 1, // Id "Joe", "Bloggs", // Name "10 City Road", "Staines", "Middlesex", "AB1 2CD" // Address ); var order = new Order(1, customer); // Act, Assert: not interesting for this example }
  • 24. [Fact] public void Placing_An_Order_Adds_The_Order_To_The_Customers_Account() { // Arrange var customer = ACustomer() .WithGivenName("Joe") .WithFamilyName("Bloggs") .Build(); var order = AnOrder().PlacedBy(customer).Build(); // Act, Assert: not interesting for this example }
  • 25. Bob the Builder • Maintains Fluent syntax • Reduces boilerplate • Install-Package BobTheBuilder
  • 26. using BobTheBuilder; [Fact] public void Placing_An_Order_Adds_The_Order_To_The_Customers_Account() { // Arrange var customer = A.BuilderFor<Customer>() .WithGivenName("Joe") .WithFamilyName("Bloggs") .Build(); var order = new Order(1, customer); // Act, Assert: not interesting for this example }
  • 27. using BobTheBuilder; [Fact] public void Placing_An_Order_Adds_The_Order_To_The_Customers_Account() { // Arrange Customer customer = A.BuilderFor<Customer>() .With(givenName: "Joe", familyName: "Bloggs"); var order = new Order(1, customer); // Act, Assert: not interesting for this example }
  • 28. Assertions • Keep to one logical assertion per test • Aim for Equality assertions • Write custom constraints / matchers • Consider the Fixture Object pattern
  • 29. Test-Specific Equality • IEqualityComparer<T> • Resemblance • Likeness
  • 30. AutoFixture • Automates many of the techniques described here • Install-Package AutoFixture • Support for all popular mocking frameworks
  • 31. [Test, AutoData] public void IntroductoryTest(int expectedNumber, MyClass sut) { int result = sut.Echo(expectedNumber); Assert.Equal(expectedNumber, result); }
  • 32. • Getting to Green • Do what you need to get there • Use Mocks Wisely • Don’t nest mocks • Use strict mocks • Write Contract and Collaboration Tests • Document the System • SUT Factory • Test Builders • Assertions
  • 33. References and Resources http://j.mp/refactoringTestHell xUnit Patterns (Gerard Meszaros): http://j.mp/11bV67X Integration Tests are a Scam (J B Rainsberger): http://j.mp/10UFgOS Boundaries (Gary Berhardt): http://j.mp/1o18XbT Advanced Unit Testing (Mark Seemann, Pluralsight): http://j.mp/psAdvUT
  • 34. Any questions? @alastairs alastair@alastairsmith.me.uk http://www.codebork.com/

Hinweis der Redaktion

  1. Developer at Red Gate on SQL Source Control Co-Organiser of DDD East Anglia Founder of Cambridge Software Craftsmanship Community
  2. Large American software company providing remote access solutions to large enterprises. Backwards processes for the late noughties: waterfall, very little unit testing, lots of manual testing. As such, there were only a few thousand tests for ~250kloc, with extremely poor code coverage provided by extremely poor tests. Extremely poor coverage, extremely poor tests. Developers were code monkeys. The system was well-architected but over-engineered. Little interest in improving practices and processes; jobs-worth manager. Towards the end of my time in this job, I came across the Software Craftsmanship meme, and it chimed with some of my frustrations: this wasn’t what I considered a professional organisation. I started learning more about unit testing, and, importantly learning how I learn. But that’s another talk.
  3. ~25k tests (and growing!) for ~250kloc. Developer testing taken seriously, but low levels of proficiency, particularly in the older tests, which formed the bulk of the suite. The product was essentially a relational database system implemented on top of SQL Server. It used a legacy ORM to access the underlying database and .NET Remoting to communicate between client applications and the server application. As .NET Remoting is just RPC, the clients had access to the same objects and object graphs as the server, so there was no separation of concerns across architectural boundaries. As the ORM was a legacy library, the server object graph was tied to the database schema. For the want of a nail, this was the Big Ball of Mud writ large. Because of this implementation, the 25k tests were mostly integration-level tests, albeit to varying degrees. Wide use of mocks and stubs, partly to “make the tests faster” – often it was faster for a developer to write tests against a live object graph than set up the myriad stubs required to make all the calls succeed.
  4. Last November, I moved to Red Gate. The golden sun was warm on my back, and the heavenly host sung: surely here everything would be better? I joined the team working on SQL Source Control, a product that allows you to place your database schema and static data under version control. It builds on SQL Compare, a product that compares two database schemas and generates a SQL script to migrate from one to the other, to place a live database into a version control system. SQL Source Control is ~150kloc, with ~23k tests. The common consensus seems to be “how does it have so much code?!”. (In fact, in a recent Down-Tools week, the team implemented an alternative method for version controlling your database in ~700 lines of code.) The basic design of SQL Source Control is to integrate against the database via SQL Compare, and to integrate with one of a number of Version Control Systems via their client libraries. The state of the tests at Red Gate, when I joined, was worse than at either of my previous companies. The test suites for SQL Compare and SQL Source Control were regularly failing, took not just hours but DAYS to run, and I later found out that neither product had EVER had a fully-green build. What was different, though, was that the teams recognised the problem and were actively addressing it. I’m going to talk about some of the things we have been doing to get ourselves out of this hole, and some general practices that you can introduce while refactoring your tests.
  5. This is what I’ve learned from those experiences: tests need maintenance, care, and attention. Whatever tests you have already are holding your system in place – possibly too rigidly – but are telling you something about your system. We refactored SQL Source Control to take business logic out of the UI so it could be replaced with a shiny new one. When refactoring the tests, we found a bunch of tests we couldn’t refactor to the new architecture and keep them passing. We found later that a significant architectural change made during the refactoring had broken the expected behaviour of the product, and that these tests had been serving as our canary in the coal mine, warning us of the problems to come. Unreliable tests are fatal to the delivery of a software project. A team cannot be confident in the changes they are making without reliable tests – was this test failure caused by a code change or an unrelated issue? If “unrelated issues” cause enough test failures, people stop caring about the test failures and the suite starts to rot. Treat your tests as you would your production code: refactor them, remove duplication of concepts, reduce coupling, and more. Use them to highlight weaknesses in your production code, such as tight coupling.
  6. Who is familiar with these books? Key recommendations from GOOS: Streamline your test code – use structure to explain and share, delegate to subordinate objects Write the test that you’d want to read Listen to your tests Reference a number of times as “GOOS” xUnit Test patterns is a door-step, but a fantastic reference for improving your tests. The patterns are also described at xunitpatterns.com. Some of the patterns are standard features of unit testing frameworks, but many are practical improvements for your tests.
  7. What have we done at Red Gate to improve the reliability of our tests? Heavy reliance on automated GUI testing. Poor unit testing – mostly system/integration tests. Test runs for SQL Compare would regularly take 50+ hours, and always fail somewhere. Test runs for SQL Source Control had to be done overnight, and would always fail somewhere. Neither product had had a green test run in recent history, if ever. Through lots of teamwork and a bit of hackery, the SQL Compare team have got their test runs down to around 15 minutes; the SQL Source Control team have also achieved reliably green tests, and have refactored the code and written new tests that have better coverage than the old tests and run faster.
  8. Can be a progression of options or a set of discrete options.
  9. Granta: 25k tests, total sequential run time ~4.5hrs. Most of theses tests were system/integration tests written against an in-process implementation of the server. More agents to support more concurrent builds. Parallelising tests reduced total execution time: parallelising the pipeline cut the time down to ~2hrs. The tests for both SQL Compare and SQL Source Control are mostly integration-level tests, and were dogged by the flakiness of the virtual environment in which they ran. We worked with our IT team to resolve problems with reliability of Virtual Machines, network connections, and more. On the SQL Source Control team, we would come into the office each morning and find the tests were red. Our testers would spend a couple of hours digging through the failures and find that it was a VM that had failed to respond, or a SQL Server that couldn’t be found. Increasing connection timeouts, etc., helped some, but not enough to make the tests reliable, and of course counterproductively slowed down the runs. This was particularly frustrating for the GUI tests written by the testers to automate much of their repetitive workload. Part of the problem was the IT team wasn’t aware that Development had important VMs running on our TESTNET, and so the infrastructure underlying that portion of the network was deliberately left to seed. There were regular issues with DNS name resolution, DHCP IP address allocation, and the VMs simply becoming unresponsive due to overloaded hardware. We even uncovered a memory leak in Hyper-V while troubleshooting these problems! To support Development, the IT team beefed up the hardware underlying the network, spent time diagnosing the configuration problems causing the DHCP and DNS errors, and yes, worked with Microsoft on the Hyper-V memory leak.
  10. A seam is a prerequisite to all improvements: you can’t test unless a seam is in place to break the dependencies. There are no magic bullets. Seams are important because they allow you to make decisions about the system and its behaviour: much like a seam on an item of clothing, it’s where the parts come together to make the whole. The shape of the seam matters: you can stitch short sleeves, long sleeves or no sleeves to a t-shirt body to make different garments, but you can’t stitch trouser legs onto a jumper. More technically, a seam represents an interface, and when you’re communicating with an interface, you have options. The Value Object pattern is a really powerful way of passing data around. It’s really easy to pass an int or a string to a method, but it’s hard to pass a class with lots of dependencies and interactions to a function. Create small, light-weight Value Objects to represent data concepts in your application: these are just as easy to pass around as ints and strings, and represent a more cohesive concept than fat classes. Think about testing addition, for example: this is easy, right? This is because addition is easy, right? Wrong: this is easy because addition takes values and returns a value. Refactor your code to do the same, and you’ll have a good time. Try spiking the work to sell the benefits; you’d be surprised how far you can get in an afternoon if you focus your spike clearly. A couple of devs on the SQL Compare team managed a PoC in an afternoon. It also allowed them to quantify the work required: they had an interface of m methods, and it took n hours to complete the single method in the PoC. Therefore an estimate of m*n hours is reasonable for completing the work. Most of all, don’t worry about the design of the seam at this stage. You need to cleave your code with an axe, not touch it up with a chisel.
  11. You’ll remember from my introduction that SQL Compare is a tool for comparing the schema of two databases and seeing where they differ; the resulting script can be used to deploy updates to a database. As such, it integrates fairly tightly with SQL Server, and needs to support many versions of that product, all the way back to SQL Server 2000! SQL Compare had a few problems with the way it was accessing SQL Server. At its heart, it essentially does SELECT * FROM [sys.tables], and these calls were littered around the code base. As a result there were issues with, e.g., casting, receiving `null` when `DBNull` was expected (or vice versa) and more. It also has to be able to handle columns that might not exist in the version of SQL Server Compare is currently connected to, including assigning a default value. All of this, previously dotted around the consumers of the database information, was pushed into the readers of the database information. With that seam in place, they were able to create a fake implementation of the seam interface. This fake implementation is what has worked so well for the team: it behaves in a manner similar to an HTTP caching proxy, recording each request it receives and issuing the appropriate response immediately if it receives one it has already seen. As a result, it is able to learn new request/response pairs automatically. Contrast this with an approach using mocks: each request would need to be specified in each test, and each response defined too. Each time a new request was added or encountered, the appropriate tests would need to be updated with the new requests and responses. It would be an on-going effort to maintain the mocks. A further technique employed in this situation is to maintain a known-good version of the artefacts generated by the tests and use it for comparison in the tests. SQL Compare generates deployment scripts for applying a change to a database, which can either be saved or executed. Within the test suite, the Compare tests have a known-good SQL script for applying the changes created during the test run; if the result matches this known-good script, we don’t need to run this script against the database to see if it’s good because we already know that it is. This approach has worked well for SQL Compare: the tests are now green, and reliably so. However, it’s a short term hack: the tests are fast and green, but are not Good Tests™, and so are not completely trustworthy. For example, they didn’t fix all of the non-determinism in the tests, not least because non-determinism was found in the product. What they have done is provide a platform for further work: the tests are now fast enough to fix. They have also had the side effect of clarifying the architecture of the product, defining different contexts representing the phases of the SQL Compare workflow and process, and how they should communicate.
  12. Properly isolating yourself from your infrastructure is hard – it requires care, attention, and discipline, and often the required speed of delivery is at odds with these principles (ideals?). This is why we end up with big balls of mud, logic in the wrong places, etc. It is, however, perhaps easier in some respects to refactor a system to a new design than it is to get the design correct in the first place. Code Complete describes software system design as a “Wicked Problem”, that is, one that is difficult or impossible to solve correctly the first time because of incomplete, contradictory or changing requirements, and solving one aspect of it may reveal or create other problems. Does this sound familiar to you?  However, wicked problems become easier to solve with repeated attempts: you learn more about the domain of the problem and enrich your understanding. This is why Agile works. Given the nature of software design as a wicked problem, it should be easier to refactor a system to a better design than to write it well in the first place. We have knowledge locked in the team’s collective head, in the test suite causing you nightmares, and in other places too. With the seams in place from the earlier stages, you can now start to refactor them to be shaped more correctly for your application. Perhaps you need an API layer over your abstraction. In SQL Source Control, we took this approach with our abstractions over the Version Control Systems with which we integrate: it wasn’t enough to abstract away the individual version control systems behind an adapter interface, we needed to abstract away the version control operations themselves. The reason being SQL Source Control uses the SVN workflow, which doesn’t always fit well with other VCS workflows like those of TFS and Git. With these refactorings completed, you should be in a good place to use mocks in the manner in which they were intended.
  13. Mocks are for behaviour verification: Moq: mock.Verify(x => x.Foo(1, 2)); NSubstitute: substitute.Received().Foo(1, 2); Stubs are for pre-conditions: they define the environment in which your production code executes Moq: Mock.Setup(x => x.Foo).Returns(true); NSubstitute: substitute.Foo().Returns(true); I’ll now break my own rule and use the word “mock” to mean “mocks and stubs” because life’s too short  Nested mocks are the root of most test brittleness: they are a side effect of the coupling in your code base, so you need to change your production code to fix your test code. At Granta, the core concepts of the system were all tightly coupled together, which meant multiple serious violations of the Law of Demeter and deeply-nested mocks and stubs. RhinoMocks compounded this for us, as it auto-mocks members of a mock, saving us some manual mocking code but making that specific implementation more rigid with each test we wrote. Strict mocks will highlight issues in your code, such as deep coupling, that make your code harder to test. They cause an exception to be thrown if an un-mocked member is accessed during the execution of the test. Since mocks are the thing being tested, and Stubs are the context in which your application executes, you really want an exception to be thrown and the test to fail if something unexpected happens. It is important to only mock types which you own: that is to say, types that belong to the component under test. If you’re testing an aspect of your business logic, you might mock an interface over your persistence mechanism, but that interface should be a part of the business logic component under test (Dependency Inversion). You should never mock types from third-party libraries, such as an ORM or an API client.
  14. By the interface, we really mean the pre- and post-conditions of the system under test. What’s the interface of an AddToBasket() operation? Basket is empty when created Items are returned from the basket in the order in which they were added Totals the cost of all items Fails when removing a non-existent item Collaboration tests are the consumer’s side of the seam interface. They answer questions about the consumer’s collaboration with the dependency, like “do I call the dependency correctly?”, and “how do I deal with the responses returned?”. Contract tests are the mirror of collaboration tests, testing the implementation side of the seam interface. They answer questions about the implementation’s conformation to the interface’s contract. There should be one Contract test written for each Collaboration test written.
  15. Tests should describe the behaviour of the system, serve as examples of usage of an API, etc. When written well, they can serve as the specification for the system, and verify that the system meets its specification. As such, they should be written as clearly and unambiguously as possible. GOOS: “Write the test that you would like to read”. Executable specifications is a really powerful technique, and one to be taken seriously. In refactoring SQL Source Control, our tests have provided us with an early warning of breaking changes, and are now pointing out areas of discrepancy between the different VCSs we support. For example, we know – and have tests to prove – that support for Perforce is not as complete as TFS; further more, we can run those tests in future to prove that we have implemented the missing feature in a consistent way with the VCSs that enjoy first-class support.
  16. Keep your test code DAMP. Defined structure with conventions e.g. name your variables according to their role in the test: sut, expected, actual/result place your expectation definition in the Assert phase rather than the Arrange phase If you’re writing a function that takes, e.g., an int as an input, use a random number on each test run to get coverage over a larger space. This can give you much greater confidence that the method is correct without worrying about having to test the entire infinite space of Natural numbers, loop over a set of input values, etc. Cyclomatic Complexity is a good rule of thumb for the complexity of a method, and is easily calculated. Load up a method, start with a score of 1, and then add a further 1 for every if, else or loop you encounter. A cyclomatic complexity of 1 for test code is a useful benchmark to keep to, because your test code has to be incontrovertibly simple: we don’t want to have to test our tests! You can use tools like NDepend to enforce a particular cyclomatic complexity score on your code, which might be a useful thing to do in your test suite if your team are unwilling to stick to it naturally. The following patterns are equally applicable to brownfield test suites via refactoring as they are to greenfield test suites. I recommend you apply them to your existing test suites in the order presented.
  17. Constructor signature changes are one of the most time-consuming, dull things to do if you have lots of tests calling that class’s constructor. Do not call the SUT’s constructor directly, instead use a Factory Method to isolate the dependency on the constructor to a single place. If you’ve written more than a handful of unit tests using this pattern, you’ll know that the number of SUT Factory Methods can quickly explode for even a small number of constructor parameters, often in a combinatorial manner. In this situation, try applying the following Test Builder pattern to your SUT Factories. Also consider whether your SUT is doing too much. The Fixture Object pattern is an extension to this and other techniques, which helps move duplicated concerns from the test methods and fixture into a new object. Common setup steps can be defined as methods on a Fixture Object, it can be a place to keep your SUT Factory Method, and it generally helps you move logic out of your test method and place it behind meaningful member names.
  18. Test Builders are the simplest technique for making your tests more DAMP. They also make your tests more resilient to change by separating out the construction of test data from its usage in the tests. As described in GOOS, Test Builders define a highly readable Fluent API for constructing objects. Let’s see an example, before and after.
  19. Downsides: Not clear which parameter is which (although can be mitigated to an extent with C#’s Named Parameters) Tightly couples the test to the constructor signature of Customer Every argument the Customer constructor defines must be specified, including those we don’t care about
  20. This test doesn’t care about the address or the ID, so they can be omitted (the Builder defines some default values for these constructor parameters). This test is much more readable now we’ve removed the noise Can write higher-level Builder methods to match your domain
  21. There are some downsides to implementing Test Builders manually, particularly repetitive boilerplate code in the Builders themselves. Tried automating the writing of them with R# which helped a bit, but there’s still a lot of code added (= extra maintenance burden). Decided a library would be useful. Additional features: alternative syntaxes, implicit casting from Builder to built type, support for primitive and complex types, support for constructor arguments. Requires .NET 4.0 because it depends on the DLR. Try NBuilder if you’re stuck on an earlier version of .NET.
  22. Alternative syntaxes: omit the call to `Build()` if the compiler can figure out you need a cast Use named parameters to the `With()` method
  23. One logical assertion often means one assertion statement, but it need not. It’s like the Single Responsibility Principle applied to tests: it should only test one thing, but if you need multiple assertion statements to test that one thing, knock yourself out. However, the more assertion statements there are, the less cohesive your logical assertion is. Aiming for Equality assertions is a good guideline, as the majority of assertions can be expressed as Equalities. Be careful of Equality semantics. Custom assertion methods can be used to compose your individual assertion statements into a single statement. If not written as framework-level constraints or matchers, these can be added to the Fixture Object to keep your Test Fixture tidy.
  24. Achieving Equality assertions can required a few tricks and hoops. You don’t want to pollute your production code with test-specific code, particularly if the equality semantics for your tests are different from those in your production code. These techniques will help get over those bumps. Resemblance: test-specific derived class that exists solely to implement equality. Useful particularly in cases where you want to match on some properties of the object but not others (such as a random GUID). Likeness: an automated Resemblance, created via Reflection, that matches properties according to name and type. It’s particularly useful if you’re testing equality of types in a mapping across a port or layer in your architecture. As such, it has a source type and a destination type; as with Resemblances, you can opt to exclude certain properties from the equality comparison.
  25. Particularly if you’re interested in the Likeness and Resemblance techniques, AutoFixture is for you. It will
  26. Simple AutoFixture example: it has taken care of the Arrange phase of the test by parameterising the test and using `AutoData`.
  27. We’ve covered techniques for getting your test suite green, including investing in in the infrastructure, and the benefits of writing your own fakes before turning to mocks directly. We’ve discussed ways of using mocks wisely, including avoiding nested mocks at all costs, using strict mocks, and the necessary duality of testing both sides of an interface using Contract and Collaboration tests. We’ve looked at practical approaches to refactoring your test code to make them more readable, including test builders, the SUT Factory, and custom assertions.
  28. ? I can’t get investment for the tests/there’s not enough time/the business doesn’t understand. Martin Fowler, “If you can’t change your organisation, change your organisation!”. We’ve had success with the ADKAR model for change managements: Awareness, Desire, Knowledge, Ability and Reinforcement.