My talk at SeleniumConf 2020 Virtual on how to treat your end-end tests and framework as first class citizens.

1. End-End tests as
first class citizens
Abhijeet Vaikar

2. Abhijeet Vaikar
Senior Software Engineer Co-organizer
Testing software and writing test code since 8 years
Maintainer
abhijeetvaikar @AbhijeetVaikar

3. Poll Time!
Are you involved with developing, maintaining
and enhancing an in-house Selenium/Appium
based test framework and test code at your
work?

4. Value from e2e automated tests
Detect bugs Frequent feedback Faster delivery cycles Flexibility
Reliable testing Reduced costs

5. “My tests were working until yesterday. Today they are not”
“Today’s regression run for Android didn’t execute at all”
“Tests are failing locally, did we change something in the framework?”
“Slack notifications are not being sent for test runs. What happened?”
“Someone accidentally checked-in sensitive values in test code”
Test Engineering team
“The branch name in the regression jenkins job has been changed.”
“Sign up tests are failing due to NullPointerException”
“What does this new method in the framework do?”

6. Ensure quality of your e2e test
framework and test code

7. Reliable Ease of use
Stable
What “Quality” means for test framework and scripts?

8. What “Quality” means for test framework and scripts?
**Preventing issues/bugs in your test framework / test code earlier to avoid surprises in your actual runs

9. Production Code
Test Code
1st
2nd
How do we ensure quality of test framework and test
code?

10. Production
Code
How do we ensure quality of test framework and test
code?
Test
Code
1st
2nd

11. E2E test automation @ Carousell - Overview
Development
environment
Version Control Continuous
Integration
Dependency
Management
Automation libs
Test-case
Management
CarouFarm
Devices Execution Env Test Notifications App/Log/video/
test account storage

12. E2E test automation @ Carousell - Overview
● FFT (Fast Feedback Tests)
● Daily regression
● Sanity runs

13. Principles and practices we adopted
Code Process & People

14. All changes on dedicated branch

15. Automated checks on PR
● Is code formatted?
● Does code compile?
● Do unit/integration tests pass?
● Does code pass SonarCloud quality
gate?
● Have reviewers approved your PR?
❌ ✅
Block PR until checks fixed
Good to merge
in master
Jenkinsfile

16. Automated checks on PR - Code formatting
Is code formatted?
Automating code format checks using Google Java Style Guide

17. Automated checks on PR - Static Code Analysis
Does code pass SonarCloud quality gate?
Using static code analysis tool SonarCloud to analyse code against industry standard quality gates

18. SonarCloud quality gates:
● Reliability
● Security
● Maintainability
● Coverage
● Duplications
Automated checks on PR - Static Code Analysis

19. Automated checks on PR - Unit/Integration tests
https://twitter.com/maaretp/status/1166705548450062336

20. Automated checks on PR - Unit/Integration tests
Driver management Test Data handlers
Cloud service
handlers
Testcase
management
system handlers
Configuration
handlers
All classes that
support correct
functioning of the
test framework
Page Objects
Test classes /
Feature files / Test
specs
Reporting handlers
Write unit/integration
tests
Don’t write
unit/integration tests
A test framework is also a software
product used by engineers.

21. Automated checks on PR - Unit/Integration tests

22. @RunWith(MockitoJUnitRunner. class)
public class LoginServiceTest {
@Mock private Dependency dependency;
private LoginService loginService;
@Before
public void setUp() {
System.setProperty("env", "prod");
}
@Test
public void testLoginGetNewTokenSuccess() {
}
@Test
public void testLoginGetExistingTokenSuccess() {
}
Automated checks on PR - Unit/Integration tests

23. Automated checks on PR - Unit/Integration tests
/** Unit Test for Class TM4J Service */
@RunWith(MockitoJUnitRunner. class)
public class Tm4jServiceTest {
private TM4JService tm4jService;
@Before
public void init() {
System.setProperty(PropertyConstants. TM4J_URL, "http://tm4j-url" );
System.setProperty(PropertyConstants. TM4J_TEST_CYCLE_ID , "TEST-AAA");
System.setProperty(PropertyConstants. TM4J_PROJECT_ID , "TEST");
}
@Test
public void testUpdateSuccessful() {
}
@Test
public void testUpdateFailed() {
}

24. Automated checks on PR - Unit/Integration tests
Selenium has
unit tests too!

25. https://twitter.com/BugHunterSam/status/1184317541239283712
Code Reviews

26. Code Reviews

27. Code Reviews

28. Helps a contributor ensure
that they have done all the
due diligence before trying
to create the PR or merge
the PR in the master branch
Pull Request checklist

29. Splitting framework code and test code into 2 separate
projects
Before After
Test Framework
&
Test Code
Test
Framework
Test Code

30. Splitting framework code and test code into 2 separate
projects
Test Code
Test
Framework
<parent>
<groupId>group-name</groupId>
<artifactId>framework</artifactId>
<version>20.08.20.170019</version>
</parent>

31. Adopting Jenkins pipeline instead of conventional job
configuration (pipeline as code)
Before After
● All configuration on Jenkins
● Difficult to track changes done
in jobs
● All configuration in a Jenkinsfile
● Jenkinsfile is version controlled in the project so
it goes through review before being used.

32. Test environment for test runs
Isolating real test runs from “debug” test runs
● Separate CI jobs for testing out changes to framework or test suite code.
● Separate TM4J test cycle for testing integrations
● Separate DB instance to test database integrations
Intent: Do not disturb the production tests no matter what.

33. Faster analysis of issues when things break

34. Faster analysis of issues when things break

35. Good coding principles
● Keeping tests short and atomic.
● Separation of concerns/responsibilities using:
○ Feature files (Describing user acceptance)
○ Step classes calling pageobjects & performing assertions.
○ Page Objects (only layer which deals with driver)
● Avoiding repetition (DRY) using abstraction wherever necessary.
● Javadocs for public methods in framework code.
● Efficient locators.
● Avoiding shared state between tests (for eg. use of static methods)

36. Good coding principles
● Knowing when to catch exceptions and when to let them fail tests.
● Being smart about waiting mechanisms.
● Using a logging framework instead of System.out.println().
● Not checking in sensitive data in the code repository. Using git-secrets.
● Removing dead code
● Using boy scout rule - Leave your code better than you found it.

37. Team collaboration, communication & planning
● Planning strategic changes to framework well. RFCs are a good way to do it.
● Thinking about possible failures when designing solutions for test framework and integrations.
● Prioritising high impact tests for end-end UI coverage and not automating all your tests on the UI.
Distribute tests across all the layers consciously.
● Conducting sprint review sessions where any significant work done is demonstrated and made
open to questions and discussions.
● Pair programming & pair brainstorming sessions.
● Encouraging an open mindset when it comes to asking questions & giving feedback on code.
● Creating JIRA tickets any time we come across tech debt.
● Proactively planning to fix tech debt regularly.

38. Key takeaways & learnings
Write unit/integration tests
for core components and
integrations that make (and
break) your framework &
test code

39. Key takeaways & learnings
Create build pipeline for
your test framework and
test code using CI

40. Key takeaways & learnings
Automate checks at PR level
to avoid issues creeping into
your daily test runs that
matter
❌ ✅

41. Key takeaways & learnings
Conduct code reviews
religiously

42. Key takeaways & learnings
Adopt good coding principles and design patterns
to ensure making changes in code is easy
DRY
Images: https://team-coder.com/solid-principles/, https://thevaluable.dev/dry-principle-cost-benefit-example/

43. Key takeaways & learnings
Establish a practice of open
& frequent communication in
the team driven by
retrospection and
continuous improvement.

44. Thank you!