Test Reporting Framework are the ultimate utilities reached by very few companies in the world likes of google & facebook. It require significant investment in terms money and resources. To optimize on both we tried the approach of extending Jenkins for the same purpose. We like to share our experiences with respect to this journey. We will talk about different cahllenges we faced in doing so and how we solved them. We strongly believe that other startup can use our experience and get benefit from this immensely.

1. JENKINS AS THE TEST REPORTING
FRAMEWORK
Nitin Sharma, SDET @Komli Media

2. SPEAKER BIO
• Working as SDET with Komli Media
• An automation geek and always try to do innovative
things and blog about it and having 4 years of
experience in Software Industry

3. WHY TEST REPORTING FRAMEWORKS ?
 To help with informed decision, whether to go
ahead with production deployment of the build
 To analyze whether sufficient testing being done
 What are code changes ?
 What are the corresponding automation changes ?
 TC Added, Deleted & Modified
 What is the test automation run trend ?
 Trend on Release basis

4. REPORTING FRAMEWORK -
REQUIREMENTS
 Web based Tool
 Cost Effective
 Easy to maintain
 Integration with TC Execution Frameworks

5. TEST REPORTING FRAMEWORKS
 TestRail – Manage all your testing efforts in a
central place and modern user interface.
 Xqual – Provide very fine interface to manage test
reporting framework.
 Test Collab – Manage release base trend over UI

6. SOLUTIONS

8. • Continuous Integration
server that allows to run
multiple automation suites
at the same.
• Supports SVN, Git and
about any other SCM
tools you can think of via
plugins.

9. WHY JENKINS
 Web based
 No additional cost, we are already dependent on it
as TC Execution framework
 Well supported
 Highly customizable

10. JENKINS ON THE BASIS OF RELEASES DONE

14. COMMON JENKIN TERMS
 Project
 Builds
NEW TERMS INTRODUCED
 Benchmark Project
 Benchmark Builds

15. BENCHMARK PROJECT
 The project in Jenkins where we maintain the
automation results on the basis of releases done to
production

16. BENCHMARK BUILD
 Build which has best results
 The one whose final status we want to maintain in
the benchmark project

17. WHAT IS ALREADY AVAILABLE ?
 Show automation run trend
 Integration with SCM tool show code diffs

18. CHALLENGES
 Integration : Want every thing in the single screen
(single project in Jenkins)
 Benchmark Build : We don’t know priori that given
build is the best build to run
 Don’t want to waste resources & time to execute
automation run again once we have decided on the
build to deploy on production
 Intermittent failures : Want to maintain their status as
Passed for a given release.
 If we have taken the approach of copying results from
one project to other project (benchmark project)

19. OUR APPROACH
 Git Plugin for code diff
 Custom Java Script for TC diff
 Custom Java Script for override intermittent failure
status
 CopyArtifact plugin to migrate results
 Custom Groovy script for re-generating the Test NG
result trends

20. INTERMITTENT FAILURES

21. PARAMETERS TO COPY BENCHMARK BUILD

22. OVERRIDE INTERMITTENT FAILURE STATUS TO
PASS

23. FUTURE TASKS
 Show modified test-cases with the releases
 Show release trend graph in different color for
manually override test-cases
 Capturing results of multiple projects in same
dashboard

24. THANK YOU !!