Presentation give at UKOUG'08 on how the JDeveloper team test JDeveloper using Abbot and Selenium

2. <Insert Picture Here>
How the Oracle JDeveloper Team Test JDeveloper
Gerard Davison : Senior Principle Software Engineer
Geoff Waymark : Principal QA Engineer

3. Presentation Agenda
• Introduction <Insert Picture Here>
• Automation
• Tools
• Abbot
• Selenium
• Conclusion

4. <Insert Picture Here>
Introduction

5. Introduction
• Admission
– We are not testing saints
– We do not have 100% code coverage
– We probably never will
• But
– We have tried and succeeded in testing the ‘hard bits’
– We have learned some lessons
– Turns out a little can go a long way

6. Introduction
• JDeveloper
– We have everything from Swing to Excel
• Our focus is on the Java parts today
– We tend to be using technologies before you
• We create some of the technology
• This means we are ahead of test tools
– JDeveloper 8mloc in 35k source files over 650 modules
• 4 continents, at least 6 versions of English
– Two builds
• Sanity – verify no broken code
• Nightly - full release build ready for a customer

7. <Insert Picture Here>
Building
Sir Winston Churchill
“To build may have to be the slow and
laborious task of years. To destroy can
be the thoughtless act of a single day ”

8. Introduction -Terminology
• Unit testing
– The testing the Java Classes directly
• Integration testing
– Testing it all working together
• Automation
– Making it all work whilst you are in bed / lunch / pub
• Poking tooling
– Pretending to be you whilst you are in bed / lunch / pub

9. <Insert Picture Here>
Automation

10. Automation
What to automate?
The big question

11. What to automate?
• What is it for?
– Cost saving
• Humans - expensive, quite squishy – not entirely reliable
• Machines - comparatively cheap, don’t get bored
– Build verification
• Is today's build worse than yesterdays?
– Ready to use
• Most worn path works
• What would have to work to demo to your boss works
– Dull things
– Performance

12. What to automate?
• Where to begin?
– Fit to test / DFT
• Earlier rather than later
• Handover automation from Dev to QA
– Unit testing
– Smoke testing with poking tools
• When to begin?
– Unit testing, straight away, if not sooner
– Integration testing takes place later in the cycle
• Less code churn
• Fewer UI quakes

13. What to automate?
• What type of Automation?
– Unit testing – can only take you so far
• Verifies that what the developer thinks should happen
happens
– Integration testing – better, but you need more complicated
tools
• Verifies that what the rest of the tool thinks should happen
happens

14. How do we automate?
• Calling Java APIs
– Unit testing - in complicated situations it can be hard to set up
an environment.
– Often requires mock objects as things rarely work in isolation
• Using a Poking tool
– Control swing using a java.awt.Robot based tool
– Running a HTML based service from a web browser
– Likely native to the OS
– Better representation of what the user will actually see

15. What to automate?
• How much Automation?
– Build verification is limited by the time it takes between builds
– Use code coverage to meet a set value
• Lines in the sand are easy to measure
– Less value than you’d think early on.
– Less is more
• Don’t have more results than you can analyze
• Can you fix the tests In good time if they break

16. <Insert Picture Here>
Tools

17. Build tooling
• Build tools
– Needs to
• run automatically
– Generally as a result of SCM operation
• notify you when something goes wrong
– SMS alerts and the like*
• provide nice reporting, rss feeds and the like
• be beneficial for the developer
* Important for “Testing whilst at the pub”

18. Build tooling
• Build tools
– We have lots of custom code built up over years
– Get an answer as quickly as possible
• Sanity builds
• Run tests in parallel
Build 1S Build 1N Build 2S
SRG 1S SRG 1N SRG 2S
LRG 1N

19. Build tooling
• Cruise Control
– Ant wrapper
– Email notification
• Hudson
– Lots of web 2.x UI
– Using Webstart you can add machine to build environment
• Useful if you have many time zones
• cron or Windows scheduler
– Not really a build tool but can assist

20. <Insert Picture Here>
Consistency
A. Huxley
“Consistency is contrary to nature,
contrary to life. The only completely
consistent people are the dead..”

21. Automation Environment
• Work on a Dev machine the same as on the
Automated test machine
• Make sure your test environment runs on all platforms
– We have to hand test on a Mac still
• Just like Java - write once, debug everywhere
– Platform specifics like file separators, install locations
• Select which browsers you want to run in
– Firefox, IE, Chrome? What version?
• Size is important - what display size should you run
your tests in?
– Do you have a minimum display size for your app

22. Automation Environment
• Quick set ups
– Don’t want to test the setup 100 times
• We use zips to store project context
• Start tool and run inside rather than once for
each test
• Collect forensics
– Make a copy of the logs - ideally for each test
– If doing UI testing, take a picture when it fails
• Keep developers happy
– It you don’t make it easy tests won’t get run
– Assign someone smart to set this up.

23. Testing Tools Environment
• Integration testing gets asynchronous real quick,
– Harder than nice linear JUnit testing
– “Wait don’t delay”
– Make sure your testing tool understands this
otherwise find a new one

24. Testing Tools Environment

25. Other tools worth mentioning
• Free
– FEST
– JFCUnit
– HtmlUnit
– WebTest
• Expensive
– WinRunner
– SilkTest
– QTP

26. <Insert Picture Here>
Abbot

27. Abbot
• Automation of Swing using java.awt.Robot
– Also SWT version, not used it though
• Provides a Java API
• We use the ability to record and playback xml scripts
– Reduces the skill level required to create a test
• Is extensible so can handle custom components
• Gerard is a committer so we can fix and extended
where necessary

28. Costello
• Watches that you are doing
• Record actions steps
• Creates references to components
– Not brittle, will survive UI code changing
• UI a little bit special
• Does the job, can be a lot quicker than writing code
• You have to do some tidying up later though

29. Integration into JDeveloper
• Invoke tests / Costello inside of JDeveloper
– For faster turnaround
• Can invoke tests from command line
• Some small tweaks to make Costello work better
– Integration with our source control system
– Better default scripts

30. <Insert Picture Here>
Abbot Demo

31. Abbot tips
• Use the “Name” property to distinguish similar
components
• Wait don’t delay
– Ignore progress bars, test for something more reliable
• Run tests in “read-only” displays
– VNC useful for this on Unix
– Fast user switching for the same effect on Macs
– Windows users just have to go for a cup of tea
• Disable all popup notification
– Nothing more annoying than an IM ruining your test run
– Screensavers are great but won’t help the tests
• Don’t record mouse movements, wont run again

32. Abbot tips
• Call
– Invoke static Java methods
• Script
– Run BeanShell script

33. Abbot in JDeveloper facts and figures
• An example
– DB team (Our best team – test coverage speaking)
• Connections = 70% code coverage (block)
• Modeller = 68% code coverage (block)
• 86 Abbot tests
• 889 JUnit tests
• 40% coverage is JUnit on it’s own
• Abbot on its own would also be 40% showing the
intersection of the tests

34. <Insert Picture Here>
Selenium

35. Selenium
• Suite of tools
– Remote Control
• Integrates tests with your build
– IDE
• Exclusive to Firefox
– Grid
• Not used by Oracle
– We have an internal tool that predates Grid

36. Selenium Remote Control
• Comes in two parts
– Server
• Starts and stops the target browser
• Acts as an http proxy for requests
• Also includes the selenium core
– Client libraries
• Client libraries for your favourite language
• Flexible
– Integrates nicely with existing reports JUnit,etc…

37. Selenium IDE
• Creates tests
– Extension to Firefox
– Simple to use
• Record and playback
• Insertion for hand coding
• Looks like a nicer Costello
– Save tests in your preferred language
• HTML – {Default}
• Java – Good for us!
• C# ,Ruby ,PHP ,Python ,Perl – Good for everybody

38. <Insert Picture Here>
Selenium Demo

39. Selenium tips
• Maximize your browser
• Has to be verification for any action performed
– If you clicked a button, what did it do?
• Assertions should have meaningful messages
• Document the test
– Clearer for everybody
– Test spec
– JavaDoc for Java

40. Selenium tips
• Limited to the DOM tree
– Stylesheet or Style classes are not inherently testable
• Use clear ID’s
– ‘table1:0:openPopup’ is better than ‘table1:0:link1’
• Improve your test to get more from it
– Refactor IDE recorded tests
• Reliability
• Internationalization
• Data driven tests

41. <Insert Picture Here>
Conclusion

42. Conclusion
• Automated tests
– Have some – it helps
– Not too much though – it takes time
– Build and Test – have to be linked
– Development and QA in partnership

43. For More Information
• Abbot
– http://abbot.sourceforge.net/
• Selenium
– http://selenium.seleniumhq.org/
• Your speakers
– gerard.davison@oracle.com
• http://kingsfleet.blogspot.com/
– geoff.waymark@oracle.com