Nathan Sharp of Siemens Energy recently spoke at the SAP Project Management in Atlanta and shared 7 important elements for the successful adoption of automated business process validation in their organization. Originally presented by Nathan Sharp of Siemens Energy at SAPinsider’s Project Management conference.

1. Case Study: How Siemens Energy
Saved Time and Money by
Managing and Automating Testing
for Its Recent Upgrade Project
Produced by Wellesley Information Services,
LLC, publisher of SAPinsider. © 2014 Wellesley
Information Services. All rights reserved.
Nathan Sharp
Siemens Energy

2. 1
Case Study Project
• Primary goal of the project
 Technical upgrade of SAP release 4.6C to ECC6.0
 Support for SAP and Oracle versions was ending
 Required for new functionality after the project
• Secondary goal of the project
 Implement and utilize automated testing software
 Aim was automation of 70% of manual business tests
 Upgrade was the pilot project

3. 2
Introduced a Governance Model
• Attended StarEast Conference (2012) and heard the strong
recommendation for a governance model
• We wanted to avoid a “Wild West” scenario in automation tool
• Governance model covers
 Granting access to the tool – but only after training
 Defining the development process
 Documenting and enforcing naming conventions
 Setting up and maintaining security in the tool

4. 3
Started Development of a “Product Bible”
• The Product Bible includes
 Roles and responsibilities
 New user information
 Folder structure
 Training material
 Issue tracking
 Test design
 Best practices

5. 4
Started Development of a “Product Roadmap”
• The Product Roadmap includes
 When we install and evaluate the next version
 Completion date for the Product Bible
 Which automation projects will be supported
 When additional modules will be acquired and/or implemented

6. 5
Appointed Automation Product Lead
• Responsible internally for the service
• Conduit for the vendor
• Organize training
• Prepare and manage budget
• Build and maintain the Product Roadmap

7. 6
Automated Testing Tool Training
• Diverse group of attendees – both business and technical
• 3 classes, 40+ attendees trained in total
• Each class 4.5 days in duration
• Used SurveyMonkey to get feedback from attendees

8. 7
Building an Example to Impress
• We chose a large end-to-end order-to-cash process
• Collaborative effort across multiple groups
• 30 reusable sub-processes and 391 steps
• No manual intervention
• Would take 2 or 3 days to accomplish manually
• Runs in 12 minutes
• Example has been used in various meetings and demos to help
understanding of test automation and to get buy-in

9. 8
Built a SharePoint System to Support Automation
4 SharePoint linked lists make up the
SharePoint testing database

10. 9
Documented Test Script Execution and Issues
SharePoint used to track testing progress
and allows for instant status reporting

11. 10
Progress at End of Round 1 Testing
• Manual or Automation tests
 Executed 96% of tests
 Passed 94% of tests
• However, less than 10% automation achieved

12. 11
Progress at End of Round 1 Testing (cont.)
• Some root causes for low amount of automation
 Core team was not trained early and first
 Not enough time between tool purchase and use
 The business group initially driving for the tool developed new
priorities
 The automation tool was mastered by only a few due to the
functionality restriction caused by both the automation tool
version and SAP version
 No powerful and interested sponsor

13. 12
Second Approach
• Round 2 of upgrade testing completed with 100% executed and
successful
• Three business analysts trained on process capture module (now
usable, as an SAP ECC 6.0 environment was available)
• Those business analysts used the period following completion of
upgrade testing and the end of project to work on further
automation

14. 13
Second Approach (cont.)
• Planned script automation fell below expectations
 Automation tasks more complicated and took longer than the
individuals themselves estimated
 Business analysts not fully dedicated to the task
 In one case, it was just not interesting work for the individual

15. 14
Third Approach
• Tests captured locally by business analysts (and later extended to
end users)
• Fully dedicated off-shore developer completed automation
• 69% of planned automation achieved with this approach

16. 15
Third Approach (cont.)
• Renewed management commitment due to the capture/off-shore
approach which works for us
• Developed a product roadmap – including new versions and
modules
• Automation extended to other projects
• “Lights-out” testing was developed. This is the scheduling of test
scripts to execute while unattended

17. 16
Lights-Out Test Example
• The script is a total of 2,347 steps and runs in approximately 16
minutes
• The script tests the following 7 closing programs:
 Reclass PC equity
 Reclass 482/483
 Z85/Z86 entries
 Income statement closeout
 Class 6 to 7 allocation
 Personnel cost reclass
 Calculate and post third-party inventories

18. 17
Lights-Out Test Example (cont.)
• Additionally, the script:
 Creates 4 unique multi-line journal entries in two different
company codes using both actual and statistical accounts (for
unique data creation)
 Changes status on WBS elements and internal orders
 Changes sets and custom CL+ tables
 Runs the profit center balancing program
 Automatically determines current date/period/fiscal year
 Validates the results of the 7 closing programs against
expected results from the JEs created

19. 18
Lights-Out Test Example (cont.)
• Benefits:
 This script detected an issue before the changes planned by a
new project were put live
 Prevented a major failure in production saving both IT and
business time

20. 19
Expectations from Automation
• Efficiency and labor savings
 Hundreds of manual testers freed up
• Shorter project timelines
• With 70% test automation, just one upgrade project on one of our
SAP boxes would eliminate testing costs of $270,000

21. 20
Expectations from Automation (cont.)
• Improved quality
 Fewer production defects
 Catch defects before they impact the business
• Supports our direction to move from Waterfall to Agile
development frameworks
 Testing is required in each development iteration and an
automation tool can allow this to happen more efficiently

22. 21
Internal Conference Held
• Two days of sessions related specifically to the automation tool
• Attended by key players from various Siemens companies
• Shared plans, issues, hints, and tips

23. 22
7 Key Points to Take Home
• Sponsorship is essential for ensuring success of any project
• Test automation is a project to itself – don’t try to combine with
another initiative
• Appointing a Product Lead helps ensure there is someone driving
progress
• Creating a Product Roadmap will give direction to an automation
initiative
• Avoid the “Wild West” in the automation tool with governance
• Ensure there is a mechanism for tracking manual and automatic
test development and execution
• A great way to generate excitement is to create an “Example to
Impress” automated test