Improve test automation operation

自動化の結果データを自動化実行に使った話

Technology

Improve test automation operation
June 30th, 2020
Sadaaki, Emura
Leisure Product Dept.
Rakuten, Inc.

2
Who am I
Name : Sadaaki Emura (nickname M)
Join in Rakuten : 2007
Career : Embedded engineer (2000-2002)
Web engineer (2002-2015)
Product manager (2015-2016)
QA (2016~)
Role: TestAutomation Engineer Lead
Birthplace : Kanazawa-city
Hobby: jog , climbing , horse racing

4
Organization
Developer group QA group
Manual test team automation test team
@IllustAC
ココ
の人
:

5
Today story
• Test Automation operation is struggling
• Improve test automation operation

6
Today story
• TestAutomation operation is struggling
• Improve test automation operation

7
Test automation brings benefits in proportion to coverage!
• Reduce total cost of regression test
• Quick feedback
• Take work easy
© FREEIMAGES

8
Test automation brings struggle actually
• Many failure reports come
• Synchronize latest specification
• a few engineers
©photoAC

9
Classify many failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable

10
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Good activity !
• Happy to report engineer!
©いらすとや

11
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Script has bug
• Script covers old specification
• Modify script
©いらすとや

12
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Test data invalid
• Scenario dependency issue
• Arrange test scenario
©いらすとや

13
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Network disconnected
• 500 error (releasing operation?)
• 503 error (what’s happened?)
©いらすとや

14
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable

15
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
0.2 5.3
21.1
73.4
Fing bug Test automation script issue
Environment issue Temporary unstable

16
Operation for Temporary unstable
1. Investigate failure report
2. Run test automation script again
3. Check if result is success
Ex. OperationCost
1 operation = 2 mins
Daily operation = 100 * 2 = 200 mins
Monthly operation = 200 * 22 = 75 hours
Waste time!

17
Today story
• Test Automation operation is struggling
• Improve test automation operation

18
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
0.2 5.3
21.1
73.4
Fing bug Test automation script issue
Environment issue Temporary unstable
• This failure account for ¾
• Recover operation is “Re-Run”

19
Auto recover system
Test Result is Failure
Predict reason
Temporary
unstable
Re run
Yes

20
Auto recover system
Test Result is Failure
Predict reason
Temporary
unstable
Re run
Yes
Test result output
• Console error message by Ranorex
• Error page screenshot

21
Auto recover system
Test Result is Failure
Predict reason
Temporary
unstable
Re run
Yes
Use data
• Console error message
• Message in screenshot
• Screenshot
• Previous prediction data
UseTechnology (Python)
• Tesseract-OCR
• Deep Neural Network with Keras

22
Auto recover system
Test Result is Failure
Predict reason
Temporary
unstable
Re run
Yes
Input test report
Get message from
screenshot
Text base
prediction
Image base
prediction
No result
Output prediction
In progress
Feedback

23
Auto recover system
Test Result is Failure
Predict reason
Temporary
unstable
Re run
Yes
• Reduce waste time operation
• Not waste test resource and time

24
Auto recover system benefit
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Oct-19 Nov-19 Dec-19 Jan-20 Feb-20 Mar-20
auto recovery manual recovery
Ex.
Date : March-20
Temporary unstable Ratio : 75%
Operation time without system: 100 hours
100 * 0.65 = 65 hours saved
65%

25
Summary
• Test automation operation takes time and includes boring task
• Let’s automate some of operation
• Auto recover system is not perfect yet, keep improvement

Similar to Improve test automation operation

20200630 Rakuten QA meetup #2 "Improve test automation operation"

We have challenging issue in test automation operation. Test automation fail by bug and not bug reason. I think one of the main reason is temporary accident. When test automation fail by this temporary accident, test will be success by running test again usually. This re-run operation is boaring task and big task for test automation operation To eliminate this kind issue and improve operation, we built system categorize issue by machine learning, re-run test only when fail reason is temporary accident. In this session , I show you these below - What's test automation issue in daily operation - How to resolve this issue. store big data, learning, system architecture etc. - Actual result for improvement

Test Automation Improvement by Machine Learning Jasst'21 Tokyo

Software Test Automation

Yash Patel

Test automation engineer

Bugday bkk-2014 nitisak-auto_perf

Nitisak Mooltreesri

20180828 QA meetup

Believe it or not - 85% of mobile apps are removed after first usage! In this presentation - given at the APM Meetup in Singapore in April 2015 - I talked about the challenges, best practices and especially metrics to avoid this situation. Key Points of the Presentation The two key trends "Internet of Things" and "DevOps" play a big role in our life when we talk about User Experience and especially mobile user experience. In this presentation I tell you what metrics to use to make sure you deliver your ideas faster to your mobile end users but also ensuring the right quality and user experience so that your users stay loyal and dont delete the mobile app after first usage.

Mobile User Experience:Auto Drive through Performance Metrics

Andreas Grabner

John Fodeh - Adventures in Test Automation-Breaking the Boundaries of Regress...

TEST Huddle

John Fodeh Adventures in Test Automation - EuroSTAR 2013

TEST Huddle

Wso2con test-automation

krishantha_samaraweera

Cloudstack Continuous Delivery

buildacloud

WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment

WSO2

** Full webinar recording: ** It is no secret that many teams struggle with automated functional UI testing - some to the point where it is completely abandoned - even though the UI is the most significant interface of the system. In this session, Adam Carmi -- Applitools CTO and Co-founder -- reviewed the main weaknesses of traditional approaches to UI testing, and how they negatively affect test stability, maintainability, coverage, execution speed, and overall ROI. He also discussed how these weaknesses become even more severe when running tests across multiple devices and browsers. Adam demonstrated how Visual AI -- the innovative technology powering Applitools' testing engine -- can be applied on your existing pipeline to efficiently implement functional UI tests with a fraction of the effort while drastically increasing test coverage and reducing test execution time. Adam also showed a live coding session, where he converted a traditional UI test into a Visual AI-based test in minutes, and executed it across dozens of devices and browsers in seconds using the Applitools Ultrafast Grid.

[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi

Applitools

JaSST'18 Hokkaido Improve Automation Testing

Presented at STPCon 2016. With the extensive amount of testing performed nightly on large software projects, test and verification teams often experience lengthy wait times for the availability of test results of the latest build. As we strive to identify and resolve issues as fast as possible, alternative methods of test execution have to be found. Learn how to use Jenkins to launch tests in parallel across a number of Virtual Machines, monitor execution health, and process results. Learn about various Jenkins plugins and how they contributed to the solution. Learn how to trigger downstream jobs, even if they are on separate Jenkins instances.

Agile performance testing

Andriy Melnyk

Selenium Success

Surinder Kaur

Agile testing - Principles and best practices

Dr Ganesh Iyer

Test parallelization using Jenkins

Rogue Wave Software

Full webinar recording -- In an era when testing is no longer limited to end-of-line waterfall-type process -- but is an integral and substantial part of the end-to-end development process -- it is ever more pressing to look under the hood of the tools, technologies, processes, and best practices utilized by top-notch testing teams that release world-class software. In this webinar, we took an in-depth look at the technical aspects required to successfully scale continuous testing -- and discussed the tools and technologies used by leading engineering teams, including their approach to managing functional, performance and visual testing at scale. We also took a hands-on approach: via a live demo, we showed how all of these come together to create a complete end-to-end quality pipeline that supports continuous testing at scale.

Building a Complete Pipeline: The Essential Components of Continuous Testing ...

Applitools

Android CD

Eugen Martynov

Similar to Improve test automation operation (20)

20200630 Rakuten QA meetup #2 "Improve test automation operation"

Test Automation Improvement by Machine Learning Jasst'21 Tokyo

Software Test Automation

Test automation engineer

Bugday bkk-2014 nitisak-auto_perf

20180828 QA meetup

Mobile User Experience:Auto Drive through Performance Metrics

John Fodeh - Adventures in Test Automation-Breaking the Boundaries of Regress...

John Fodeh Adventures in Test Automation - EuroSTAR 2013

Wso2con test-automation

Cloudstack Continuous Delivery

WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment

[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi

JaSST'18 Hokkaido Improve Automation Testing

Agile performance testing

Selenium Success

Agile testing - Principles and best practices

Test parallelization using Jenkins

Building a Complete Pipeline: The Essential Components of Continuous Testing ...

Android CD

More from Rakuten Group, Inc.

CI/CDツールであるJenkinsとIDEであるIntelliJ IDEA。両者の共通点は、Javaで書かれている点と、プラグインによって機能を拡張できる点です。つまり、Javaを用いてプラグインを実装することで、自分が欲しい機能を追加することができます。本セッションでは、コードレビュープロセスを改善するためにJenkinsとIntelliJ IDEAのプラグインを自作した経験を踏まえて、実装方法に触れつつ、得られた結果や苦労した点について話します。

コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話

楽天における安全な秘匿情報管理への道のり

What Makes Software Green?

Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...

DataSkillCultureを浸透させる楽天の取り組み

大規模なリアルタイム監視の導入と展開

楽天における大規模データベースの運用

楽天サービスを支えるネットワークインフラストラクチャー

楽天の規模とクラウドプラットフォーム統括部の役割

Rakuten Services and Infrastructure Team.pdf

The Data Platform Administration Handling the 100 PB.pdf

Supporting Internal Customers as Technical Account Managers.pdf

Making Cloud Native CI_CD Services.pdf

How We Defined Our Own Cloud.pdf

Travel & Leisure Platform Department's tech info

Travel & Leisure Platform Department's tech info

OWASPTop10_Introduction

Introduction of GORA API Group technology

100PBを越えるデータプラットフォームの実情

社内エンジニアを支えるテクニカルアカウントマネージャー

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

More from Rakuten Group, Inc. (20)

コードレビュー改善のためにJenkinsとIntelliJ IDEAのプラグインを自作してみた話

楽天における安全な秘匿情報管理への道のり

What Makes Software Green?

Simple and Effective Knowledge-Driven Query Expansion for QA-Based Product At...

DataSkillCultureを浸透させる楽天の取り組み

大規模なリアルタイム監視の導入と展開

楽天における大規模データベースの運用

楽天サービスを支えるネットワークインフラストラクチャー

楽天の規模とクラウドプラットフォーム統括部の役割

Rakuten Services and Infrastructure Team.pdf

The Data Platform Administration Handling the 100 PB.pdf

Supporting Internal Customers as Technical Account Managers.pdf

Making Cloud Native CI_CD Services.pdf

How We Defined Our Own Cloud.pdf

Travel & Leisure Platform Department's tech info

OWASPTop10_Introduction

Introduction of GORA API Group technology

100PBを越えるデータプラットフォームの実情

社内エンジニアを支えるテクニカルアカウントマネージャー

Recently uploaded

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Why Teams call analytics are critical to your entire business

panagenda

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

FWD Group - Insurer Innovation Award 2024

AXA XL - Insurer Innovation Award Americas 2024

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Axa Assurance Maroc - Insurer Innovation Award 2024

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Manulife - Insurer Transformation Award 2024