Решение для управления тестированием на платформе SBM
•Als PPTX, PDF herunterladen•
0 gefällt mir•542 views
Пример реализации решения на платформе SBM для управления процессами тестирования. Как и любое решение, выполненное на платформе SBM, процессы можно легко адаптировать под свою специфику, все информационные связи между кейсами, результатами, версией сборки, дефектами, запросами на разработку и т.д. наглядно представлены на формах и интерактивных отчетах. Решение состоит из 4 составляющих: 1) Библиотека сценариев тестирования (test cases) 2) Репозиторий результатов исполнения тестов 3) Модуль планирования функционала с возможностью комбинирования состава пакета сценариев 4) Модуль управления доступностью тестовых серверов
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Решение для управления тестированием на платформе SBM
Ähnlich wie Решение для управления тестированием на платформе SBM (20)
Mehr von Softmart
Mehr von Softmart (18)
Решение для управления тестированием на платформе SBM
- 1. Решение для управления процессами тестирования Ирина Луцюк, Serena Software
- 3. Тест кейсы 3
- 4. Где хранились • В системе контроля версий (PVCS VM) как файлы 4
- 5. Где хранились • В сети на сервере как офисные документы 5
- 6. Где хранились • Страницы Wiki 6
- 7. В каких форматах • Офисные документы в форматах MS Word и Excel 7
- 8. В каких форматах • На листочках бумаги 8
- 9. Статус-кво по тест-кейсам • Не было единого хранилища • Совершенно непонятна степень покрытия функционала приложений сценариями тестирования • Не было единого формата и требований к содержанию • Оценка времени, требуемого для прохождения сценария, не проводилась • Сложилась негибкие, неустойчивые, но как-то применяемые практики • Невозможно проследить зависимости • История изменений не велась и не хранилась 9
- 11. Статус-кво по результатам тестирования • Отчеты по факту прохождения каждого тест-кейса заполнялись не всегда • Прослеживание связей между функциональностью/функциями/требованиями не происходило (не было такой возможности) • Не фиксировались связи с запросами на изменения • Не отслеживались конфигурации, на которых проводилось тестирование • Невозможно проиллюстрировать достигнутый уровень качества 11
- 13. Статус-кво по планированию тестирования • Очень неформальное • Основной принцип – необходимо «все проверить» к такому-то сроку безотносительно масштабов изменений • Посмотреть прогресс или статус было невозможно • Совершенно бесполезно для оценки готовности релиза к финальному выпуску 13
- 15. Библиотека тест-кейсов (TMS Test Library) • Жизненный цикл сценария тестирования In Work In Review Approved Out of Date 15
- 16. Библиотека тест-кейсов (TMS Test Library) • Жизненный цикл сценария тестирования In Work In Review Approved Out of Date 16
- 17. Библиотека тест-кейсов (TMS Test Library) • Жизненный цикл сценария тестирования In Work In Review Approved Out of Date 17
- 18. Библиотека тест-кейсов (TMS Test Library) • Жизненный цикл сценария тестирования In Work In Review Approved Out of Date 18
- 20. Кратко по библиотеке тест-кейсов • Простой процесс • Единый репозиторий • Интуитивное использование • Унифицированная форма тест-кейса • Покрытие тестами относительно функциональной декомпозиции • Связи с запросами на изменение • Хорошая оперативная и аналитическая отчетность • Уведомления пользователям • Полный контроль над решением (можем сами менять ЛЮБЫЕ элементы реализации) 20
- 21. Результаты тестирования • Жизненный цикл тестирования To Be Executed Passed Failed Blocked 21
- 22. Результаты тестирования • Жизненный цикл тестирования To Be Executed Passed Failed Blocked 22
- 23. Результаты тестирования • Жизненный цикл тестирования To Be Executed Passed Failed Blocked 23
- 24. Результаты тестирования • Жизненный цикл тестирования To Be Executed Passed Failed Blocked 24
- 26. Результаты тестирования • Каждый запуск тест-кейса фиксируется • Результат ассоциируется с конкретным сценарием тестирования • Есть прямые связи с запросами на изменения, которые тестируются • Есть возможность проследить информацию по сборкам и релизам • Фиксируется информация по статусам тестирования, времени тестирования и т.д. • Расширенная отчетность 26
- 27. Планирование тестирования Пакеты тест-кейсов • Test Requirement Lifecycle New Assigned For Testing Planning In Progress In Review Ready CCRB Completed 27
- 29. Планирование тестирования Пакеты тест-кейсов • Простой процесс планирования • Гибкость в определении покрытия тест-кейсами • Наглядное представление покрытия функционала кейсами по • Функциональной декомпозиции • Конфигурации и Среды тестирования • Проектных характеристик • Понятные оценки продолжительности тестирования • Наглядное представление объема работы • Простое отслеживание статуса 29
- 30. QA Отчет по прогрессу тестирования 30
- 31. Управления средами тестирования New In Pool Configuring Available In Use 31
- 33. Управления средами тестирования • Стандартизация описания сред тестирования • Наглядное представление доступности сред • Взаимосвязь между пакетом тестирования и средой исполнения тестовых сценариев 33
- 34. Статистика использования Manual Test Cases Test Results 34
- 36. Планы по развитию • Интеграция с платформами автоматизированного тестирования • Возможность запускать автоматические сценарии из Serena • Единый репозитарий результатов исполнения тестов • Улучшение пользовательского интерфейса в части наполнения пакета тестирования тест-кейсами • Возможность представления Шагов тест-кейса и Результатов в табличной форме. • Возможность наглядной привязки ошибки к конкретному шагу тест-кейса 36
- 37. Спасибо 37
Hinweis der Redaktion
- When I joined QA team in Serena I found that the Test Management and namely test cases, test results management, test planning were very very imperfect. It all worked – there were the test cases, sometimes test results and some planning. But if we compare 5 years ago test management with the train than it was a steam engine train. It works but it’s slow, very massive, uncomfortable, non-elegant and in agile environment simply useless.Time: 1 min
- So, the test cases. As I said the test cases existed when I came. But the main problem was that there was no a consistency.
- First of all a location of test cases. A large number of them were stored in a source control system.
- But at the same time there was a large number of them on people’s computers. There were lots of shared folders in the intranet with very similar names (like on the screenshot ‘docs in the one place’)
- If you were looking for a test case, test script or some test prerequisite instructions and you didn’t see it in one of the previous locations you had to search on the wiki.
- There was no unified format for test cases. Some of them were created as tables in Word documents, some in excel spreadsheets. You could also see plain text files, bash scripts and many different types.
- And of course the most popular was a handwritten text. Test cases on the same paper sheet with a telephone of a dentist and a reminder about a party on the next week.
- Let’s summarize what we just learnedNo single source. Hard to find a needed information. Hard to find where to create a new test case. Lots of duplicating stuffAnd as result – no idea about the coverage because we could not put all the test cases together.There was no unified format. This one doesn’t sound as a serious problem but if you have a large team of very creative and imaginative people believe me it is a problem :)We could not obtain estimates for a whole suite of test cases or for a portion of them when we needed to test some particular feature. We simply could not say how much time a testing will take.No flexibility. No flexibility in grouping test cases, splitting and merging them. Really it all was impossible.No traceability of product and project changes in test documentation.No history of changes. A detailed history of changes, not just - a someone has updated the document as source control says.
- Situation with the test results was even worse.
- Test results were created occasionally because people didn’t know in which format they should be created. Because similar to the test cases there was no any format for test results. The test results were not traceable against features or functions, or product requirements. We never could say how well a feature was tested and what’s the quality of it.We didn’t have a proper relationship of a test casewith a change request. Again, to see a test report for some change.You know that all products of serena have very wide lists of supported platforms and configurations. So, with our old approach we could never say what was tested where. What was test coverage for Win2000 or Solaris9.The main goal of testing is to provide a quality report.And with our old approach we were unable to. We could not say what was a quality level, we just submitted bugs and they didn’t actually demonstrate an overall quality.
- Situation with test planning was even worse than with the test results.
- Test plans were very informal. They didn’t tell us what we had to test, how deep this testing should be, what test configuration should be used for it.Test plans were time driven and they weren’t focused on changes. A task could sound like this – spend 3 days of testing for Web Client. Which such tasks you can never track a progress – how much was done, what else needs to be done and what’s the status.Such test plans could never be used in the release readiness decision
- The times of steam engine have gone. I’ll tell you about TMS, it is a Test Management System created with the help of SBM product. So, when I found that a test management required a change I evaluated different approaches. I’ll tell about some. First was TestLink open source product. As the most of open source systems it seemed to be what we needed but not exactly. It had ugly interface and was very inflexible. Another variant was Quality Center. As the most of proprietary products it did exactly what we needed and it was hugely expensive (and it still is).Eventually a solution was exactly in our hands – we could build our own system in SBM. And we did. If you remember I said that my first team and product were Dimensions CM, I didn’t have a good experience of using SBM but it didn’t stop me. With the intuitive interface of SBM composer and clear understanding of what I want to do we did very well.
- I started with the very basic element of test management – a test case. TMS Test Library is a repository of manual test cases. A lifecycle of a test case is very straightforward.‘In Work’ is an initial state where the test case appears when it’s created. This not an active state, it means that the work on the test case hasn’t been finished yet. A submitter can come back to it and provide additional information later.
- When submitter feels confident about a contents a test case should be sent to someone for review. Usually it’s a QA guy who’s responsible for the test coverage in the area which was used in the test case.Test case can of course go back to submitter with the comments. But if there’re no comments a reviewer moves a test case to the Approved state.
- ‘Approved’ is a final and active state for the test case.It’s not recommended to use a test case while it’s not Approved by a reviewer. When the test case is in the Approved state Run button becomes available. Clicking this button creates a test result item automatically. So, if you want to execute some test and log the result you should click Run. A little bit later I’ll tell more about test results.
- There’s also ‘out of date’ state which can be used for test case which is not applicable in none of the currently supported releases. For example there was a feature which was later removed of changed significantly. In this case you don’t delete a test case but action it to the ‘our of date’ state. Of course you can activate it in the future.
- As many of you know SBM provides you a flexibility in the way how you build interface, web forms. Current format is compatible with the format of change request forms which makes a usage of both systems even easier. On the top you can see a state change history.On the left hand side under the state history there’s a key information of a test case - a title, test prerequisites, test steps and expected results.On the right hand side you can see several tabs where we show supplemental information for the test case.On the main tab you can see what’s the current project, who are submitter and reviewer of this test case.Then it is a list of releases where this test case is applicable. Under releases you can see information about product and functional area to which a test case is related. Component, functional area and sub-functional area are elements of product feature breakdown structure or functional decomposition. These are the mandatory fields because linking test cases to feature breakdown enables us in seeing a test coverage for product functions and components.Then there’s a test category (smoke, regression, full and others), test case estimated run time – a very important attribute which now makes it possible to get estimates for a collection of test cases.Then it’s automated field where currently we only note if this test case is automated or not, or needs to be automated. This attribute will be later a part of a new TMS solution aimed to integrate a management of automated tests into TMS.On the Scrum tab you can see information related to the Scrum team working with this test case and its activities. As you can see you can relate test case to a user story, put a story link and specify a themeOn the ‘test results’ tab you can see a whole history of executions of the current test case. It is a report showing you a project where it was executed, a date, a name of person who executed, a result of this execution and defects related to it.On the Related Defects tab you can see a list of defects which were ever reported against this test case. Or vice versa – a defect which was reported by customer, fixed and tested.On this report you can see a current state of a defect, its severity, business priority, list of requestors if there are any.An integration between test management and change request management tools is very significant as it provides you an up to date information about the quality.Targets tab is used to show configuration related information – any specific requirements for this test case. In this example it’s shown that the integrated product is VS 2010. It means that this test case is only applicable to VS 10.You can also see here other components of configuration – server, client, db, web browser, etc.There’s a history of change on the last History tab. Here you can see who changed what exactly and when. Thereby no change will be lost
- Let’s see a summary of TMS Test Library featuresWe’ve got the process in place and it is really simple. Test case’s lifecycle contains just 4 states – submit, review, approve and use!We finally have a single source for all test cases, test scenarios, test instructions. No more files in the network and source control. Using of a system is really simple. First of all it is a very well known SBM, additionally all the forms have a layout very similar to forms in our Change Request system also powered by SBMWe finally have a unified format for our test cases. We made sure that all important information is provided by making those attributes mandatory. Nobody asks how I should write a test case.A benefit we’ve got from using TMS is in the possibility of estimating a coverage with the test cases of product features and functional areas.A very useful feature is a relationship between the two very important artifacts of testing – test cases and change requests (defects and enhancements). You can always see all know issues raised against the test scenario, their current states, projects where they seat, their severity, their requestors, etc. And you can see which test case was used for defect validation or you can see which one can be used for validation.-SBM provides you with the advanced reporting engine. You are free to build any kind of report to see your progress and obtain testing metrics. You’re not limited with the predefined reports which do not always satisfy all your needs.You can set up and use email notifications to make a process faster. So whenever a test case is submitted you get a notification, or you receive it when you become an owner of an item with review request. A key benefit which we have is a full control over a system. We made lots of changes since a first test case was submitted in October 2009. You see how it goes, you hear what people say and you have a possibility to adapt your tool.
- Let’s have a look at test results. There’re the two ways who you can create a test result. First is using a Run transition (or in user words – clicking a Run button) for manual test case. This will create a test result item which is actually a copy of a manual test with additional information which you provide on the Run transition – it is information about test configuration which you are using for test execution – (server and client platform, data base, browser, etc), it is release name and build number. There’s a possibility to create multiple test result items and this will be covered later.Once test result is created it appears in the ‘to be executed state’. It’s an initial state demonstrating that the test is going to be executed physically and test result logged.So, you executed a test in an application and you log a result. There’re three possible results which you can use for you execution
- A first is ‘passed’ which means that what is described in a test case is absolutely what you see in a product. You need to action your test result to the ‘passed’ state.
- A ‘failed’ state which neither developers nor managers like is used to report a failed test. When you action your test to the ‘failed’ you are requested to provide failure related information – an actual result, what was different from an expected result described in a test case. And another mandatory attribute is a defect number. As we all know every failure should have a corresponding item in the bug tracking system. You can either pick up and existing defect item (in the case this test case is failing by already known reason) or go and create a new defect and related it to you test result.
- A ‘blocked’ state is used to show that there’s another problem not related to the current test cases but it’s preventing us from executing it.
- Let’s see the form for test result