Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Cross mobile testautomation mit Xamarin & SpecFlow

3.234 Aufrufe

Veröffentlicht am

Test automation can be implemented most efficiently as a by-product of Specification-By-Example (SbE). It combines acceptance criteria specification and acceptance test driven development (ATDD, BDD) to build automatically validated specifications of the system. The practice is well established in many teams for “traditional” enterprise application development (web clients, rich clients, services), and supported with a broad range of tools.
In mobile development, however, we seem to start over again with bare-bones test automation tool support that provokes post implementation test automation, which is costly and hard to maintain. Teams that had already successfully applied ATDD/BDD fall back into old habits when moving to mobile development. This is due to the lack of tool support and a lack of confidence that the principles that worked before can also be applied in mobile development.
Join Gaspar, Christian and Andreas for a brief introduction to BDD and Specification-By-Example. They’ll then show how it can be put into practice with SpecFlow and Calabash for a mobile app that is developed using Xamarin.

Veröffentlicht in: Technologie

Cross mobile testautomation mit Xamarin & SpecFlow

  1. 1. COPYRIGHT, TECHTALK - WWW.TECHTALK.AT ANDREAS WILLICH (@SABOTAGEANDI) CHRISTIAN HASSA (@CHRISHASSA) Mobile Quality Crew Meetup, Vienna, April 23rd 2015 Cross Mobile Testautomation with Xamarin & SpecFlow
  2. 2. 2 Topics • Xamarin • Test automation in agile projects • Specification-By-Example • SpecFlow • UI automation on mobile devices • Mobile app automation with SpecFlow
  3. 3. 3 Xamarin
  4. 4. 4 Native development for each platform
  5. 5. 5 Abstracting the platform
  6. 6. 6 Xamarin approach
  7. 7. 7 Compilation and run time Android APK Android/Linux kernel IL code for platform specific profile VS solution with projects iOS APP Android APK Shared Code (C#,VB.NET) Mono CLR for iOS (Monotouch) Mono CLR for Android (Monodroid) Multi-Platform profile assembly Dalvik Android.* Java.* IL+JIT AOT ARM Binary iOS App (C#) Android App (C#) Xamarin.iOS profile assembly Xamarin.Android profile assembly iOS/Cocoa Touch iOS APP ARM binary
  8. 8. 8 Native mobile apps with C#/Xamarin 1. Reuse existing .NET/C# Know-How 2. Powerful platform and tools (Linq, Xml, Events/Delegates, Parallel Programming, Visual Studio/TFS, …) 3. Shared code for multi-platform apps
  9. 9. 9 Required Know-How C#
  10. 10. 10 Xamarin.iOS C#
  11. 11. 11 Xamarin.Android twitterListView.setOnItemClickListener (new OnItemClickListener () { public void onItemClick (AdapterView<?> parent, View view, int position, long id) { Intent tweetDetails = new Intent (TwitterScreen.this, TweetDetailsScreen.class); tweetDetails.putExtra ("TweetID", TwitterFeed[position].ID); startActivity (tweetDetails); } }; twitterListView.ItemClick += (s, e) => { var tweetDetails = new Intent (this, typeof (TweetDetailsScreen)); tweetDetails.PutExtra ("TweetID", TwitterFeed[e.Position].ID); StartActivity (tweetDetails); }; C#
  12. 12. 12 Development environment OS Mac OSX, Windows Windows UI Designer Android, iOS Build Android, iOS (via Xamarin Build Host on Mac OSX) Debug Android (Device and ARM/x86 emulator) iOS (Device and emulator on Mac)
  13. 13. 13 .NET libaries availabile on Xamarin Assembly feature set mscorlib.dll SL OpenTK.dll OpenGL/OpenAL System.dll SL+ System.Core.dll SL System.Data.dll NET35- System.Data.Service.Client.dll vollständiger OData Client System.Json.dll SL System.Runtime.Serialization.dll SL System.ServiceModel.dll SL (WCF) System.ServiceModel.Web.dll SL+ System.Transactions.dll NET35 System.Web.Services NET35- System.Xml.dll NET35 System.Xml.Linq.dll NET35 Mono.CompilerService s.SymbolWriter.dll Für Compiler Mono.Data.Sqlite.dll ADO.NET für SQLite Mono.Data.Tds.dll TDS Protokoll Support Mono.Security.dll Cryptographic API monotouch.dll iOS Bindings Mono.Android.dll Android Bindings
  14. 14. 14 Xamarin.Forms One UI definition for 3 platforms (iOS, Android, WP) in XAML (data binding) translates to native UI controls on each platform can be mixed with custom UI
  15. 15. 15 Test Automation in agile projects
  16. 16. 16 Test automation becomes expensive when … • automating manual tests • automation makes tests unreadable • automating only after implementation structure readability right moment
  17. 17. 17 Structure Manual tests Asserts Multiple combined features Structure ACT-ASSERT- ACT-ASSERT- ACT-ASSERT- … Dependent features Long test path with high chance to break Cause and impact of error hard to trace Automated Check Single aspect of a single feature ARRANGE – ACT – ASSERT Independent features Short test path with lower chance to break Cause and impact of error easy to relate
  18. 18. 18 What to automate User journeys Acceptance- criteria Units exploratory testing Test automation pyramid Source: Mike Cohn many few hard easy Automatability
  19. 19. 19 // Go to web page 'http://localhost:40001/' using new browser instance BrowserWindow localhostBrowser = BrowserWindow.Launch( new System.Uri(this.RecordedMethod1Params.Url)); // Click 'Register found item' link Mouse.Click(uIFundstückerfassenHyperlink, new Point(56, 9)); // Click 'Save' button Mouse.Click(uISpeichernButton, new Point(44, 14)); int fundNr1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9)); // Click 'Register found item' link Mouse.Click(uIFundstückerfassenHyperlink, new Point(63, 7)); // Click 'Save' button Mouse.Click(uISpeichernButton, new Point(34, 11)); int fundNr2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9)); Assert.IsTrue(fundNr1 + 1 == fundNr2); // Click 'Close' button Mouse.Click(uICloseButton, new Point(26, 11)); Readability
  20. 20. 20 A readable test case Scenario: New found items should receive a consecutive number for the current year Given the previous found item of the current year had the number 145 When I register a new found item Then the last found item of the current year should have the number 146
  21. 21. 21 When to test (right moment) Acceptance criteria checks (ATDD, BDD) Unit Tests (TDD) business view technical view Exploratory tests User Journey tests Performance, Scalability, Usability,Security, … productdefinition productcritique New dimension: defining the product Synergy: Specification of requirements and tests Agile Testing Quadrants: Brian Marick
  22. 22. 22 Automated checks don’t replace manual testing User journeys Acceptance- criteria Units exploratory testing manual check in definition of done for stories Main success paths Undiscovered acceptance criteria No/(fewer) manual regression checks Fewer paths are enough More time for exploration
  23. 23. 23 Specification-By-Example
  24. 24. 24 Agile “requirements” Specification- By-Example Image from: 50 Quick ideas to improve your user stories: https://leanpub.com/50quickideas Why? Outcome How? Specification
  25. 25. 25 Agile specifications define upfront define just-in-time Code Reminder for a conversation Goals Impacts Epics User Stories Acceptance criteria Why? Outcome How? Specification
  26. 26. 26 public void CalculateDiscount(Order order) { if (order.Customer.IsNew) order.FinalAmount = Math.Round(order.Total * 9/10); } Register as “bart_bookworm” Go to “/catalog/search” Enter “ISBN-0955683610” Click “Search” Click “Add to Cart” Click “View Cart” Verify “Subtotal” is “$33.75” We’d like to encourage new customers to buy in our shop. Therefore we want offer 10% discount for their first order. Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
  27. 27. 27 Acceptance criteria Agile specifications define upfront define just-in-time Code Goals Impacts Epics User Stories Bugs Why? Outcome How? Specification
  28. 28. 28 Specification with examples Examples … make abstract descriptions better understandable are usually not formally documented or exchanged Examples Tests Requirements describe validate fulfillment of consist of
  29. 29. 29 Register as “bart_bookworm” Go to “/catalog/search” Enter “ISBN-0955683610” Click “Search” Click “Add to Cart” Click “View Cart” Verify “Subtotal” is “$33.75” public void CalculateDiscount(Order order) { if (order.Customer.IsNew) order.FinalAmount = Math.Round(order.Total * 9/10); } We’d like to encourage new customers to buy in our shop. Therefore we want offer 10% discount for their first order. Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
  30. 30. 30 Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com New customers should receive 10% discount: Given the customer hasn‘t ordered yet When the customer adds a book for EUR 37.50 to the shopping cart Then the shopping cart shows a total price of EUR 33.75.
  31. 31. 31 Actually, this is not quite right: Books on sale should be excluded! Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
  32. 32. 32 “Happy Path” Technical feasibility Exceptions, border cases 3 Amigos Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
  33. 33. 33 Collecting Acceptance Criteria “I would try to put a book into the shopping cart …” “I would try to remove a book from the shopping cart…” “I’d check whether the shopping cart is empty, when I enter the shop …” Books can be added to shopping cart. Books can be removed from shopping cart. Shopping cart should be empty when entering the shop. ... ? … As a potential customer I want to collect books in a shopping cart So that I can order several books at once. “Imagine this story is already implemented: How would you verify it?” “I would try to add 1000 books to the shopping cart …”
  34. 34. 34 Exploration through examples Books in catalogue: Title Author Specification-By-Example Gojko Adzic Impact Mapping Gojko Adzic Explore It! Elisabeth Hendrickson Competitive Engineering Tom Gilb … I want to find books in the catalogue by author and title Search for … Books found … Spec Specification-By-Example Hend Explore It! et Explore It!, Competitive Engineering Context Action Assertion
  35. 35. 35 Key examples: Breaking the model Books in catalogue: Title Author Specification-By-Example Gojko Adzic Impact Mapping Gojko Adzic Explore It! Elisabeth Hendrickson Competitive Engineering Tom Gilb … I want to find books in the catalogue by author and title Search for … Books found … Spec Specification-By-Example Hend Explore It! et Explore It!, Competitive Engineering What happens, if I search for “Explore Specification”? Can I search for single characters, e.g. “e”? Is the number of search results limited, or paged? Is the search also performed in the sub-title of a book?
  36. 36. 36 Purpose of the examples • Shared understanding: acceptance criteria • Documentation: specification details • Regression-tests: violated specifications
  37. 37. 37 SpecFlow
  38. 38. 38 SpecFlow – BDD for .NET #62 in Visual Studio Gallery based on popularity (Apr 23 2015) #303 most downloaded package on NuGet in the last 6 weeks (Apr 23 2015) ~1‘400 unique daily visitors on specflow.org > 30’000 active users > 40 contributors
  39. 39. 39 Feature: Description of feature or user story Szenariogrundriss: Beschreibung des Akzeptanzkriteriums Szenario: Beschreibung des Akzeptanzkriteriums Gherkin Feature Files Background: context for all scenarios in the feature file Scenario: Description of acceptance criterion Angenommen/Wenn/Dann: Automatisierte Szenario Schritte Given/When/Then: automated scenario steps Scenario Outline: Description of acceptance criterion Angenommen/Wenn/Dann: Automatisierte Szenario Schritte Given/When/Then: automated scenario steps with <place holder> Examples: table with examples for <place holders>Examples: table with examples for <place holders> Given: automated scenario steps @tagname @tagname @tagname @tagname
  40. 40. 40 Abstract acceptance criteria As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once. Books can be added to the shopping basket Books can be removed from the shopping basket Shopping basket is initially empty The same book can be added multiple times to the shopping basket
  41. 41. 41 Scenarios: Examples in Gherkin As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once. Books can be added to the shopping basket Given my shopping basket is empty When I add the book “Harry Potter” to my shopping basket Then my shopping basket should contain 1 copy of “Harry Potter”
  42. 42. 42 As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once. Books can be added to the shopping basket Scenarios: Examples in Gherkin Given my shopping basket contains 1 copy of “Harry Potter” When I add the book “Harry Potter” to my shopping basket Then my shopping basket should contain 2 copies of “Harry Potter” The same book can be added multiple times to the shopping basket
  43. 43. 43 The same book can be added multiple times to the shopping basket Structure of Scenarios Given my shopping basket contains 1 copy of “Harry Potter” When I add the book “Harry Potter” to my shopping basket Then my shopping basket should contain 2 copies of “Harry Potter” Title: Describes intention/abstract acceptance criterion Arrange: Context, describes state of the system Act: Execution of the feature Assert: Assertion of observable behaviour And I should see the warning: “Book already existed in basket” Triple-A constraint “Checks” Chaining up steps
  44. 44. 44 Automated continuous validation Given my shopping basket contains 1 copy of “Harry Potter” When I add the book “Harry Potter” to my shopping basket Then my shopping basket should contain 2 copies of “Harry Potter” System „Step Definitions“ are binding individual steps to an automatable interface of the application. Automatable interface UI Automation Automation does not necessarily have to bind to the UI. Automatability of system is supported/evolving with development.
  45. 45. 45 UI automation on mobile devices
  46. 46. 46 Available tools/libraries • Android • Android 2.3+: Instrumentation • Android 4.2+: UiAutomator • selendroid • Appium • Robotium • Calaba.sh • Xamarin.UITest • iOS • UIAutomation • Appium • ios-driver • Calaba.sh • Xamarin.UITest • Frank • KIF (Keep it functional)
  47. 47. 47 Selenium: cross platform, existing know how • Android • selendroid • Appium • iOS • Appium • ios-driver
  48. 48. 48 Appium (www.appium.io) • Provides WebDriver API (JSON Wire Protocol) • Uses native UI Automation library • Can test native & hybrid Apps, mobile web apps • Clients for C#, Java, Ruby, Python, PHP, JavaScript • Works with devices and emulators
  49. 49. 49 Examples
  50. 50. 50 Automation with SpecFlow and Appium
  51. 51. 51 SpecFlow & Appium • Write acceptance criteria independent of the platform • Write bindings against a Selenium- like- API • Run it on a devices or emulator
  52. 52. 52 Demo
  53. 53. 53 SpecFlow automation on the controller layer
  54. 54. 54 Different levels of automation Controller Business Layer Data Layer Model View Device automation Trigger behaviour through controller Assert behaviour on model, db, .. Setup pre-conditions through service interfaces Out-of-process In-process
  55. 55. 55 How? • Reuse the same feature files • Write new bindings against the Controller/ViewModel/…
  56. 56. 56 Demo
  57. 57. 57 Faster feedback from tests UI Automation Test Controller Test
  58. 58. 58 Why is it faster? • No overhead for deploying and starting the app • No need to wait for UI animations to finish • No delays for searching UI controls
  59. 59. 59 Advantages Controller Testing • Faster • Better testable App architecture (Xamarin.Forms & MVVM)
  60. 60. 60 Controller tests cannot fully replace UI tests • UI tests are still needed • Diversity of mobile devices • Changes of manufacturer • Different Hardware  Screen sizes  Densities  … • Automated UI tests are still faster than manual testing
  61. 61. 61 Conclusions
  62. 62. 62 Excuses for avoiding mobile test automation • “This is just a small app” • “Only complex end-end tests would make sense” • “Maintaining automated tests is too costly” • Diversity of devices • Small apps also run into regressions • Bad feedback on app store cannot be undone
  63. 63. 63 Automated testing in mobile App development • Business readable automated checks • specification of acceptance criteria • test first development • Testability is an ongoing architectural concern • every new feature needs to be testable • not every acceptance criterion needs to be tested through UI • Automated tests don’t replace manual testing entirely • exploratory testing • assertions that are hard to automate • …
  64. 64. 64 Questions and further information Andreas Willich (awi@techtalk.at) Christian Hassa (ch@techtalk.at)

×