TAP In Depth

1. TAP in Depth Michael Peters Plus Three, LP YAPC::NA 10 Pittsburgh, PA

2. TAP Test Anything Protocol ● And we do mean *anything* ● Mainly used for regression testing code ● Could be used for systems monitoring, performance testing, etc ● 1987, current version 13 ● Patient Zero was Perl, but the infection has since spread to lots of other languages ● Ongoing work to make an IETF standard

3. Definitions Test ● or an "assertion" (ok/not ok) ● smallest granule of measurement ● did it do what we wanted Stream ● or a "set" ● a group of tests ● usually output of running a single test file, but is not necessarily tied to a file or any specific output. Test Suite ● A group of TAP streams

4. Definitions Harness ● A process which collects a group of TAP streams Plan ● number of tests that are expected to be run in a given stream Directive ● Changes the meaning of a given test ● TODO/SKIP Reason ● Explains the "why" of a directive

5. Definitions Description ● What is the test asserting? What functionality is being tested? Producer ● Anything that generates TAP Consumer ● Anything that processes TAP ● Could just blindly echo it ● Usually parses it

6. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed

7. Document Structure Line Oriented Protocol ● Everything is separated by a new-line (CRLF) ● Not everything fits on 1 line ● Everything starts and ends on it's own line Encoding ● Everything is UTF-8 ● Unless the Producer/Consumer agree otherwise 1 Version, 1* plans, 0+ tests, 0* comments, 0* ignored elements

19. Pass vs Fail Tests ● "ok" == pass ● "not ok" == fail ● Yay! binary!

20. Pass vs Fail Tests ● "ok" == pass ● "not ok" == fail ● If only it were that simple ● Directives! Normal TODO SKIP ok pass * pass not ok pass fail ** * a passing TODO might indicate that a test shouldn't be a TODO anymore ** SKIP means the test wasn't run, so it shouldn't be able to fail

21. Pass vs Fail Streams ● Can fail if any of the following are true (1) Any tests in this stream failed (2) Tests run did not agree with the plan (3) Other factors ● process exit codes (non-zero) ● HTTP status code ● Passed only if none of the above are true and (1) There was at least 1 passing test - or - (2) There was a "skip all" ● 1..0 # SKIP some module not installed

22. Pass vs Fail Test Suites ● Fails if there are any streams that failed ● Passes if nothing failed and there's at least 1 stream ● Pass or fail at the suite level is boolean ● Tools can show percentages ● But even a 99.999% pass is still a fail

23. Producing Perl Modules ● Test::More (classic and for good reasons)

24. Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder

25. Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder ● Test::Most

26. Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder ● Test::Most ● Test::Class

27. Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder ● Test::Most ● Test::Class ● Test::LongString, Test::WWW::Mechanize, Test::Exception, Test::File::Contents, Test::HTML::Content, Test::MockObject, Test::Pod, Test::Warn, Test::WWW::Selenium, Test::NoWarnings, Test::Output, Test::Number::Delta, Test::Perl::Critic

28. Producing Other Languages ● Javascript →Test.Simple ● C/C++ →libtap, libtap++ ● PHP → phpt, PHPUnit, SimpleTest or Apache-Test ● Java →JTap ● Ruby →taptap, Bacon ● Python → PyTAP ● Also Haskell, Erlang, Forth, C#, Shell, Postgres

29. Consuming Perl ● Test::Harness ● prove ● TAP::Harness ● TAP::Harness::Archive ● GUI Test Runner ● Smolder

30. Comming Soon to a terminal near you Nested TAP ● Group related blocks of TAP together in the same stream ● Mimic xUnit style ● Aggregate TAP streams

31. Comming Soon to a terminal near you Nested TAP 1..3 1..3 ok 1 - guests arrived on time ok 2 - table was available ok 3 - order correct ok 1 - guests served 1..2 ok 1 - chicken is fried not ok - steak is medium well not ok 1 - food prepared properly 1..2 ok 1 # SKIP guests did not come back ok 2 # SKIP guests did not come back ok 1 # SKIP guests did not come back

32. Comming Soon to a terminal near you Metadata ● Everything is crammed into the descriptions or comments not ok - fooey, Bug #123 # Failed test 'fooey, Bug #123' # at barry.pl line 20. # got: foo bar # expected: bar foo ● If you want to do anything useful with the information in that message you need to parse it. Parsing error messages is evil

33. Comming Soon to a terminal near you Metadata ●Data should be structured (YAML-ish) not ok - fooey --- file : barry.pl line : 20 want : foo bar have : bar foo display : > Failed test 'fooey' at barry.pl line 20. want: "foo bar" have: "bar foo" Bug-ID : 123 ...

34. Comming Soon to a terminal near you Metadata ● Opens up a whole world of possibilities ● Visual diffs on want/have ● Tagging ● Linking between systems (source code, specs, issue tracking) ● Consumers can show or hide what ever they want or understand. ● Unknown uses (standardization vs growth)

35. Usage Scenarios Typical Perl tests

36. Usage Scenarios Automated Javascript Testing * example from Michael Schwern - http://use.perl.org/~schwern/journal/39092

37. Usage Scenarios Automated Javascript Testing

38. Usage Scenarios Automated Javascript Testing

39. TAP in Depth Michael Peters Plus Three, LP Thanks! Questions?

TAP In Depth

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (9)

Ähnlich wie TAP In Depth

Ähnlich wie TAP In Depth (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

TAP In Depth