TAP-Harness + friends

TAP::Harness
(+ friends)

by Steve Purkis

Friday, 27 January 2012

About Steve...
• Software developer & manager
• Sporadic TAP::Harness dev
• did a lot of refactoring

• redesigned the plugin framework w/David Wheeler

• wrote the HTML formatter


What we’ll cover...
• What’s TAP?
• TAP::Harness Overview
• Plugins & Module::Build integration
• TAP::Formatter::HTML
• TAP & Hudson CI


Unit Tests?
• Surely you all write tests by now...
• right?


Why Test?
• “Kwalitee”
• measurable approximation of quality (http://qa.perl.org/phalanx/kwalitee.html)
• prevent regressions
• gives others confidence what you’ve written is working

• developer aid
• break solutions down into chunks
• test your assumptions
• gives you confidence each chunk is working (so you can forget about it and move onto the next chunk!)

• team co-ordination
• gives your team confidence what you’re committing is working
• right up there with version control, bug tracking, dev environments & build tools


What’s TAP?
• “Test Anything Protocol”
• Grew from Perl’s test output format:
1..7
ok 1 - use Test::Approx;
ok 2 - equal strings
ok 3 - equal numbers
ok 4 - completely different strings
ok 5 - similar strings, 20% threshold
ok 6 - similar strings, 10% threshold
ok 7 - big strings, default threshold

See: http://testanything.org/

What’s TAP?
Origins
=head1 AUTHORS

Either Tim Bunce or Andreas Koenig, we don't know. What we know for sure
is, that it was inspired by Larry Wall's TEST script that came with perl
distributions for ages. Numerous anonymous contributors exist. Andreas
Koenig held the torch for many years, and then Michael G Schwern.
- http://cpansearch.perl.org/src/PETDANCE/Test-Harness-2.56/lib/Test/Harness.pm

For more details:
http://testanything.org/wiki/index.php/TAP_History

See: http://testanything.org/

What’s TAP?
• Now available in
many languages!

• A few projects I ﬁnd interesting:

• node.tap

• MyTAP

• pgTAP

• PHPUnit


TAP Structure
Example 1 Example 2
TAP version 13 1..3
# Generated from test.pl not ok 1 - get remote # TODO not yet implemented
1..13 # Failed (TODO) test 'get remote'
ok 1 # at test.pl line 18.
ok 2 - a good test not ok 2 - a bad test
not ok 3 - a bad test # Failed test 'a bad test'
# Failed test 'a bad test' # at test.pl line 11.
# at test.pl line 11. Bail out! I can't take it anymore
Oops, I printed to STDOUT! ok 3 - a good test
Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run.
not ok 4 - get remote # TODO not yet implemented
# Failed (TODO) test 'get remote'
# at test.pl line 18.
ok 5 # skip to the loo!
ok 6 - open connection to server # MyDirective with more info
# Looks like you planned 13 tests but ran 6.
# Looks like you failed 1 test of 6 run.

http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#Grammar


TAP Structure
Example 1 Example 2
TAP version 13 Optional TAP version, must come 1st 1..3



TAP Structure
Example 1 Example 2
TAP version 13 1..3
# Generated from test.pl Comments start with # not ok 1 - get remote # TODO not yet implemented



TAP Structure
Example 1 Example 2
TAP version 13 1..3
1..13 Test Plan # Failed (TODO) test 'get remote'



TAP Structure
Example 1 Example 2
TAP version 13 1..3
ok 1 test status & identiﬁer # at test.pl line 18.



TAP Structure
Example 1 Example 2
TAP version 13 1..3
not ok 3 - a bad test test description # Failed test 'a bad test'



TAP Structure
Example 1 Example 2
TAP version 13 1..3
Oops, I printed to STDERR!
non-TAP: ignored by parser # Looks like you failed 1 test of 3 run.



TAP Structure
Example 1 Example 2
TAP version 13 1..3
not ok 4 - get remote # TODO not yet implemented TODO directive



TAP Structure
Example 1 Example 2
TAP version 13 1..3
ok 5 # skip to the loo! SKIP directive



TAP Structure
Example 1 Example 2
TAP version 13 1..3
ok 6 - open connection to server # MyDirective with more info custom directive



TAP Structure
Example 1 Example 2
TAP version 13 1..3
# at test.pl line 11. Bail out! I can't take it anymore stop testing & parsing



TAP Lifecycle

http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard


Why TAP?
• Advantages:
• Simple

• Human-readable

• Easy to generate & parse

• Widely used by the Perl community


What else is there?
• Similar output formats:
• JUnit report (XML generated by most xUnit libs & other tools like
googletest)
• SubUnit
- probably many more that I don’t know of...

• Complementary tools:
• A QA team ;-)
• CI & Build tools (CPAN, Smolder, Hudson, buildbot, etc)

- probably many more that I don’t know of...


Up Next...
• What’s TAP?
• TAP & Hudson CI


What’s TAP::Harness?
• A set of extensible tools that make it easy
to produce & consume TAP

- prove cmdline utility

- Module::Build & ExtUtils::MakeMaker hooks

- Parser, Source Handlers & Formatters

- Plugin framework


Using TAP::Harness?
• You're already using it!
• Several years ago Test::Harness became a compatibility wrapper around TAP::Harness
• The old Test::Harness has been retired: it was getting hard to maintain & extend

• Thanks to these ﬁne folk:

Michael G. Schwern
Andy Lester
Ovid
Andy Armstrong
Eric Wilhelm
& more!


Using TAP::Harness?
prove from Perl
prove -l t use TAP::Harness;

my @tests = glob( 't/*.t' );
Module::Build my $harness = TAP::Harness->new({
formatter_class =>
perl Build.PL 'TAP::Formatter::JUnit',
./Build test merge => 1
});
$harness->runtests( @tests );
ExtUtils::MakeMaker
perl Makefile.PL
make test


Under the hood...
TAP
...
prove Formatter

Console File
Module::Build Harness Color plugins
Backwards compatibility

EU::MakeMaker Parser Parser
SourceHandler
Aggregator
Test::Harness Perl Executable
Lots of hidden
complexity here! File Handle

RawTAP plugins


Class diagram notes
Class diagram

Red: plugins are available
separate plugin systems!

prove & App::Prove

TAP::Harness

cmdline utility to run tests & display results

main interface to TAP::*

parses args

loads formatter & sourcehandler plugins

loads App::Prove plugins

creates parsers

kicks off TAP::Harness

aggregates results

formats data

Parser

Parser::Aggregator

SourceHandlers

1 parser per stream (ie: test script)

collate results across many TAP streams

what actually run your tests

has lots of helpers, not shown here

because: 1 parser per stream!

or open a ﬁle...

opens TAP stream

stream input into the parser

parses input as it arrives

sends events to the aggregator


Customising?
• Write Plugins
• Sub-class


Available Extensions
• Plugins:
• Formatters: JUnit, HTML, Spreadsheet, TextMate, TeamCity
• Sources: pgTAP, MyTAP, PHP
• Prove: Idempotent

• Known Subclasses
• TAP::Harness::JUnit
• TAP::Harness::Multiple

• Other tools
•

TAP::Filter

• All as of Jan 2012...

Where’s T::H Going?
• Where is TAP::Harness going?
• Obviously: ongoing maintenance releases
• I’d like to see:
• Multiple Formatters
• Arbitrary plugin args

• The rest is up to you!
• write plugins!
• ﬁnd bugs!
• send patches!
• get involved.


App::Prove plugin...
package App::Prove::Plugin::Foo;
# From http://search.cpan.org/dist/Test-Harness/lib/App/Prove.pm#Sample_Plugin
# Sample plugin, try running with:
# prove -PFoo=bar -r -j3
# prove -PFoo -Q
# prove -PFoo=bar,My::Formatter

use strict;
use warnings;

sub load {
my ($class, $p) = @_;
my @args = @{ $p->{args} };
my $app = $p->{app_prove};

print "loading plugin: $class, args: ", join(', ', @args ), "n";

# turn on verbosity
$app->verbose( 1 );

# set the formatter?
$app->formatter( $args[1] ) if @args > 1;

# print some of App::Prove's state:
for my $attr (qw( jobs quiet really_quiet recurse verbose )) {
my $val = $app->$attr;
$val = 'undef' unless defined( $val );
print "$attr: $valn";
}

return 1;
}

1;


Formatter plugin...
package TAP::Formatter::Text; package TAP::Formatter::Text::Session;

use strict; use strict;
use warnings; use warnings;

use TAP::Formatter::Text::Session; sub new {
my ( $class, $args ) = @_;
sub new { my $self = bless {%$args}, $class;
my ( $class, $args ) = @_; }
my $self = bless {}, $class;
} sub header {
my ( $self ) = @_;
sub prepare { print "Test start: $self->{name}n";
my ( $self ) = @_; }
print "getting prepared!n";
} sub result {
my ( $self, $result ) = @_;
sub open_test {
my ( $self, $test, $parser ) = @_; my $str = '';
my $session = TAP::Formatter::Text::Session->new({ if ($result->is_test) {
name => $test $str = $result->is_ok ? '+' : '!!';
}); } elsif ($result->is_comment) {
$session->header; $str = '#';
return $session; }
}
if ($result->has_directive) {
sub summary { $str .= '->';
my ( $self, $test, $parser ) = @_; }
print "Text plugin summary!n"; print " $strt" . $result->as_string . "n";
} }

# shouldn't really need to create these... sub close_test {
for (qw( directives verbosity timer failures comments my ( $self, $test, $parser ) = @_;
errors stdout color show_count normalize )) { print "Session closed!n";
eval "sub $_ {}"; }
}
1; 1;

SourceHandler plugin...
package TAP::Parser::SourceHandler::Web;
sub make_iterator {
use strict; my ($class, $source) = @_;
use warnings;
$class->_croak('$source->raw must be a scalar ref')
use File::Slurp qw( read_file ); unless $source->meta->{is_scalar};
use LWP::UserAgent;
use TAP::Parser::IteratorFactory; my $file = ${ $source->raw };
use TAP::Parser::Iterator::Array; my $uri = read_file( $file );
chomp $uri;
use base qw( TAP::Parser::SourceHandler );
my $ua = LWP::UserAgent->new;
TAP::Parser::IteratorFactory->register_handler( __PACKAGE__ ); $ua->agent("TAP-Parser-SourceHandler-Web/0.1 ");
my $req = HTTP::Request->new(GET => $uri);
sub can_handle { my $res = $ua->request($req);
my ( $class, $src ) = @_;
my $meta = $src->meta; # Check the outcome of the response
my $config = $src->config_for( $class ); $class->_croak("couldn't GET $uri: " . $res->status_line)
unless $res->is_success;
return 0 unless $meta->{is_file};
if (my $ext = $meta->{file}->{ext}) { my $tap = [ split(/n/, $res->content) ];
return 0.99 if $ext =~ /.ur[il]$/i; my $iterator = TAP::Parser::Iterator::Array->new($tap);
}
return $iterator;
return 0; }
}
1;


Module::Build
• Using custom sources & formatters
my $build = Module::Build->new
(
...
build_requires => {
'TAP::Harness' => '3.22',
'Test::More' => '0.01',
'Module::Build' => '0.30',
},

# TAP::Harness configuration
test_file_exts => [qw(.t .tap .txt )],
use_tap_harness => 1,
tap_harness_args => {
sources => {
Web => { },
File => {
extensions => ['.tap', '.txt'],
},
},
color => 1,
formatter_class => 'TAP::Formatter::Text',
},

);


TAP::Formatter::HTML

Interpret your test results with HTML
Based on Yuval Kogman’s Test::TAP::HTMLMatrix


Common uses
• Integration tests
• Nightly smokes
• Being lazy
• avoid scrolling through tonnes of output


Demo
• Download from CPAN
• runprove -m -Q -P HTML=outfile:output.html


Meet Hudson
• Free Java CI
servlet
- http://hudson-ci.org/
• Consumes JUnit XML
reports

• Provides useful stats &
reports


JUnit XML?
• Lots of build tools unit test frameworks
use it!
• Current options for TAP::Harness:
• TAP::Formatter::JUnit (alpha)


Demo
• 3 build tasks:
• TAP::Formatter::JUnit


Hudson conﬁg
TAP::Harness::JUnit:
Execute:
cd /Users/spurkis/dev/Perl-accessors
# For DEMO ONLY:
# avoid running perl here; defaults to junit_output.xml
# so just move the file to Hudson's workspace
prove -l --harness TAP::Harness::JUnit || echo 1
mv junit_output.xml "$WORKSPACE/test-report-$BUILD_ID.xml"
Publish JUnit test report: test-report*.xml

TAP::Formatter::JUnit:
Execute:
prove -l --formatter TAP::Formatter::JUnit > "$WORKSPACE/test-report-$BUILD_ID.xml"
Publish JUnit test report: test-report*.xml

TAP::Formatter::HTML:
Prep:
cp /usr/local/perl-5.10.1/lib/site_perl/5.10.1/TAP/Formatter/HTML/*.[cj]s* $HUDSON_HOME/userContent
Execute:
prove -lm -P HTML=outfile:$WORKSPACE/test-report-$BUILD_ID.html,force_inline_css:0,css_uri:/userContent/default_page.css,css_uri:/
userContent/default_report.css,js_uri:/userContent/jquery-1.4.2.min.js,js_uri:/userContent/jquery.tablesorter-2.0.3.min.js,js_uri:/
userContent/default_report.js
Archive artifacts: test-report*.html


That’s all...
• What’s TAP?
• TAP & Hudson CI


See Also
http://testanything.org/
http://search.cpan.org/dist/Test-Harness/
http://search.cpan.org/dist/Test-Harness/lib/TAP/Harness/Beyond.pod
http://search.cpan.org/dist/TAP-Formatter-HTML/
http://search.cpan.org/dist/TAP-Formatter-JUnit/
http://search.cpan.org/dist/TAP-Harness-JUnit/
http://search.cpan.org/dist/Smolder/
http://hudson-ci.org/


Questions
Test::Harness Developers
tapx-dev@hexten.net
patches welcome!

Steve Purkis
spurkis@cpan.org


Ideas: Future Topics
• Extending TAP: adding a custom
TAP::Parser::Result
• Creating a new TAP::Parser::Grammar
• Map of TAP::Parser internals


TAP-Harness + friends

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (11)

Ähnlich wie TAP-Harness + friends

Ähnlich wie TAP-Harness + friends (20)

Mehr von Steve Purkis

Mehr von Steve Purkis (15)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

TAP-Harness + friends