The document discusses using the MetaCPAN API to retrieve and filter data from the MetaCPAN database. It provides examples of using the API to fetch author data for a specific author, search for authors by profile or country, and filter results. It also covers more advanced techniques like using Boolean filters, sorting results, and pagination via the scrolling API. Additional resources and tips are provided, such as caching responses and enabling compression.

1. (Ab)Using the
MetaCPAN API for
Fun and Profit
Olaf Alders (OALDERS)
@wundercounter

2. Architecture
• Built on ElasticSearch
• Uses Catalyst as a thin wrapper
• You don’t need to know this

3. Real life examples

4. iCPAN - iPhone

5. iCPAN - iPad

6. Android

17. What can we build?

18. What can we build?
• Do something with Github

19. What can we build?
• Do something with Github
• Get a list of all CPAN authors who
have enabled the “hireable” flag in
their Github profiles

20. Let’s Get Started

21. Let’s Get Started
• We want to fetch some data

22. Let’s Get Started
• We want to fetch some data
• We’ll use Sawyer’s MetaCPAN::API

23. #!/usr/bin/env perl
use strict;
use warnings;
use MetaCPAN::API;
my $mcpan = MetaCPAN::API->new();
my $author = $mcpan->author('MSTROUT');

24. {
dir => "id/M/MS/MSTROUT",
email => ["perl-stuff@trout.me.uk"],
gravatar_url => "https://secure.gravatar.com/avatar/...",
name => "Matt S Trout",
pauseid => "MSTROUT",
website => ["http://www.trout.me.uk/"],
}

27. MetaCPAN Explorer

28. my $author = $mcpan->author('MSTROUT');

29. my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
size => 1,
},
);

30. {
_shards => { failed => 0, successful => 5, total => 5 },
hits => {
hits => [
{
_id => "KHAMPTON",
_index => "cpan_v1",
_score => 1,
_source => {
city => "Los Angeles",
country => "US",
dir => "id/K/KH/KHAMPTON",
email => ["khampton@totalcinema.com", "kip.hampton@tamarou.com"],
gravatar_url => "http://www.gravatar.com/avatar/...",
name => "Kip Hampton",
pauseid => "KHAMPTON",
profile => [
{ id => "ubu", name => "coderwall" },
{ id => "ubu", name => "github" },
{ id => "kiphampton", name => "twitter" },
],
region => "CA",
updated => "2011-07-22T20:42:06",
website => ["http://totalcinema.com/"],
},
_type => "author",
},
],
max_score => 1,
total => 9780,
},
timed_out => bless(do{(my $o = 0)}, "JSON::XS::Boolean"),
took => 1,
}

31. my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
size => 1,
},
);
# dump $result->{hits}->{hits}->[0]->{_source};

32. {
city => "Los Angeles",
country => "US",
dir => "id/K/KH/KHAMPTON",
email => ["khampton@totalcinema.com", "kip.hampton
@tamarou.com"],
gravatar_url => "http://www.gravatar.com/avatar/...",
name => "Kip Hampton",
pauseid => "KHAMPTON",
profile => [
{ id => "ubu", name => "coderwall" },
{ id => "ubu", name => "github" },
{ id => "kiphampton", name => "twitter" },
],
region => "CA",
updated => "2011-07-22T20:42:06",
website => ["http://totalcinema.com/"],
}

33. my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
size => 100,
},
);

34. my $filter = {
{ term => { 'author.profile.name' => 'stackoverflow', } },
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);

35. use Pithub;
my $p = Pithub->new;
AUTHOR:
foreach my $author ( @{ $result->{hits}->{hits} } ) {
foreach my $profile ( @{ $author->{_source}->{profile} } ) {
if ( $profile->{name} eq 'github' ) {
my $username = $profile->{id};
$username =~ s{https?://github.com/(w*)/?}{$1}i;
next AUTHOR if !$username;
if ( $p->users->get( user => $username )->content->{hireable} ) {
# do something...
}
next AUTHOR;
}
}
}

36. Getting fancy

37. my $filter = {
and => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } }
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);

38. my $filter = {
and => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);

39. my $filter = {
and => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);
# “missing” isn’t really helpful in this search
# just an example of how you might use it

40. my $filter = {
or => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
},
);

41. my $filter = {
or => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
fields => [ 'pauseid', 'country' ],
},
);

42. my $filter = {
or => [
{ term => { 'author.profile.name' => 'github', } },
{ term => { 'author.country' => 'US', } },
{ exists => { 'field' => 'author.region' } },
{ missing => { 'field' => 'author.location' } },
]
};
my $result = $mcpan->post(
'author',
{ query => { match_all => {} },
filter => $filter,
size => 100,
sort => [ { 'author.pauseid' => 'ASC' } ],
},
);

43. Getting Help
• #metacpan or irc.perl.org
• https://metacpan.org/about/resources

44. Resources
• https://github.com/CPAN-API/cpan-
api/wiki/Beta-API-docs
• http://www.slideshare.net/
clintongormley/terms-of-endearment-
the-elasticsearch-query-dsl-explained

45. Bonus Slides

46. Base URL
• http://api.metacpan.org/v0

47. Convenience Endpoints

48. Convenience Endpoints
• /author/DOY
• /distribution/Moose
• /release/Moose
• /module/Moose
• /pod/Moose

50. Exporting Pod
• /pod/Moose?content-type=text/html (default)
• /pod/Moose?content-type=text/plain
• /pod/Moose?content-type=text/x-pod
• /pod/Moose?content-type=text/x-markdown

51. The (real) Endpoints

52. The (real) Endpoints
• /author

53. The (real) Endpoints
• /author
• /distribution

54. The (real) Endpoints
• /author
• /distribution
• /favorite

55. The (real) Endpoints
• /author
• /distribution
• /favorite
• /rating

56. The (real) Endpoints
• /author
• /distribution
• /favorite
• /rating
• /release

57. The (real) Endpoints
• /author
• /distribution
• /favorite
• /rating
• /release
• /file

58. The (real) Endpoints
• /author
• /distribution
• /favorite
• /rating
• /release
• /file

60. Using a cache
use HTTP::Tiny::Mech;
use MetaCPAN::API;
use WWW::Mechanize::Cached;
my $mcpan = MetaCPAN::API->new(
ua => HTTP::Tiny::Mech->new(
mechua => WWW::Mechanize::Cached->new()
)
);

62. Enable Compression
• use WWW::Mechanize::Gzip
• use WWW::Mechanize::Cached::Gzip
• Or set the appropriate request header

63. Use the scrolling API
• The scrolling API allows you to iterate
over an arbitrary number of results
• Be aware that when you scroll, your
docs will come back unsorted