CPANTS talk at YAPC::EU 2012

1. CPANTS
Kwalitative website and its tools
Kenichi Ishigaki
(charsbar)
@YAPC::EU 2012
August 22, 2012

2. Kenichi Ishigaki
(charsbar)
From Shibuya.pm,
Tokyo, Japan.

3. Freelancer
- Perl programmer
- Writer/Translator

4. Around 40 CPAN
distributions

5. DBD::SQLite

6. Acme::CPANAuthors

7. We have been
enjoying the
CPANTS game
since 2005.

8. 輝け！全日本最強
CPAN Author
決定選手権
by Koichi Taniguchi
http://blog.livedoor.jp/nipotan/archives/16108466.html

9. He picked up
Japanese authors
by eye.

10. Our names are
easy to find.

11. There were not
so many authors.
- Total: ~4000
- Japanese: ~50

12. YAPC::Asia increased
the number of
Japanese authors.

13. YAPC::Asia / Japanese authors
2006 (Mar) 98
2007 (Apr) 154
2008 (May) 191
2009 (Sep) 228
2010 (Oct) 255
2011 (Oct) 270

14. Needed
something to pick
up Japanese
authors more
easily.

15. That's why I
created a list of
Japanese authors
and a script to
maintain it.

16. I've been
reporting the
Japanese top 10
authors since
2008.

17. I've been adding
something new
every year.

18. 2008: sum of the
kwalitee scores
per author

19. 2009: authors
who released
most in the year

20. 2010: authors/
population ratio

21. 2011: launched a
website (finally)
acme.cpanauthors.org

23. It had one big
problem.

24. No data.

25. The official
CPANTS site had
been down for
some time.

26. I needed to set
up mine.

27. I created a
private repository
and put everything
into it.

28. Merged recent
commits from
domm's
repository.

29. Added a few
columns.

30. Tweaked
Catalyst/DBIC
stuff.

31. It worked.

32. Warnings were
left.

33. I needed to find
some tuits to
remove them.

34. Perl QA
Hackathon

36. Warnings were
removed.

37. Ported some of
the changes I did
locally to daxim's
repository.

38. Showed a new
acme.cpanauthors.org
featuring CPANTS info.

42. Unfortunately,
the porting took
too much time.

43. I didn't merge
the changes back
to my repository.

44. OSDC.TW

45. I finally merged
the changes.

46. Got several
reports that
CPANTS was
broken.

47. What broke
CPANTS was a
small change.

48. "modules" : [
{
"file" : "lib/Path/Extended.pm",
"in_basedir" : 0,
"in_lib" : 1,
"module" : "Path::Extended",
"uses" : {
"Sub::Install" : 1,
"strict" : 1,
"warnings" : 1
}
}
]

49. I don't think this
change is bad.

50. Module::CPANTS::
ProcessCPAN
shouldn't have
died by this.

51. It should have
had tests.

52. Is should have
run faster.

53. It should have
been easier to fix
analysis.

54. Enough issues for
a summer.

55. What should we
do?

56. - We need tests.
- we need to find
test cases.
- we need to do it
many times.

57. Making it run
faster is the
first priority.

58. I wrote a
barebone script
to store data in
parallel.

59. JSON
create table if not exists analysis (
id integer primary key autoincrement,
path text unique,
distv text,
author text,
json text,
duration integer
);

60. Raw SQL
statements

61. Parallel::ForkManager

62. SQLite queue

63. Beware a race condition
my ($id) = $dbh->selectrow_array("
SELECT id FROM queue
WHERE status = 0 LIMIT = 1
");
$dbh->do("
UPDATE queue SET status = 1
WHERE id = ?
", undef, $id);

64. sqlite_update_hook
my $id;
my $dbh->sqlite_update_hook(sub {
(undef, undef, undef, $id) = @_;
});

65. $dbh->do("
UPDATE queue
SET status = 1,
WHERE id IN (
SELECT id FROM queue
WHERE status = 0 LIMIT 1
)
");

66. Archive::Any::Lite

67. Archive::Any::Plugin::Bzip2

68. WorePAN
- Bundling is bad
- We need a specific
version
- Derived from OrePAN

69. use WorePAN;
my $worepan = WorePAN->new(
root => 'path/to/a/directory/',
files => [qw(
I/IS/ISHIGAKI/WorePAN-0.01.tar.gz
)],
use_backpan => 1,
no_network => 0,
cleanup => 1,
);

70. use WorePAN;
my $worepan = WorePAN->new(
root => 'path/to/a/directory/',
files => [qw(
I/IS/ISHIGAKI/WorePAN-0.01.tar.gz
)],
local_mirror => '/home/ishigaki/minicpan/',
no_network => 1,
cleanup => 1,
);

71. use WorePAN;
my $worepan = WorePAN->new(
root => 'path/to/a/directory/',
dists => {
'Catalyst-Runtime' => 5.9,
'DBIx-Class' => 0,
},
cleanup => 1,
);

72. Bonus features
my $worepan = WorePAN->new(
root => 'path/to/a/CPAN/mirror/',
cleanup => 0,
);
my $authors = $worepan->authors;
my $modules = $worepan->modules;
my $file = $worepan->files;
my $dists = $worepan->latest_distributions;

73. $worepan->add_files(qw{
/path/to/a/local/distribution-0.01.tar.gz
});
$worepan->update_indices;

74. Now we have
enough tools.

75. Processing time is
significantly
decreased.

76. What's next?

77. ::Site refactoring

78. I'm preparing the
data now.

79. Creating more
databases/tables.

80. Merging
information from
external sources.
- CPAN indices
- CPAN uploads database

81. Calculating scores
on prerequisite
modules.

82. It will be this
year's something
new in my annual
report.

83. And then, I'll
move on to fixing
the metrics.

84. Some of them are
badly broken.
"versions" : {
"lib/Data/Phrasebook.pm" : "use vars qw($VERSION);¥n",
"lib/Data/Phrasebook/Debug.pm" : "use vars qw($VERSION);¥n",
"lib/Data/Phrasebook/Generic.pm" : "use vars qw($VERSION);¥n",
"lib/Data/Phrasebook/Loader.pm" : "use vars qw($VERSION);¥n",
"lib/Data/Phrasebook/Loader/Base.pm" : "use vars qw($VERSION);¥n",
"lib/Data/Phrasebook/Loader/Text.pm" : "use vars qw($VERSION);¥n",
"lib/Data/Phrasebook/Plain.pm" : "use vars qw($VERSION);¥n",
"lib/Data/Phrasebook/SQL.pm" : "use vars qw($VERSION);¥n",
"lib/Data/Phrasebook/SQL/Query.pm" : "use vars qw($VERSION);¥n"
},

85. Error is not a stash.
"error" : {
"easily_repackageable" : "easily_repackageable_by_fedora",
"easily_repackageable_by_fedora" : "fits_fedora_license",
"metayml_conforms_spec_current" : [
"1.4",
"Expected a map structure from data string or file. [Validation: 1.4]"
],
"metayml_conforms_to_known_spec" : [
"1.0",
"Expected a map structure from data string or file. [Validation: 1.0]"
],
"no_pod_errors" : " home cpants tmp analyze 11442 8001be43fb65..."
}

86. Should have
initialize/finalize phases.
Module::CPANTS::Kwalitee::Distros
doesn't clean up after mirrored
Debian CPANTS file
https://rt.cpan.org/Ticket/Display.html?id=51514

87. There are much more
to do.
- JSON API for metacpan.org and so on.
- Email Reporting like CPAN Testers
- Evaluate new Kwalitee indicators
- New metrics like portable filename
- Blog about recent tendency
- More comprehensive tests
- Analysis per perl version/architecture
- Cover Perl::Critic, CPAN::Critic::Module::Abstract
- 35 RT tickets and several github isses

88. Resources
github.com/charsbar/www-cpants
github.com/charsbar/worepan
github.com/daxim/Module-CPANTS-Analyse

89. Questions?

90. Thank you