This document describes the architecture of a Twitter scraping application. It uses Gearman for asynchronous job processing to fetch Twitter data from the API or stream. Workers register fetch jobs and process the requests. The application includes modules for the CLI, reader, and streaming components and uses JSON and LWP libraries. It retrieves tweets and stores them in a database for analysis.

2. •
•
•
•

3. •
•

4. •

6. •
–
•
–
•
–
•
–

8. •
•

9. •
•
•
•

11. •
•
•

12. •
•
•
•

13. •
•
•
•

14. •
•

15. •
•

16. yusuke@dev:~/work/twib/Twib/trunk$ ls
bot cli etc migration schema web

17. •
•
•
•
•

18. •
Package Twib::CLI::Reader::Feed;
use JSON::XS;
use LWP::UserAgent;
Package Twib::CLI::Reader::Stream;
use AnyEvent::Twitter::Stream;

19. •
•

20. •
–
•
–
•
–

22. #API Reader
my $client = Gearman::Client‐>new;
$client‐>job_servers(qw|localhost:7003|);
$client‐>dispatch_background( "fetch", $json, {} );
#Worker
my $worker = Gearman::Worker‐>new;
$worker‐>job_servers(qw|localhost:7003|);
$worker‐>register_function( "fetch" => &fetch );
$worker‐>work while 1;
sub fetch {
my $job = shift;
my $args = $job‐>arg;
$fetcher‐>fetch( decode_json($args) );
}

23. •
–
–
•
–
•
–
•
–

24. – –
– –
– –
–
–
–

25. •
•
•
•

26. •
–
•
–

27. •
•
•
•

28. •
•
•
–
•

30. •
•
•
–
–

31. •
–
•
–
•