Gearman is a framework for writing distributed applications across many nodes. It allows you to do work in parallel, load balance processes and write applications across several programming languages. In this presentation we'll learn how to get started writing Gearman-powered applications in Perl.
5. Example 1 – Video Processing
User uploads a video
Server processes the video
Server must transcode the video to
several pre-set resolutions/codecs
Server must extract sample still images
Server must run speech-to-text
recognition to extract subtitles (closed
captions)
Once all of that is completed, server
must update the video metadata to
contain all of the newly available
data and metadata
6. Example 2 – Map/Reduce Style Big Data
User
searches for information
Server must search catalog 1
Server must search catalog 2
…
Server must search catalog n
Server
must return combined
search results to user
7. Example 3 – AntiVirus Scanner
User uploads a file
Server must scan with McAfee
Server must scan with Norton 360
Server must scan with Sophos
…
Server must scan with Engine XYZ
Server returns scan results to user
9. Gearman in Perl
Gearman
/ Gearman::Server (PP)
Gearman::XS (“official” libgearman)
AnyEvent::Gearman and AnyEvent::Gearman::Client (not
the same)
They aren’t 100% up-to-date
They aren’t 100% feature-compatible
The first two are both (individually) good for 90% of use
cases (in my personal experience )
10. Gearman in other languages
C/C++ - official libgearman+ CLI
PHP – GearmanManager – well maintained
framework, PHP_Gearman (PECL), Net_Gearman (Pure PHP)
Node.JS
.NET/C#
JAVA
Python
UDFs – MySQL, PostgreSQL, Drizzle
Write your own to implement the Gearman binary protocol
11. Creating a worker
use strict;
use warnings;
use Gearman::Worker;
my $worker = Gearman::Worker->new;
$worker->job_servers('127.0.0.1:4730');
$worker->register_function('say_hello', &hello);
$worker->work ; # will never return
sub hello {
my $arg = $_[0]->arg;
return "Hello, $argn";
}
13. Writing a client
use strict;
use warnings;
use Gearman::Client;
my $client = Gearman::Client->new;
$client->job_servers('127.0.0.1:4730');
print ${$client->do_task(say_hello => 'world')};
# do_task returns a reference to the response
14. Writing an asynchronous client
use strict;
use warnings;
use Gearman::Client;
my $client = Gearman::Client->new;
$client->job_servers('127.0.0.1:4730');
my $tasks = $client->new_task_set;
$tasks->add_task(say_hello => 'world', {
on_complete => &done
});
$tasks->wait;
sub done {
print ${$_[0]};
}
16. A more sophisticated worker 1/3
use
use
use
use
strict;
warnings;
Gearman::XS qw(:constants);
Gearman::XS::Worker;
my $worker = new Gearman::XS::Worker;
my $ret = $worker->add_server('127.0.0.1');
if ($ret != GEARMAN_SUCCESS) {
printf(STDERR "%sn", $worker->error());
exit(1);
}
$worker->add_options(GEARMAN_WORKER_NON_BLOCKING);
$worker->set_timeout(500);
17. A more sophisticated worker 2/3
$ret = $worker->add_function("hello", 3, &hello, {});
$ret = $worker->add_function("uhoh", 3, &uhoh, {});
if ($ret != GEARMAN_SUCCESS) {
printf(STDERR "%sn", $worker->error());
}
my $go = 1;
$SIG{TERM} = sub {print "Caught SIGTERMn";$go = 0;};
while ($go) {
my $ret = $worker->work(); # will return after 500ms since we set timeout + nonblocking mode above
if (!($ret == GEARMAN_SUCCESS ||
$ret == GEARMAN_IO_WAIT ||
$ret == GEARMAN_NO_JOBS)) {
printf(STDERR "%sn", $worker->error());
}
$worker->wait();
}
18. A more sophisticated worker 3/3
sub hello {
my $job = shift;
$job->send_status(1,2);
my $string = "Hello, “.$job->workload()."n";
$job->send_status(2,2);
return $string;
}
sub uhoh{
my $job = shift;
$job->send_warning("uh oh");
$job->send_data($job->workload() . "n");
$job->send_fail();
}
20. A more sophisticated client 1/3
use
use
use
use
strict;
warnings;
Gearman::XS qw(:constants);
Gearman::XS::Client;
my $client = Gearman::XS::Client->new;
my $task;
my $ret = $client->add_server('127.0.0.1');
if ($ret != GEARMAN_SUCCESS) {
printf(STDERR "%sn", $client->error());
exit(1);
}
23. A more sophisticated client (output)
[issac@localhost ~]$ perl asclient.pl
STATUS: 1/2
STATUS: 2/2
COMPLETE: Hello, world
WARNING: uh oh
DATA: it broke
FAIL: uhoh
[issac@localhost ~]$
24. That’s All, Folks!
Issac Goldstand
issac@itsgoodconsulting.com
Link To Slides
http://www.itsgoodconsulting.com/blog/issac-presenting-distributed-apps-with-gearman-at-telaviv-pm/