An introduction of what are Gearman and its architecture, and how to use it to dispatch jobs in Perl.

1. Dispatching jobs
with Gearman

2. or

3. How to
farm out jobs

4. or

5. Using Gearman
in Perl

6. Using Gearman
in Perl

7. gearman.org

13. The name "Gearman" was chosen as an
anagram for "Manager," "since it dispatches
jobs to be done, but does not do anything
useful itself"

14. Originally written in Perl

15. Rewritten in C

16. Clients in
Perl, PHP, Phython,
Java, C# (.NET)

17. Clients in
Perl, PHP, Phython,
Java, C# (.NET),
even MySQL and PostgreSQL

19. WTF?

20. Part I
Architecture

21. Application
Job

22. Application
Job (or task)

23. Application
Job
gearmand

24. Application
Job
gearmand
Job
Worker

25. Application
Job
gearmand
Job Response
Worker

26. Application
gearmand
Worker
Worker
Worker

27. Application
gearmand
gearmand
Worker
Worker

28. Application
Application
gearmand
gearmand
Worker
Worker

29. Workers are scalable

30. Run any number you need

31. Run on remote servers

32. Application does not care

33. Application only talks with
gearmand server

34. Application only talks with
gearmand server
(one or more)

35. One or more application

36. One or more application
Applications throw jobs

37. One or more job servers

38. One or more job servers
Job servers dispatch jobs

39. One or more workers

40. One or more workers
(clones or different)

41. One or more workers
Workers do jobs

42. One or more workers
Workers do jobs
and may issue new jobs

43. Scalable also means
redundant

44. Scalable also means
redundant and reliable

45. Scalable also means
redundant and reliable*
* when it works properly

46. Different
approaches
to the architecture

47. Synchronous
or
asynchronous

48. Worker either generates
a response or works silently

49. Synchronous
Application
Job Response
gearmand
Job Response
Worker

50. Asynchronous
Application
Job Response
gearmand
Job Response
Worker

51. With a response
Application
Job Response
gearmand
Job Response
Worker

52. With no response
Application
Job
gearmand
Job “I’m done”
Worker

53. With no response
Application
Job
gearmand
Job “I’m done”
Worker
Result
Database

54. Installation

55. Installation
1. Job server
2. Client libraries
3. Worker libraries

56. Installation
1. Job server C
2. Client libraries Perl
3. Worker libraries

57. Installation
1. Job server C
2. Client libraries Perl XS
3. Worker libraries

58. Installing job server
https://launchpad.net/gearmand
$ ./configure
$ make
# make install

59. Installing Perl modules
# cpan Gearman
or
# cpan Gearman::Client
# cpan Gearman::Worker

60. Installing Perl modules
# cpan Gearman
or
# cpan Gearman::XS

61. Starting gearmand deamon
$ gearmand ‐d

62. Starting gearmand deamon
$ gearmand ‐d
or
$ gearmand ‐d
‐L 127.0.0.1
‐p 4730

63. Starting worker(s)
$ ./worker.pl &

64. Starting worker(s)
$ ./worker.pl &
or
$ ./workerA.pl &
$ ./workerB.pl &

65. Starting worker(s)
$ ./worker.pl &
or
$ ./workerA.pl &
$ ./workerB.pl &
$ ./workerA.pl &
$ ./workerB.pl &

66. Running client
$ ./client.pl

67. Worker example
#!/usr/bin/perl
use v5.10;
use strict;
use Gearman::Worker;
my $worker = new Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo => &echo);
$worker‐>work while 1;
sub echo {
my $job = shift;
say '+1';
return $job‐>arg + 1;
}

68. Worker example
my $worker = new Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo => &echo);
$worker‐>work while 1;

69. Worker example
my $worker = new Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo => &echo);
$worker‐>work while 1;
Connect to the server

70. Worker example
my $worker = new Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo => &echo);
$worker‐>work while 1;
Register worker method

71. Worker example
my $worker = new Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo => &echo);
$worker‐>work while 1;
Wait for incoming jobs

72. Worker example
#!/usr/bin/perl
use v5.10;
use strict;
use Gearman::Worker;
my $worker = new Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo => &echo);
$worker‐>work while 1;
sub echo {
my $job = shift;
say '+1';
return $job‐>arg + 1;
}

73. Job method
sub echo {
my $job = shift;
say '+1';
return $job‐>arg + 1;
}
Receive data

74. Job method
sub echo {
my $job = shift;
say '+1';
return $job‐>arg + 1;
}
Evaluate response

75. Client example
#!/usr/bin/perl
use v5.10;
use strict;
use Gearman::Client;
my $client = new Gearman::Client;
$client->job_servers('127.0.0.1:4730');
for (1..10000) {
my $result_ref = $client->do_task('echo', 10);
say $$result_ref;
}

76. Client example
for (1..10000) {
my $result_ref =
$client->do_task('echo', 10);
say $$result_ref;
}
Issue jobs

77. Asynchronous client example
use v5.10;
use strict;
use Gearman::Client;
my $client = new Gearman::Client;
$client->job_servers('example.com:4730');
my $tasks = $client->new_task_set;
$tasks->add_task(
'echo' => 10,
{
on_complete => &completed
}
);
$tasks->wait;
sub completed {
my $result = shift;
say $$result;
}

78. Asynchronous client example
my $tasks = $client->new_task_set;
$tasks->add_task(
'echo' => 10,
{
on_complete => &completed
}
);
$tasks->wait;
Get task set placeholder

79. Asynchronous client example
my $tasks = $client->new_task_set;
$tasks->add_task(
'echo' => 10,
{
on_complete => &completed
}
);
$tasks->wait;
Place the task

80. Asynchronous client example
my $tasks = $client->new_task_set;
$tasks->add_task(
'echo' => 10,
{
on_complete => &completed
}
);
$tasks->wait;
and register callback method

81. Asynchronous client example
my $tasks = $client->new_task_set;
$tasks->add_task(
'echo' => 10,
{
on_complete => &completed
}
);
$tasks->wait;
Wait until the job is done

82. Asynchronous client example
sub completed {
my $result = shift;
say $$result;
}
Use result somehow

83. XS worker example
use v5.10;
use strict;
use Gearman::XS::Worker;
my $worker = new Gearman::XS::Worker;
$worker->add_server('127.0.0.1', 4730);
$worker->add_function('echo', 0, &echo, undef);
$worker->work() while 1;
sub echo {
my $job = shift;
return $job->workload() + 1;
}

84. XS client example
use v5.10;
use strict;
use Gearman::XS::Client;
my $client = new Gearman::XS::Client;
$client->add_server('127.0.0.1', 4730);
my ($ret, $result) = $client->do('echo', '15'); # arg as string!
say $result;

85. Async XS client example
use v5.10;
use strict;
use Gearman::XS::Client;
my $client = new Gearman::XS::Client;
$client->add_server('127.0.0.1', 4730);
my ($ret1, $task1) = $client->add_task('echo', '15');
my ($ret2, $task2) = $client->add_task('echo', '51');
$client->run_tasks();

86. Monitoring job server

87. TCP protocol

88. Connect with telnet

89. $ telnet localhost 4730

90. $ telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

91. $ telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
status

92. $ telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
status
echo2 1 1 2
echo 5 2 5
.

93. echo2 1 1 2
echo 5 2 5
.

94. Known worker methods
echo2 1 1 2
echo 5 2 5
.

95. Jobs running
echo2 1 1 2
echo 5 2 5
.

96. Jobs queued
echo2 1 1 2
echo 5 2 5
.

97. Workers available
echo2 1 1 2
echo 5 2 5
.

98. Part III
Step by step
example

99. worker.pl
for(...) {
... worker.pl
} worker.pl
worker.pl

100. Live demo

101. __END__
Andrew Shitov
andy@shitov.ru