1. OpsTools
with Perl
Hokkaido.pm #7
2012/05/12
riywo

2. Self Introduction
Ryosuke IWANAGA
a.k.a. riywo(りーお)
DeNA(2009∼)
Mobage Server-side
Ops-Engi / DBA / Manager

3. Acknowledgement
Thanks for JPA and Hokkaido.pm :)
地域PM向け講師派遣支援プログラム
http://japan.perlassociation.org/
services/send-a-monger

4. But...
Am I a Perl Monger?
I’m not a Programer
Mobage uses Perl heavily but I have
never written the business logic...
But, I read Perl heavily
Because I should cope with all system
trouble including application bugs ;P
I wrote some operation tools with Perl
And I also use some Perl tools

5. But...
Am I a Perl Monger?
So...
I’ll try to talk about my knowledge, thought,
and tools as much as possible
しょぼくてごめんなさい＞＜

6. Perl Ops-Tools
Introduction

7. kazeburo-ware

8. http://blog.nomadscafe.jp/
@kazeburo

9. st
ca
re
Fo
d
u
lo
C
http://www.slideshare.net/kazeburo/introduction-to-cloudforecast-yapcasia-2010-tokyo

10. CloudForecast
Server Resource Collecting/Monitoring
Easy install
git clone & cpanm --installdeps .
SNMP / RRDs is needed
LightWeight - about 5,000 lines (*.pm)
WAF: Shirahata.pm
See also
リソースモニタリングツール「CloudForecast」入門 - As a Futurist...
( http://blog.riywo.com/2011/02/27/043646 )

11. s t
ca
re
Fo
th
w
ro
G
http://blog.nomadscafe.jp/2011/12/growthforecast.html
http://blog.nomadscafe.jp/2012/02/growthforecast1json.html
http://d.hatena.ne.jp/tagomoris/20120218/1329558305

12. GrowthForecast
API and Graph Render of RRDTool
Easy install
git clone & cpanm --installdeps .
RRDs is needed
LightWeight - about 1,800 lines (*.pm)
WAF: Kossy.pm
See also
GrowthForecastというグラフ表示ツールで る話 - blog.nomadscafe.jp
( http://blog.nomadscafe.jp/2011/12/growthforecast.html )

13. **Alien::RRDtool**
RRDs is not CPAN module...
Alien::RRDtool Install rrdtool as CPAN module
extlib/lib/perl5/{archname}/RRDs.pm
extlib/lib/perl5/auto/share/dist/Alien-RRDtool/
Especially useful for perlbrew
* Developed by gfx
See also
Alien::RRDtool - RRDtoolをCPANから入れる - Perl Advent Calendar
Japan 2011 Hacker Track( http://perl-users.jp/articles/advent-
calendar/2011/hacker/2 )

14. chase-tail

15. chase-tail
$ tail -f error_log | chase-tail -
l 10 -t various_error
One sheet perl script
Highlight some keywords
Detect fast flooding of log
See also @hirose31
垂れ流されるログのおしりを追いかける - (ひ)メモ( http://
d.hatena.ne.jp/hirose31/20120120/1327025989 )

16. App::Ikachan

17. App::Ikachan
$ curl -F channel=#yappo -F
message=oppai http://hoge.com/
notice
HTTP API of IRC posting
use it from any languages
Easy install (git clone & cpanm) @Yappo
See also
YappoLogs: App::Ikachan - 様々なサーバのバッチ処理の結果等を
IRC のチャンネルに通知するサーバ( http://blog.yappo.jp/yappo/
archives/000760.html )

18. Percona Toolkit

19. Percona Toolkit
Mixed tools - Maartkit and Aspersa
Almost all tools are written in Perl
System utility / MySQL Ops @Percona
“pt-query-digest” is quite efficient for
performance tuning of MySQL server
See also
Percona Toolkit - Percona Software( http://www.percona.com/
software/percona-toolkit/ )

20. Touryo

21. “Touryo” is configuration
management tool
I needed a Configuration Management Tool
Chef is good but difficult to customize
So, I started to think a plan of Touryo
Simple and Lightweight
Testable
able to write DSL in Perl freely

22. How to write DSL in
Perl?
I didn’t know it X(
I asked gfx(@__gfx__) who is a great Perl
Monger and works at DeNA

23. use strict; [dsl.pl]
use warnings;
test {
Hoge->do_test; my $hoge = shift;
print "Hello $hoge!n";
};
#---------------------
package Hoge;
my $func;
sub Hoge::test(&) {
my $block = shift;
$func = $block;
}
sub do_test {
package Hoge;
do "dsl.pl";
$func->("DSL");
}

25. What can Touryo do?
$ touryo test web01
web01
crontab apache FAIL?
$ touryo run web01
sysctl fastcgi admin:
touryo
rpms app
game1:web
- “touryo test host”
- check the state of the host
db01 db02 - correctly installed?
crontab crontab - correctly configured?
sysctl sysctl
rpms rpms - “touryo run host”
mysql - setup to the “correct” state
mysql
game1:
db:master
backup -> Then, what is the “correct” state?
game1:
db:backup

26. How to define the “correct” state? (1)
Touryo config
web01 role: game1
blueprint:
crontab apache
- crontab
sysctl fastcgi - sysctl
- rpms
rpms app
role: game1:web role: game1:db
game1:web
blueprint: blueprint:
- apache - mysql
db01 - fastcgi role: game1:db:backup
db02 - app blueprint:
crontab crontab - backup
sysctl sysctl
admin:
rpms rpms touryo - Touryo use “blueprint” config
mysql mysql - means what should be installed
game1: - The config defined by “role”
backup
db:master
game1:
- should be defined the role of hosts
db:backup - inherited searching
- separated by comma

27. What is the “blueprint”?
crontab blueprint
blueprint/crontab/root.tx
web01 0 <: $cron_hour :> * * * /path/to/command
crontab apache blueprint/crontab/blueprint.pl
admin:
touryo
sysctl fastcgi bootstrap {
my $c = shift;
rpms app $c->add_attribute({
cron_hour => 5,
game1:web
cron => 'root.tx',
});
- blueprint locate the directory };
db01blueprint name
- named the
db02
crontab test {
- contain “blueprint.pl”crontab my $c = shift;
sysctl
- Perl DSL for Touryo sysctl is_diff $c->sudo_ssh->crontab,
- able to use templaterpms engine $c->template('cron')->content;
rpms done_testing;
- Text::Xslate::Syntax::Kolon };
mysql mysql
- using “attribute” valuables
game1:
-db:master backup
have other useful functions run {
game1: my $c = shift;
- sudo_ssh, template, crontab
db:backup $c->sudo_ssh->crontab($c->template('cron'));
- etc... };

28. What is the “attribute”?
blueprint/crontab/root.tx
web01 0 <: $cron_hour :> * * * /path/to/command
admin:
crontab apache blueprint/crontab/blueprint.pl touryo
bootstrap {
sysctl fastcgi my $c = shift;
$c->add_attribute({
rpms app cron_hour => 5,
game1:web cron => 'root.tx',
});
};
- “attribute” can be used to
db01 db02
change the behavior of test/run
default rendering
crontab crontab 0 5 * * * /path/to/command
- changing logic, template, etc..
sysctl sysctl
- overwriting of “attribute” If defined attribute on
- bootstrap
rpms rpms the Touryo Config ...
- role config Touryo Config
mysql mysql role: game1:web
- defined for the role
game1: backup attribute:
-db:masterconfig
node cron_hour: 9
game1:
- defined for the node(host)
db:backup 0 9 * * * /path/to/command

29. How to define the “correct” state? (2)
Touryo config
web01
role: game1
crontab apache
blueprint: attribute:
sysctl fastcgi - crontab somaxconn: 1024
- sysctl
rpms app
- rpms
game1:web cron_hour: 9
somaxconn: 10240 role: game1:web
fcgi_proc: 100 blueprint: attribute:
- apache cron_hour: 9
- fastcgi somaxconn: 10240
web02 - app fcgi_proc: 100
crontab apache
sysctl fastcgi node: web02
rpms app blueprint: attribute:
admin: - git fcgi_proc: 10
game1:web git touryo
cron_hour: 9 - You can vary correct “blueprint” and
somaxconn: 10240 “attribute” by Touryo config
fcgi_proc: 10
- role config and node config

30. Lightweight
/Testable/Freedom
find lib/ -name "*.pm" | xargs cat | wc -l
about 1500 (including html templates)
Like test scripts of Perl Software, you can
test the servers’ configuration
you can write anything on blueprint.pl
Conversely you have to write anything

31. Other Features
in terms of Programming
use Amon2
easy to develop Web Interface and CLI
use TAP::Parser to get the test result
test function uses Test::More
use IPC::Cmd and “ssh” / “sudo ssh”
easy to configure with .ssh/config or others
blueprint dependance resolving
easy to cascade the config

32. What language
should I use?

33. There are many
languages
script / compile
Perl,Ruby,Python,PHP / C,C++,Java
procedural / OO / functional
Perl,C / Java,Ruby / Scala,Haskell
popular / minority
???(religious problem...)

34. My Thought
You can use any languages you like
Engineer have to treat multi languages
Development speed / Maintenance cost
They are usually trade-off
Popular LLs are often suited for both
Simple / Light / Thin / Useful Modules
Everyone can understand it easily
Perl/Ruby/Python are similar

35. I ♡ Perl
Fastest to develop for me
I am used to Perl because DeNA use Perl
There are many Perl Mongers around :)
colleague and friends
CPAN has many useful modules
long history and super hackers

36. Finally...

37. DeNA ♡ Engineers
We need more powerful engineers
We use Perl and other languages
Ruby/JavaScript/C/C++/Java/Objective-C
SocialGame/Platform/SDK/Data mining
Or let’s make very useful tools with me :)
Server Ops/Database Admin/Network
If you are interested in us, pleeeeeeeeease
contact me or DeNA staffs!
hirose31, gfx, hidek, zigorou, nekokak...

38. Thanks!
http://blog.riywo.com
http://twitter.com/riywo
Thanks for JPA, @kazeburo, @hirose31, @Yappo, @Percona,
and All Perl Mongers!