1. Inside Sqale’s Backend
RubyConf Taiwan 2012
Gosuke Miyashita
paperboy&co., Inc.，Japan

2. cpan:mizzy
github.com/mizzy
mizzy.org
@gosukenator

5. Kentaro Kuribayashi
Living on the Edge Rails

6. Kensuke Nagae
Introducing nonopaste-cli

7. Shinya Tsunematsu
Building production server
environment of ruby in modern way

8. Hiroshi SHIBATA
How to discover the Ruby's defects
with web application

9. Gosuke Miyashita
Inside Sqale’s Backend

10. We are from
paperboy&co., Japan

12. What is Sqale?

13. Cloud Application
Platform like Heroku

15. Architecture Overview

16. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

17. Containers

18. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

19. Virtual Environments
Assigned To Users

20. Similar to Dynos of
Heroku

21. Containers made by
LXC (Linux Containers)

22. EC2 Instance (1 Virtual Machine)
Container Container Container Container Container
for for for for for
user A user A user B user B user B
Container Container Container Container Container
for for for for for
user C user D user D user E user E
Container Container Container Container Container
for for for for for
user E user F user F user F user F

23. Nginx
Unicorn
sshd
supervisrod
on each container

24. Amazon Linux
+
Patched kernel(3.2.16)

25. grsecurity kernel patch
for various restrictions

26. original kernel patches
to restrict tcp port
bind and fork bomb

29. Anti fork bomb patch
makes some changes to
cgroup and fork process

30. See
paperboy-sqale/sqale-patches
on GitHub

31. Web Proxy

32. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

33. HTTP/HTTPS
ELB
nginx nginx
Container Container Container Container Container Container
for for for for for for
user A user B user B user C user C user C

34. nginx
lua-nginx-module
redis2-nginx-module

35. http://www.i4pc.jp/
Which containers?
Redis nginx
host001:8083, host001:8084
or
host001
nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084
Container Container Container Container
for for for for
lokka-mizzy lokka-mizzy i4pc-mizzy i4pc-mizzy

36. nginx.conf (excerpt)
location / {
set $container "";
set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua;
proxy_pass http://$container;
}

37. dynamic-proxy.lua (excerpt)
local reply = ngx.location.capture("/redis")
if reply.status ~= ngx.HTTP_OK then
ngx.exit(503)
end
local containers, type =
parser.parse_reply(reply.body)

38. dynamic-proxy.lua (excerpt)
while #containers > 0 do
tmp = table.remove(
containers,
math.random(#containers))
if ngx.shared.downed_containers:get(tmp) then
ngx.log(ngx.DEBUG, tmp .. " is down")
else
container = tmp
break
end
end

39. dynamic-proxy.lua (excerpt)
ngx.var.container = container
ngx.var.next_containers
= luabins.save(containers)

40. nginx.conf (again)
location / {
set $container "";
set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua;
proxy_pass http://$container;
}

41. nginx.conf (excerpt)
location @failover {
error_page 502 = @failover;
rewrite_by_lua_file failover.lua;
proxy_pass http://$container;
}

42. failover.lua (excerpt)
local downed_container = ngx.var.container
if downed_container then
ngx.shared.downed_containers:set(
downed_container,
1,
sqale.NEGATIVE_CACHE_SECONDS
)
end

43. failover.lua (excerpt)
while #containers > 0 do
tmp = table.remove(
containers,
math.random(#containers))
if ngx.shared.downed_containers:get(tmp) then
ngx.log(ngx.DEBUG, tmp .. " is down")
else
container = tmp
break
end
end

44. failover.lua (excerpt)
if not container then
ngx.exit(503)
end
ngx.var.container = container
ngx.var.next_containers
= luabins.save(containers)

45. nginx.conf (agin)
location @failover {
error_page 502 = @failover;
rewrite_by_lua_file failover.lua;
proxy_pass http://$container;
}

46. SSH Router

47. SFTP
Git over SSH HTTP/HTTPS
AWS SSH
Web Proxy
SSH Router
to Containers
File Deploy
Containers
Repositories Servers

48. Git SFTP SSH Login
SSH Router
File File
Repositories Repositories Containers
(Git Server) (File Server)

49. How implement this
routing?

50. OpenSSH with script
authentication patch

52. See
mizzy/openssh-script-auth
on GitHub

53. Change routes by
SSH_ORIGNAL_COMMAND

55. In case of
SSH_ORIGINAL_COMMAND
is “git-*”

56. git push
(ssh sqale@gateway.sqale.jp git-recieve-pack
‘/mizzy/lokka.git’)
Run AuthorizedKeys
Script
SSH Router MySQL
Verify the public key
and get the user’s git
server
command=“ssh sqale@git001.sqale.lan
git-recieve-pack
File ‘/var/repos/mizzy/lokka.git’”
Repository
(Git Server)

57. In case of
SSH_ORIGINAL_COMMAND
is “sftp-server”

58. sftp sqale@gateway.sqale.jp
(ssh sqale@gateway.sqale.jp sftp-server)
Run AuthorizedKeys
Script
SSH Router MySQL
Verify the public key
and get the user’s file
server
command=“ssh sqale@file001.sqale.lan
sftp-server”
File git push File
Repository Repository
(File Server) (Git Server)

59. In case of
SSH_ORIGINAL_COMMAND
is empty

60. ssh sqale@gateway.sqale.jp
Run AuthorizedKeys
Script
SSH Router MySQL
Verify the public key
and get the user’s
cotainers list
Display the user’s containers list and
wait the user’s selection
command=“ssh sqale@
Container users001.sqale.lan -p 8081”

63. Ruby On Rails
Puppet/Chef
Unicorn
Resque
And many gems

64. 多謝