4. About me
! Work at
! Programmer
!
as a ...
(Most recently)
! Also Interested in
! Software Engineer
! Cloud Computing
! Big Data
! Something new technologies
! Supporting GlusterFS/Red Hat Storage Introduction
with Red Hat K.K.
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
4
5. About me
! GlusterFS since 2007 (v1.3.7)
! for my internet crawler at first.
! Love Gluster because of the ...
! Potential
! Performance
! Code
! Community
! Introduced or introducing it into ...
! Printer and scanner solution (field trial)
! Email services
! File storage services (WebDAV, NFS)
! Backup services
! Shared storage platform
! A board member of the Gluster
Community
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
5
6. Getting started today!
See http://www.gluster.org/community/documentation/index.php/QuickStart
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
6
8. Tips 1: Installation
! Question 1: Have you installed GlusterFS?
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
8
9. Tips 1: Installation
! Question 1: Have you installed GlusterFS?
! Question 2: Did you find it easy?
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
9
10. Tips 1: Installation
! Question 1: Have you installed GlusterFS?
! Question 2: Did you find it easy?
! Question 3: Did you do it on each single node?
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
10
11. Tips 1: Installation
!
!
!
!
Question 1: Have you installed GlusterFS?
Question 2: Did you find it easy?
Question 3: Did you do it on each single node?
Question 4: Don't you think you should take easier way?
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
11
12. Tips 1: Installation
!
!
!
!
!
Question 1: Have you installed GlusterFS?
Question 2: Did you find it easy?
Question 3: Did you do it on each single node?
Question 4: Don't you think you should take easier way?
Question 5: Do you use deployment automation tools?
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
12
13. Tips 1: Installation
Development team says that hand-to-hand construction of systems is inefficient so they want to
show an installation way with a deployment tool in the documents for installation.
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
13
14. Tips 1: Installation
Operational team says that they never want to learn the usage of deployment tools and never
trust them so they don't accept such a document and they will install the system by hand work.
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
14
15. Tips 1: Installation
My offer is that what development team should do is only copy the commands that the
deployment tool write on your stdout and paste them into the documents for operational team.
https://twitter.com/keithseahus/status/398453753352691714
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
15
23. Tips 1: Installation
I use this mainly in my projects
written in Ruby so am used to it.
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
23
28. capistrano-glusterfs
$ tree capistrano-glusterfs
capistrano-glusterfs
├── Capfile
├── Gemfile
├── Gemfile.lock
├── LICENSE
├── README.md
├── config
│ ├── deploy
│ │ ├── production.rb ... settings for production environment
│ │ ├── production.rb.example
│ │ ├── staging.rb ... settings for staging environment
│ │ └── staging.rb.example
│ ├── deploy.rb ... global settings and main sequences
│ └── deploy.rb.example
└── lib
└── capistrano
└── tasks
├── gluster.cap ... tasks to execute gluster command remotely
├── glusterd.cap ... tasks to start/stop all remote glusterd
└── glusterfs.cap ... tasks for deploying GlusterFS
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
28
29. capistrano-glusterfs
$ cat capistrano-glusterfs/config/deploy/staging.rb
set :stage, :staging
set :branch, "master"
server
server
server
server
server
server
server
server
server
server
'192.168.200.100',
'192.168.200.101',
'192.168.200.102',
'192.168.200.103',
'192.168.200.104',
'192.168.200.105',
'192.168.200.106',
'192.168.200.107',
'192.168.200.108',
'192.168.200.109',
roles:
roles:
roles:
roles:
roles:
roles:
roles:
roles:
roles:
roles:
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
glusterd gluster), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
29
30. capistrano-glusterfs
$ bundle exec cap -vT | grep -i "glusterfs:"
cap glusterfs:build
# Starting installation
cap glusterfs:compile
# Compile GlusterFS
cap glusterfs:configuration
# GlusterFS related configurations
cap glusterfs:dependancies
# Install GlusterFS dependancies
cap glusterfs:make_install
# Install GlusterFS
cap glusterfs:preparation
# Prepare installing GlusterFS
$ bundle exec cap -vT | grep -i "glusterd:"
cap glusterd:restart
# Restart glusterd
cap glusterd:start
# Start glusterd
cap glusterd:status
# Get the status of glusterd
cap glusterd:stop
# Stop glusterd
$ bundle exec cap -vT | grep -i "gluster:"
cap gluster:peer:help
# Help command for peer
cap gluster:peer:status
# list status of peers
cap gluster:pool:list
# list all the nodes in the pool
cap gluster:volume:help
# display help for the volume command
cap gluster:volume:info
# list information of all volumes
cap gluster:volume:list
# list all volumes in cluster
cap gluster:volume:status
# display status of all volume(s)/brick
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
30
32. capistrano-glusterfs
prepare and install
dependancies
before starting
build GlusterFS from
source and install it
after published
http://www.capistranorb.com/documentation/getting-started/flow/
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
32
34. capistrano-glusterfs
task :dependancies do
....
dependancies = %w[
openssh-clients
make
rpm-build
bison
flex
automake
libtool
ncurses-devel
readline-devel
openssl-devel
libxml2-devel
libibverbs-devel
libacl-devel
libattr-devel
python-devel
python-setuptools
lvm2-devel
systemtap-sdt-devel
libaio-devel
xfsprogs
]
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
34
35. capistrano-glusterfs
task :compile do
on roles(:glusterfs), in: :sequence, wait: 5 do
execute "cd #{DIR}; sh ./autogen.sh"
execute "cd #{DIR}; sh ./configure
--prefix=#{INSTALL_PATH}
--enable-bd-xlator
--enable-fusermount
--enable-systemtap
--enable-debug"
execute "cd #{DIR}; make"
end
end
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
35
36. capistrano-glusterfs
task :make_install do
on roles(:glusterfs), in: :sequence, wait: 5 do
execute "cd #{DIR}; sudo make install"
symlink = "/usr/local/glusterfs"
execute "sudo -i rm -f #{symlink}" if test("[ -L #{symlink} ]")
execute "sudo -i ln -snf #{INSTALL_PATH} #{symlink}"
end
end
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
36
37. capistrano-glusterfs
$ git clone git@github.com:keithseahus/capistrano-glusterfs.git
$ cd capistrano-glusterfs
$ bundle exec cap staging deploy
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
37
40. capistrano-glusterfs
$ bundle exec cap staging gluster:pool:list
INFO [201946fe] Running /usr/bin/env sudo -i gluster pool list on
192.168.200.101
DEBUG [201946fe] Command: sudo -i gluster pool list
DEBUG [201946fe] UUID
Hostname State
DEBUG [201946fe] d185fa19-d1a1-489c-a653-4510122f5278 localhostConnected
INFO [201946fe] Finished in 0.294 seconds with exit status 0 (successful).
$ bundle exec cap staging glusterd:status
INFO [cb482772] Running /usr/bin/env sudo -i /sbin/service glusterd status on
192.168.200.101
DEBUG [cb482772] Command: sudo -i /sbin/service glusterd status
INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful).
DEBUG [cb482772] glusterd (pid 25855) is running...
INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful).
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
40
41. capistrano-glusterfs
$ bundle exec cap staging gluster:volume:help
INFO [4628fd38] Running /usr/bin/env sudo -i gluster volume help on
192.168.200.101
DEBUG [4628fd38] Command: sudo -i gluster volume help
DEBUG [4628fd38] volume info [all|<VOLNAME>] - list information of all volumes
DEBUG [4628fd38] volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>]
[device vg] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> ... [force] - create a
new volume of specified type with mentioned bricks
DEBUG [4628fd38] volume delete <VOLNAME> - delete volume specified by <VOLNAME>
DEBUG [4628fd38] volume start <VOLNAME> [force] - start volume specified by
<VOLNAME>
DEBUG [4628fd38] volume stop <VOLNAME> [force] - stop volume specified by
<VOLNAME>
DEBUG [4628fd38] volume add-brick <VOLNAME> [<stripe|replica> <COUNT>] <NEWBRICK> ... [force] - add brick to volume <VOLNAME>
DEBUG [4628fd38] volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ...
{start|stop|status|commit|force} - remove brick from volume <VOLNAME>
DEBUG [4628fd38] volume rebalance <VOLNAME> [fix-layout] {start|stop|status}
[force] - rebalance operations
DEBUG [4628fd38] volume replace-brick <VOLNAME> <BRICK> <NEW-BRICK> {start
[force]|pause|abort|status|commit [force]} - replace-brick operations
DEBUG [4628fd38] volume set <VOLNAME> <KEY> <VALUE> - set options for volume
<VOLNAME>
...
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
41
44. Tips 2: NFS Workloads
! Question 1: Do you mount GlusterFS in NFS?
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
44
45. Tips 2: NFS Workloads
! Question 1: Do you mount GlusterFS in NFS?
! Question 2: Which is your main workload, write or read?
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
45
46. Tips 2: NFS Workloads
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
46
47. Tips 2: NFS Workloads
TCP connection
in gluster protocol
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
47
48. Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs
page
cache
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
48
49. Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs
page
cache
TCP connection
in NFS v3
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
49
50. Tips 2: NFS Workloads
NFS Client
read access
to a file
user app
/mnt/nfs
vfs
page
cache
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
50
51. Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs
page
cache
kernel
performance/io-cache
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
51
52. Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs
activate cachefilesd
/var/cache/fscache
page
cache
cachefilesd
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
52
53. Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs
read files from its own RAM!
/var/cache/fscache
page
cache
cachefilesd
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
53
54. Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
54
55. Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
55
56. Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd
# if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne
0; then
sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf
fi
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
56
57. Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd
# if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne
0; then
sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf
fi
# chkconfig cachefilesd on
# service cachefilesd start
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
57
58. Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd
# if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne
0; then
sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf
fi
# chkconfig cachefilesd on
# service cachefilesd start
# mount -t nfs -o fsc,vers=3 $NFS_SERVER $MOUNT_POINT
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
58
60. Tips 2: NFS Workloads
nodes:
VirtualBox
CentOS 6.4 x86_64
4 vCPUs, Core i7 2.4GHz
2GB RAM
Striped SSDs
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
60
61. Tips 2: NFS Workloads
nodes:
VirtualBox
CentOS 6.4 x86_64
4 vCPUs, Core i7 2.4GHz
2GB RAM
Striped SSDs
postmark 1.51
set size 1048576 10485760
set number 500
set transactions 1000
set location /mnt/nfs/rep2
set subdirectories 100
set read 8192
set write 8192
set buffering true
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
61
62. Tips 2: NFS Workloads
nodes:
VirtualBox
CentOS 6.4 x86_64
4 vCPUs, Core i7 2.4GHz
2GB RAM
Striped SSDs
MB/sec (read)
postmark 1.51
set size 1048576 10485760
set number 500
set transactions 1000
set location /mnt/nfs/rep2
set subdirectories 100
set read 8192
set write 8192
set buffering true
↓Oops...
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
62
63. Tips 2: NFS Workloads
Okay... but this is still a hypothesis.
We need your feedback!
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
63
65. Tips 3: Logging
! Question 1: Do you monitor GlusterFS logs?
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
65
66. Tips 3: Logging
! Question 1: Do you monitor GlusterFS logs?
! Question 2: Do you utilize the logs?
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
66
67. Tips 3: Logging
! Question 1: Do you monitor GlusterFS logs?
! Question 2: Do you utilize the logs?
! Question 3: Do you use a log collector or aggregator?
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
67
69. Tips 3: Logging
I pick it up because:
• easy to install
• easy to configure
• make unstructured logs structured
• collect logs from multiple resources
• output aggregated and structured
logs into a target as you prefer
• easy to create a module
• high performance
• well maintained
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
69
70. Tips 3: Logging
NFS Client
user app
/mnt/nfs
vfs
/var/cache/fscache
page
cache
cachefilesd
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
70
71. Tips 3: Logging
NFS Client
Log server
user app
/mnt/nfs
vfs
/var/cache/fscache
page
cache
cachefilesd
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
71
72. Tips 3: Logging
NFS Client
Log server
user app
/mnt/nfs
vfs
/var/cache/fscache
page
cache
cachefilesd
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
72
73. Tips 3: Logging
NFS Client
Log server
user app
/mnt/nfs
vfs
/var/cache/fscache
page
cache
cachefilesd
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
73
74. Tips 3: Logging
NFS Client
Log server
user app
/mnt/nfs
vfs
/var/cache/fscache
page
cache
cachefilesd
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
74
75. Tips 3: Logging
NFS Client
Log server
user app
/mnt/nfs
vfs
/var/cache/fscache
page
cache
cachefilesd
kernel
peer 1
glusterd
glustershd
peer 2
glusterfs
d
shd
3
4
N
fs
...
glusterfsd
brick
brick
brick
fsd
brick
b
b
b
b
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
75
76. Tips 3: Logging
/etc/td-agent/td-agent.conf (on the log server)
<source>
type forward
port 24224
bind 0.0.0.0
</source>
<match filtered.glusterd.**>
type file
path /home/keith/var/log/fluent/glusterd
</match>
Today I will get the td-agent
output the logs into a file.
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
76
77. Tips 3: Logging
/etc/td-agent/td-agent.conf (on the GlusterFS peers)
<source>
type tail
path /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log
pos_file /var/log/td-agent/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log.pos
tag glusterd
format /^(?:[(?<date_ymd>[0-9]{4}-[01][0-9]-[0-3][0-9]) (?<time_hms>[0-2][0-9]:[0-5][0-9]:[0-6]
[0-9]).(?<time_usec>[0-9]{6})]) (?<log_level>[TDINWECA]) (?:[(?<source_file_name>[^[]:/ ]*):(?
<source_line>[0-9]*):(?<function_name>[^[]:/ ]*)]) (?<component>[^[]:/ ]*): *(?<message>.*)$/
rotate_wait 5
</source>
<match glusterd.**>
type hostname
key_name host
add_prefix filtered
</match>
Too complex to input
and maintain!
<match filtered.glusterd.**>
type forward
send_timeout 60s
recover_wait 10s
heartbeat_interval 1s
phi_threshold 8
hard_timeout 60s
<server>
…
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
77
79. Tips 3: Logging
You can specify log levels
to forward.
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
79
80. Tips 3: Logging
You can specify each field
name to output in JSON.
Comment out if you don't
need the fields.
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
80
81. Tips 3: Logging
Parse error should be
treated as a failure of
fluent-plugin-glusterfs.
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
81
82. Tips 3: Logging
It makes the logs structured in JSON.
[root@glusterfs-unstable-01 plugin]# tail -3 /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etcglusterfs-glusterd.vol.log
[2013-07-07 15:52:43.778961] I [socket.c:3495:socket_init] 0-management: using system polling thread
[2013-07-07 15:52:43.779397] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data
available)
[2013-07-07 15:52:43.779668] I [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
[root@dev-centos ~]# grep glusterfs-unstable-01 /home/keith/var/log/fluent/glusterd.
20130708.b4e0ed6e32655d35f | tail -3
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"778961","log_level":"I","source_file_name":
"socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-01.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779397","log_level":"W","source_file_name":
"socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-01.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779668","log_level":"I","source_file_name":
"socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-01.localdomain"}
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
82
83. Tips 3: Logging
The logs on each peer could be output in a single log file.
[root@dev-centos ~]# grep glusterfs-unstable-01 /home/keith/var/log/fluent/glusterd.
20130708.b4e0ed6e32655d35f | tail -3
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"778961","log_level":"I","source_file_name":
"socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-01.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779397","log_level":"W","source_file_name":
"socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-01.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779668","log_level":"I","source_file_name":
"socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-01.localdomain"}
[root@dev-centos ~]# grep glusterfs-unstable-02 /home/keith/var/log/fluent/glusterd.
20130708.b4e0ed6e32655d35f | tail -3
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"759502","log_level":"I","source_file_name":
"socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-02.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"759838","log_level":"W","source_file_name":
"socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-02.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"760057","log_level":"I","source_file_name":
"socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-02.localdomain"}
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
83
84. Tips 3: Logging
Also available on Gluster Forge:
https://forge.gluster.org/fluent-plugin-glusterfs
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
84
85. Tips 3: Logging
More detailed explanation in Japanese is here!:
http://www.nttpc.co.jp/company/r_and_d/technology/gluster_fs_fluentd.html
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
85
87. Conclusion
! Deployment automation tools are
useful to install and operate
GlusterFS on multiple nodes.
! With cachefilesd, NFS can might
be more efficient under heavy
read workload.
! Aggregating GlusterFS logs is easy
enough and you can utilize them.
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved.
87
88. To contact us, e-mail here -> storage-contact@nttpc.co.jp
Copyright (C) 2013, NTTPC Communications, Inc. All Rights Reserved. 88