Gluster Developer Summit 2016

1. Metadata Performance
Bottlenecks in Gluster
Poornima G
Rajesh joseph

2. Agenda
 Existing performance xlators
 Metadata operations
 create files
 mkdir recursive
 ls -lR
 rm files
 rmdir recursive
 mv files
 Small file create / PUT
 Small file read / GET

3. Performance Xlators

Write-behind and Flush behind

Io-cache

Read-ahead

Quick-read

Md-cache

Readdir-ahead

Open-behind

Symlink cache

4. Test Setup and Tools

Centos 7.2 clients and server

3 FUSE Clients simultaneously performing
operation on their home directories

Latest master, with below volume options:
cluster.lookup-optimize on
readdir-optimize on
consistent-metadata no

Performance matrics collected using volume
profile

Workload is some simple scripts.

5. #Create Files
13
12
11
1025
10
18
1
STAT/LOOKUP - 13% (Negative Lookups ~ 12%) CREATE - 12%
REMOVEXATTR - 13% SETATTR - 8%
INODELK - 25% (Removexattr + setattr) ENTRYLK - 10%
FLUSH - 10% READDIRP - 1%

6. Create Files

Negative lookups – 11-12%

Removexattr (security.ima) – 23%

Lease on parent dir and newly created files can
eliminate, sending inodelk(25%) and
entrylk(10%) to the bricks. But only benefits
single client use cases.

Create is not a function of number of bricks.

7. #mkdir recursive
17
10
9
43
21
STAT/LOOKUP - 13% (Negative lookups ~ 12%) MKDIR - 10% SETXATTR - 9%
INODELK - 43% (SETXATTR and MKDIR) ENTRYLK - 21%

8. mkdir recursive

Negative lookup ~10% and cache miss

In DHT, Setxattr can be compounded with mkdir?
saves inodelks and setxattr

Lease on parent dir and newly created files can
eliminate, sending inodelk and entrylk to the
bricks. But only benefits single client use cases.

It is not a function of number of dht subvol, as
mkdir is sent initially on hashed subvol followed
by parallel mkdir on the rest.

9. #ls -lR
34
19
47
STAT/LOOKUP - 34% (~20% on dir ~10% cache miss) OPENDIR - 19% READDIRP - 47%

10. ls -lR

Readdirp - sequential and fn(number of dht
subvols). Hence not scalable.
Eg: reading 800 empty directories on
10*1 = 16000 sequential readdirp calls
30*1 = 48000 sequential readdirp calls
Readdir-ahead only masks part of this latency,
hence parallel readdirp in DHT.

Eliminate EOD detection readdirp call

DHT – do not NULL the readdirp entries of
directory

Md-cache – eliminate revalidate lookups

11. #rm files in one large dir
2
36
18
44
STAT/LOOKUP - 2% ENTRYLK - 36% UNLINK - 19% READDIRP - 44%

12. Delete files

Parallel readdirp – may not improve performance
drastically for large directories.

Lease on parent directory, can make entrylk only
client side. But helps only in single client use
case.

13. #rmdir small directories recursively
12
9
55
3
5
18
STAT/LOOKUP - 12% OPENDIR - 9% INODELK - 55% (rmdir sequential inodelk)
RMDIR - 3% ENTRYLK - 5% READDIRP - 18%

14. rmdir Recusrsive

Rmdir - fn (number of dht subvolumes). Hence
not scalable.

Inodelks are taken sequentially on all dht subvols
before rmdir (rmdir and inodelk unlock –
parallel). WIP to remove inodelk.

Lease on parent directory, can make inodelk and
entrylk only client side. But helps only in single
client use case.

Revalidate lookup – md-cache

15. Small file create
20
21
101
50
STAT/LOOKUP - 20% (Negative LOOKUP - 15-16%) ENTRYLK - 21%
CREATE - 10% WRITE - 1%
FLUSH - 50% (FINODELK + FXATTROP + WRITE + FLUSH)

16. Small file Create / PUT

ENTRYLK, INODELK can be made only client side
by using leases, but helps single client use case

Compound fop XATTROP + WRITE

PUT fop, only gfapi clients(Swift, core utils) will
be able to leverage.

17. Small file Read
95.6
2.51.40.5
STAT/LOOKUP - 95% OPEN - 2.5% READ - 1.2% FLUSH - 0.5%

18. Conclusion
We have not yet saturated, the perf
improvements. Performance of these
metadata operations can be improved
greatly, without compromising on
consistency.