This document discusses using Gluster object storage with OpenStack Swift. Gluster-Swift mounts the Swift storage using FUSE and allows Swift to interface with Gluster backends. This avoids reimplementing the Swift object API. Gluster-Swift overrides Swift's distribution and replication to use the Gluster backend. The Swift API is implemented using FUSE operations on the Gluster volume. Future work includes upgrading Gluster-Swift, packaging, optimizations, and potentially developing a native Gluster object interface.

1. Object storage with Gluster
Prashanth Pai
ppai@redhat.com

2. gluster-swift
➔ Just another application using fuse mount
➔ No libgfapi, yet
➔ Uses and “plugs into” OpenStack Swift
➔ Against having to write our own object front-end
➔ No catching-up to do
➔ Swift is largely decoupled from OpenStack
➔ Overrides vanilla Swift’s distribution and replication

3. Object access trivia
➔ Object = {Data + Metadata + Identifier}
➔ Abstracted
➔ Stateless (HTTP)
➔ No partial PUTs
➔ Ranged GETs supported
➔ No renames, only COPY
➔ Fixed logical hierarchy
➔ Accounts/Container/Objects
➔ But you can’t mount it

4. Vanilla Swift
Ring
SQLite DB
SQLite DB
XFS
Account
Server
Container
Server
Object
Server
Proxy
Server
Client

5. Gluster-Swift
Ring
Account
Server
Container
Server
Object
Server
Proxy
Server
Client
Same node
FUSE
mount

6. Proxy Server - pipeline
/etc/swift/proxy-server.conf
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache
container_sync bulk tempurl ratelimit tempauth copy
container-quotas account-quotas slo dlo versioned_writes
proxy-logging proxy-server
➔ Very similar to translators in gluster
➔ Middlewares are highly modular
➔ Many features are implemented as middlewares

7. Amazon S3 API Support
➔ Implemented as a middleware (swift3)
➔ Converts incoming S3 requests into Swift requests
S3 Request
Swift Response
Swift3
middleware
Swift Request
S3 Response
Proxy
Server
S3 Client

8. Pluggability in Swift
• Backends implement DiskFile
• Object server is pluggable officially.
• Account and container code overrides aren’t.
• Makes cross-version compatibility tricky.

9. Swift Object Ops → Fops
PUT
mkdir()*
getxattr()
open()
write()
fsetxattr()
fsync()
rename()
close()
GET
open()
fstat()
fgetxattr()
read()
close()
POST
getxattr()
stat()
setxattr()
DELETE
getxattr()
stat()
unlink()
rmdir()*
*multiple calls - recursive
HEAD
getxattr()
stat()

10. Swift Listing Ops → Fops
Container GET
(plain text) – only names
getxattr()
opendir()
readdir()*
getxattr()1
Container GET
(json/xml) – names + metadata
getxattr()
opendir()
getdents()*
getxattr()2
*multiple calls – recursive
1
only on dirs
2
on every inode

11. # getfattr -d /mnt/gluster-object/test/c1/animals/cat/kitten
user.swift.metadata="{
"Content-Length":"4",
"Etag":"4a4be40c96ac6314e91d93f38043a634",
"X-Timestamp":"1475214280.33138",
"X-Object-Type":"file",
"X-Type":"Object",
"Content-Type":"application/x-www-form-urlencoded"}"
# curl -i http://host:8080/v1/AUTH_test/c1/animals/cat/kitten -X PUT -d'meow'
HTTP/1.1 201 Created
Last-Modified: Fri, 30 Sep 2016 05:44:41 GMT
Content-Length: 0
Etag: 4a4be40c96ac6314e91d93f38043a634
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txae858eb9169a4e1eb2a09-0057edfbc8
Date: Fri, 30 Sep 2016 05:44:40 GMT
Object PUT

12. Swift API Violations
# curl -i http://localhost:8080/v1/AUTH_test/c2 -X GET
HTTP/1.1 200 OK
Content-Length: 7
X-Container-Object-Count: 2
Accept-Ranges: bytes
X-Storage-Policy: Policy-0
X-Container-Bytes-Used: 0
X-Timestamp: 0000000001.00000
Content-Type: text/plain; charset=utf-8
X-Trans-Id: tx18e21ccd03f04f4698a1a-0057ed1b84
Date: Thu, 29 Sep 2016 13:47:48 GMT
HTTP Header needs to go first, then the response body.

13. Swift API Violations
# curl http://host:8080/v1/AUTH_test/c1/animals/cat -X PUT
Now, cat is a file
# curl http://host:8080/v1/AUTH_test/c1/animals/cat/kitten -X PUT
Here, cat is supposed to be directory but it’s already a file.
Backend object path on filesystem
gluster-swift:
/mnt/gluster-object/test/c1/animals/cat
Vanilla Swift:
/mnt/sdb1/objects/778/69f/c2b307d78b6c419c0c1b76d91c08c69f/1412628708.01757.data

14. TODOs
➔ Bring up gluster-swift to ‘newton’
➔ Centos SIG packages
➔ Zero copy and copy offload
➔ Use libgfapi python bindings
➔ Use hummingbird project
➔ Native object ? glfs_put(..), glfs_get(..)
Ideas

15. Demo ?
gluster-swift