1. Introduction to OpenAFS
Fabrizio Manfred Furuholmen
Beolink.org

2. Agenda Beolink.org
 Introduction
 Architecture
 Client
 Administration tasks
 Setup your Cell
2
16/02/2012

3. Introduction Beolink.org
What is a Distributed File system ?
“A distributed file system takes advantage of the
interconnected nature of the network by storing
files on more than one computer in the network
and making them accessible to all of them..”
3
16/02/2012

4. Introduction Beolink.org
Andrew File System
Andrew File System is a distributed file system
designed to:
 handle terabytes of data
 handle thousands of users
 working in WAN environment
4

5. Introduction Beolink.org
AFS is25yearsold !
5

6. Introduction Beolink.org
Briefhistoryof a AFS
 1983 Andrew Project started at Carnegie Mellon University (CMU)
 1987 Coda research work begun (based on AFS)
 1988 First use of AFS version 3 (First use of AFS outside CMU)
 1988 Institutional File System project at University of Michigan
 1989 Transarc Corporation founded to commercialize AFS
 1993 Arla project started at KungligaTekniskaHögskolan
 1998 Transarc Corporation becomes wholly owned subsidiary of IBM
 2000 IBM releases OpenAFS as OpenSource (IBM License)
 2000 OpenAFS release version 1.0 based on Transarc 3.6
 2001 OpenAFS release version 1.2 first release with better support of new
operating system and fix several memory leak
 2005 OpenAFS release version 1.4 with a lot of new feature
 2005 AFS was discontinued from IBM
 2008 U.S. Department of Energy Funds OpenAFS Development
 2010 OpenAFS release version 1.6 (?)
6

7. Benefits of using AFS Beolink.org
 Location independence
User does not need to know which fileserver holds the file, the user only
needs to know the pathname of a file.
 Scalability
An architectural goal of the AFS designers was client/server ratios of
114.000:1 A ratio of 2000:1 has been successfully exceeded at some sites.
 Security
AFS makes use of Kerberos for mutual authentication, both the service
provider and the requester prove their identities
AFS uses access control list (ACLs) to enable users to restrict access to their
own directories, users can also create groups
AFS Federation with inter cell grant
 Uniform Namespace
No matter where users are logged in, they see the same files
 Replicates AFS Volumes
Frequently accessed data can be read-only replicated on several servers (rw
with osd version). Client will access the closest volume copy or load balance
from a different replica
7

8. Benefits of using AFS Beolink.org
 Improved robustness to server crash
Clients maintain Local copies of accessed files, replicated read-only volumes
on alternate fileservers can satisfy requests for a files
 Wide Area Network
AFS communications protocols is optimized for WAN. Retransmitting only the
single bad packet in a batch of packets (RPC)
 Improve system management capability
Configuration changes can be made from any client in the AFS cell
AFS volumes can move from one server to another without users noticing it
 Operating system independent
AFS client software runs on many systems (12 platforms)
8

9. Elements Beolink.org
Cell
•Cell is collection of file servers and
workstation
•The directories under /afs are
cells, unique tree
•Fileserver contains volumes
Volumes
•Volumes are "containers" or sets of
related files and directories
•Have size limit
•3 type rw, ro, backup
Mount Point Directory
Server A
•Access to a volume is provided through
a mount point Server C
•A mount point is just like a static
directory Server A+B
9

10. Architecture Beolink.org
10
16/02/2012

11. Consistency Beolink.org
“..That notion of callbacks gives OpenAFS a
much stronger consistency guarantee than
most other distributed filesystems.”
Cache Manager
Client-side caching lets clients access data from their local cache without
going across the network for every access.
Callbacks
OpenAFS uses callbacks, which are a promise from the file server to the
client that if the file changes, the server will contact the client to tell the
client to invalidate the cached contents.
11

12. Write operation Beolink.org
Example write operation client side
1. create file rpc
2. write chunks into cache
(interrupted by store_data
RPC)
3. read from cache
4. transfer over network
5. write to /vicepXX
12

13. Write operation Beolink.org
Example write operation server side
1 Create file
2 Check metadata, permission, quota
and return file path
3 write file into /vicepXX
4 Update meta data on server
5 Update db
13

14. Client side Beolink.org
Client
14

15. Installation Beolink.org
Supported clients
AIX 5 and 6 (though 6.3)
 FreeBSD 7, 8 and current
 HP-UX 11.0, 11i v1 and v2
Irix 6.5
 Linux 2.2, 2.4, 2.6 (ia32, ia64, x86_64, ppc, ppc64, arm,
sparc, sparc64, s390, s390x)
MacOS 10.3, 10.4, 10.5, 10.6 (including 64 bit).
OpenBSD 4.4, 4.5, 4.6, 4.7.
 Solaris 2.6, 7, 8, 9, 10, 11 (and OpenSolaris)
 Also Windows ...
15

16. Installation Beolink.org
Configuration
Download and install client package and kernel module
 Configure krb5 if you use it
 Configure AFS Files
ThisCell : the name of your cell
CellServDB : cell list ( of the world)
cacheinfo : cache configuration (dimension and location)
16

17. Authentication Beolink.org
Authentication Kerberos 5
kinit, retrieve a kerberos ticket
aklog, convert the krb5 ticket in afs token
 Authentication Kaserver
klog, retrieve a afs token
 Token operations Ticket cache: FILE:/tmp/krb5cc_0
klist, list tikets Default principal: manfred/admin@FARM.ZEROPIU.COM
Valid starting Expires Service principal
08/16/10 16:03:46 08/17/10 16:03:46
krbtgt/FARM.ZEROPIU.COM@FARM.ZEROPIU.COM
08/16/10 16:03:54 08/17/10 16:03:46 afs/farm.zeropiu.com@FARM.ZEROPIU.COM
 tokens, list afs token
Tokens held by the Cache Manager:
kdestroy, ticket destroy User's (AFS ID 15) tokens for afs@farm.zeropiu.com [Expires Aug 17 16:03]
unlog, token destroy
Don’t Forget
 Credentials expire after some time
 AFS service ticket is in the kernel memory
17

18. Access rights Beolink.org
ACLs are only for directories ! (Files soon)
 ACL inheritance, AFS copies ACL on a parent directory over to a
new subdirectory at the time of creation
 ACL awareness, not many commands are aware of ACLs (copy)
ACL Permission
lookup (l) List contents of directory
insert (i) Add Files or directories
delete (d) Delete entries in directory
administer (a) Manipulate ACL for directory
read (r) Read file content, query file status
write (r) Write file content, change Unix permissions
look (k) Full file advisory lock
18

19. Server side Beolink.org
Servers
19

20. Architecture Beolink.org
20
16/02/2012

21. Process Beolink.org
Server Process Function
bosserver Basic OverSeer Server
fileserver Serves the files
volserver Serves volume data
vlserver Volume location server
ptserver Protection server
buserver Backup server
upserver Update server
upclient Update client
21

22. Architecture Beolink.org
AFS ServersTypes
 Fileserver machine
 file storage
 Database server machine
 File and Volume localization
 Groups administration
 Authentication provider
 Backup database
 Binary distribution
 Master server for afs binary (specific
architecture)
 System control machine
 Time server
 AFS configuration master

23. Commands Beolink.org
VOS
PTS
Administration FS
Commands
BOS
23

24. FS Beolink.org
fs: Commands are:
apropos search by help text
checkservers check local cell's servers
fs command
checkvolumes check volumeID/name mappings
cleanacl clean up access control list
copyacl
diskfree
copy access control list
show server disk space usage
 Cache management administration
examine
exportafs
display file/volume status
enable/disable translators to AFS
 Quota management
flush
flushmount
flush file from cache
flush mount symlink from cache
ACLs management
flushvolume flush all data in volume
getcacheparms get cache usage info
 Mount management on the AFS path
getcalleraccess list callers access
getcellstatus get cell status
getclientaddrs get client network interface addresses
getcrypt get cache manager encryption flag
getfid get fid for file(s) setcachesize set cache size
getserverprefs get server ranks setcbaddr configure callback connection address
help get help on commands
listacl list access control list
setcell set cell status
listaliases list configured cell aliases setclientaddrs set client network interface addresses
listcells list configured cells setcrypt set cache manager encryption flag
listquota list volume quota setquota set volume quota
lsmount list mount point setserverprefs set server ranks
messages control Cache Manager messages
mkmount make mount point
setvol set volume status
newalias configure new cell alias storebehind store to server after file close
newcell configure new cell sysname get/set sysname (i.e. @sys) value
quota show volume quota usage uuid manage the UUID for the cache manager
rmmount remove mount point whereis list file's location
rxstatpeer Manage per peer RX statistics
rxstatproc Manage per process RX statistics
whichcell list file's cell
setacl set access control list wscell list workstation's cell
setcachesize set cache size
24

25. Administration Beolink.org
BOS Command
 Process creation
 Process administration (start, stop, status ...)
 Manage Users Administrator for process
 Volume check
25

26. Administration Beolink.org
VOS Command
 Create volume
 Volume Replication
 Volume Information
 Move volume
DON’T FORGET THERE ARE DIFFERENT PATH FOR RW
and R0
RW = /afs/cell/.mount_point
RO = /afs/cell/mount_point
26

27. Administration Beolink.org
PTS Command
 Create id for users and groups
 Users have a positive number
 Groups use negative number
 Management of Group/User membership
 Management metadata (group quota, flags)
IMPORTANT
 Users can create their own groups
 Special groups
system:anyuser
system:authuser
system:administrators
27

28. Limits Beolink.org
Limits
28

29. Limits Beolink.org
General Limits
OpenAFS can support a maximum of 114.000 clients per
server
tmpfs no work as AFS Cache, (ramdisk work)
 Max 255 partition per server (/vicepa-/vicepiv), no limits in
partition size
 Max 4,294,967,295 volumes per partition (this a limit of
VLDB),
 Max file limit per directory is 64,000 files (less than 16
characters).
 Windows Limits
 No integration on Microsoft DFS
 No native implementation
29

30. Limits Beolink.org
AFS does not allow certain type of files:
 Pipes
 Device files
 Socket
 AFS cannot do byte range locking on all platform
 Client has working byte rage locks
 Full file locks on the server
ACLs works on directories not files (yes)
 AFS does not support mandatory file locks
 DES Encryption for file transport
30

31. Weaknesses Beolink.org
AFS is not so well suited for these situation
 No reuse of read data
 Access to file larger than cache
 Mostly write access
 Larger numbers of directory entry changes from multiple clients
31

32. Introduction Beolink.org
Full circle
 Storage is cheap.
 Managing storage is more expensive.
Wide access to data is still critical.
Today and into the future.
32

33. Thanksto... Beolink.org
Alf Watchsmann
for usage of “Introduction to AFS and its Best Practices”
Please read the original presentation for a complete overview
http://workshop.openafs.org/afsbpw10/
For more information read Documentation on www.openafs.org
Other presentation are available on www.beolink.org
33

34. I look forwardto meeting you… Beolink.org
XVII European AFS meeting 2011
HAMBURG – GERMANY
Who should attend:
 Everyone interested in deploying a globally accessible
file system
 Everyone interested in learning more about real
world usage of Kerberos authentication in single
realm and federated single sign-on environments
 Everyone who wants to share their knowledge and
experience with other members of the AFS and
Kerberos communities
 Everyone who wants to find out the latest
developments affecting AFS and Kerberos
More Info: http://www.openafs.org/
34
16/02/2012

35. Thank you
manfred@freemails.ch
Beolink.org