3. In a Nutshell: Old vs New Old: 1 Layer New: 2 Layers 3 FileContext User API User API FileSystem FS Impl API AbstractFileSystem FS Impl API FS implementations S3 DistributedFS S3Fs LocalFS LocalFs Hdfs
4. Motivation (1): 1st class URI file system namespace First class URI file system namespace Old: Shell offers a URI namespace view: e.g. cp uri1 uri2 But at user-API layer, Create a FileSystem instance for each target scheme-authority Incorrect: FileSystem.create(uriPath..) must take path in the FileSystem instance Correct: Fs = FileSystem.get(Uri1, …) Fs.create(pathInUri1, …) Original patch for symbolic link depicts the problem FileSystem.open(UriPath, … ) is invalid if UriPath is foreign But one can fool the FileSystem into following a symlink to the same UriPath. Need a layer that provides first class URI file namespace 4
5. Motivation (2) : Separate Layers Two layers are merged in old FileSystem User Api which provides notion of default file system and working dir Implementation Api for implementers of the file systems Why separate? Simpler to implement file systems an API for implementing file systems (like VFS in Unix). It does not need to deal with Slash, wd, umask, etc Each file system instance is limited to its namespace User Api layer provides a natural place for The context: Slash, wd, umask, …. Uri namespace – cuts across the namespace of all file system instances. Hence a natural place to implement symbolic links to foreign namespaces 5
6. Motivation (3): Cleanup API and Semantics FileSystem API & some semantics are not very good We should have adapted Unix Apis where appropriate Ask yourself: are you smarter than Ritche & Thompson and understand the issues well enough to be different Semantics: the recursive parent creation This convenience can cause accidental creation of parents E.g. A problem for speculative executions Semantics: Rename method, etc Too many overloaded methods … (eg. Create) The cache has leaked through: FileSystem.getNewInstance() Ugliness: e.g. copyLocal() copy(), … FileSystem leaked into Path Adding InterruptedException Some could have been fixed in the FileSystem class, but was getting messy to provide compatibility in the transition A clean break made things much easier 6
7. Motivation (4): The Config The client-side config is too complex: Client should only need: Slash, wd, umask; that’s it nothing more. But Hadoop needs server-side defaults in client-side config An unnecessary burden on the client and admin Cluster admin cannot be expected to copy the config to the desktops Does not work for a federated environment where a client connects to many file systems each with its own defaults Solution: client grabs/uses needed properties from target server A transition to this solution from the current config is challenging if one needs to maintain compatibility within the existing APIs A common complaint is that Hadoop config is way too complicated 7
8. 8 The New File System APIs HADOOP-4952, HADOOP-6223
9. First: Some Naming Fundamentals Addresses, routes, names are ALL names (identifiers) Numbers, or strings, or paths, or addresses or routes are chosen based on the audience or how they are processed ALL names are relative to some context Even absolute or global names have a context in which they are resolved Names appear to be global/absolute because you have simply chosen a frame-of-reference and are excluding the world outside that frame-of-reference. When two worlds, that each have “global” names, collide names get ambiguous unless you manage the closure/context There is always an implicit context – if you make that implicit context to be explicit by naming it, you need a context for the name of the context A more local context makes apps portable across similar environments A program can move from one Unix machine to another as long the names relative to the machine’s root refer to the “same” objects A Unix process’s context: Root and working dir plus default default domain, etc. 9
10. We have URIs, why do we need Slash-relative names? Our world: a forest of file systems, each referenced by its URI Why isn’t the URI namespace good enough? The URI’s will bind your application to the very specific servers that provide that URI namepace. A application may run on cluster 1 today and be moved to cluster two in the future. If you move the data to the second cluster the app should work Better to let each cluster have its on default fs (i.e. slash) Also need the convenience of working dir 10
11. Enter FileContext: A focus point on a forest of file systems A FileContext is a focus point on a forest of file systems In general, it is set for you in your environment (just like your DNS domain) It lets you access the common files in your cluster using location independent names Your home, tmp, your project’s data, You can still access the files in other clusters or file systems In Unix you had to mount remote file systems But we have URIs which are fully qualified, automatically mounted Fully qualified Uri is to Slash-relative-name as Slash-relative-names is to wd-relative-name … its just contexts …. 11 /foo / wd / wd hdfs://nn3/foo ….
12. Examples Use default config which has your default FS myFC = FileContext.getFileContext(); Access files in your default file system myFC.create(“/foo”, ...); myFC.setWorkingDir(“/foo”) myFC.open (“bar”, ...); Access files in other clusters myFC.open(“hdfs://nn3/bar”, ..) You can even set your wd to another fs! myFC. setWorkingDir(“hdfs://nn3/foo”) Variations on getting your context A specific URI as the default FS myFC = FileContext.getFileContext(URI) Local file system as the default FS myFC = FileContext.getLocalFSFileContext() Use a specific config, ignore $HADOOP_CONFIG Generally you should not need use a config unless you are doing something special configX = someConfigPassedToYou. myFC =FileContext.getFileContext(configX); //configX not changed but passed down 12
13. So what is in the FileContext? The default file system (Slash) - obtained from config A pointer to the file system object is kept The working dir (lack of Slash) Stored as a path which is prefixed to relative path names Umask – obtained from config Absolute permissions after applying mask are sent to layer below Any other file system accessed are simply created 0.21 – uses the FileSystem which has a cache 0.22 – use the new AbstractFileSystem Do we need to add a cache? Hadoop-6356 13
14. HDFS config – client & server side Client side config: Default file system Umask Default values for blocksize, buffersize, replication are obtained at runtime from the specific filesystem in question Finally, federation can work Server side config: What used to be there before (except the above two items) + cleaned config variables for SS defaults for blocksize, etc. 14
18. The Jira: Approx 9 months FileContext: Three main discussions Use Java’s new IO. Main issue is that their default and wd is tied to the file system of the JVM in order to provide compatibility with older APIs Config – a simpler per-process environment? Differed to Doug’s view that a per-process env is not sufficient for a MT Java application Besides, out of scope for this Jira – a radical change SS config What I would have liked to do, but didn’t 2 Apis: Java Interface and Abstract class Not the Hadoop way! For example if we had done this in the old FileSystem, it would have facilitated dynamic class loading for protocol compatibility as an interim solution 16
19. What is next? Exceptions: Adding InterruptedException and declaring the sub-exceptions of IOException Issue: Apps need to manage an interrupt differently then other exceptions IO-streams throw IOInterruptedException FileContext – three choices: IOInterruptedException InterruptedException An unchecked exception The Cache: Do we need it? How do we deal with Facebook’s application that forced the cache to leak through Other AbstractFileSystem impls (can use the delegator) 17
