Weitere ähnliche Inhalte
Ähnlich wie MapR NFS Guide for Mounting Clusters and Accessing Data
Ähnlich wie MapR NFS Guide for Mounting Clusters and Accessing Data (20)
Mehr von mapr-academy (9)
Kürzlich hochgeladen (20)
MapR NFS Guide for Mounting Clusters and Accessing Data
- 1. MapR NFS
7/6/2012
© 2012 MapR Technologies Storage Service Architecture 1
- 2. MapR NFS
Agenda
• Before NFS
• Exercise: Using hadoop fs
• MapR NFS Overview
• NFS Mounting Models
• Exercise: Mount the Cluster to Itself and Copy a File
© 2012 MapR Technologies Storage Service Architecture 2
- 3. MapR NFS
Objectives
At the end of this module you will be able to:
• Describe the tools that are used with HDFS and their limitations
• List the features and benefits of MapR Direct Access NFS
• Identify several use cases for NFS on Hadoop
• Describe the different models for mounting NFS volumes with MapR
© 2012 MapR Technologies Storage Service Architecture 3
- 5. Before NFS
Getting work done without MapR NFS
– Example: mining web logs for meaningful data
• First: get the logs into the cluster
– Do you stop writing to the logs?
– How long does it take?
– What do you do in the meantime?
hadoop fs -put, copyfromlocal
distcp
Flume
© 2012 MapR Technologies Storage Service Architecture 5
- 6. Before NFS
Getting work done without MapR NFS
– Example: mining web logs for meaningful data
• Next: getting answers from the data
– What tools do you use to access data in your cluster?
– Who is using these tools?
hadoop fs
copytolocal
© 2012 MapR Technologies Storage Service Architecture 6
- 7. Exercise:
hadoop fs
© 2012 MapR Technologies Storage Service Architecture 7
- 9. Introducing Direct Access NFS
Drag and drop
Open files and apps
Watch live data come in
Fully read/write
Multiple concurrent reads/writes
Easy to get data in/out
Same namespace paths as Hadoop
Mount with
-o tcp,ver=3,nolock
© 2012 MapR Technologies Storage Service Architecture 9
- 10. Getting Data into the Cluster with NFS
Simply mount your cluster filesystem locally
– /mapr/my.cluster.com/myvolume/…
Redirect (or copy) logs to cluster filesystem
log data
© 2012 MapR Technologies Storage Service Architecture 10
- 11. Using NFS with Clients
Mount your cluster filesystem locally
– /mapr/my.cluster.com/myvolume/…
Access all of the data in your cluster with standard applications
– Double-click a file (spreadsheet, report, etc.)
• edit, save, etc.
– Run a perl script
– grep, sed, awk, shell scripts, etc.
Use Hadoop apps (Hive, Pig, Hbase, etc.)
© 2012 MapR Technologies Storage Service Architecture 11
- 12. Using NFS with Clients
Data is constantly changing
How do I view the data at a point in time?
– Easy: take a snapshot
– cd to the snapshot volume
– View the data of interest
– Note: overhead for taking a snapshot: none to minimal
• The only write overhead is the data that changed
– Recall snapshot architecture
© 2012 MapR Technologies Storage Service Architecture 12
- 13. Using NFS with Legacy Applications
What if I have legacy applications that work with my data?
– Log analysis, reporting tools, etc.
Mount the cluster where your tools run
– Take a snapshot if necessary
– Then use your app just as before
© 2012 MapR Technologies Storage Service Architecture 13
- 15. NFS Mounting Models
Export to the world
– NFS gateway runs on selected gateway hosts
Local server
– NFS server runs on local host
– Enables local compression and check summing
Export to self
– NFS gateway runs on all data nodes, mounted from localhost
© 2012 MapR Technologies Storage Service Architecture 15
- 16. Export to the World
NFS
NFS
Server
NFS
Server
NFS
Server
NFS Server
Client
© 2012 MapR Technologies Storage Service Architecture 16
- 17. Local Server
Application
NFS
Server
Client
Cluster
Nodes
© 2012 MapR Technologies Storage Service Architecture 17
- 18. Universal Export to Self
Cluster Nodes
Application
NFS
Cluster Server
Node
© 2012 MapR Technologies Storage Service Architecture 18
- 19. Nodes are Identical
Application
Application
NFS
NFS
Cluster Server
Node Cluster Server
Node
Application
NFS
Cluster Server
Node
© 2012 MapR Technologies Storage Service Architecture 19
- 20. Exercise:
Mount the Cluster to
Itself and Copy a File
© 2012 MapR Technologies Storage Service Architecture 20