Hadoop
•Als PPT, PDF herunterladen•
0 gefällt mir•593 views
Hadoop is an open-source software framework that allows for distributed processing of large datasets across computer clusters using a simple programming model. It has an architecture that includes Apache HDFS for storage, MapReduce for processing, and additional components like Hive for data summarization and HBase for column-oriented storage. Effective troubleshooting requires understanding Hadoop's architecture and where performance bottlenecks may occur.
![What is Hadoop: The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model 1 ,[object Object],[object Object],Execution overview 2](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (14)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Hadoop
- 1. Hadoop Realtime Question http://goo.gl/4F7Tj
- 9. Trouble Shooting: May the source be with you
Hinweis der Redaktion
- Extended reading: higher order function. map , reduce Cluster Computing and MapReduce Lecture online course provided by Google on YouTube,talking about "the cloud' MapReudce paper
- Extended Reading: problem of NameNode,the famous single point of failure. Facebook`s solution of hadoop HA The Next Generation of Apache Hadoop MapReduce s4.io another distributed computing platform Twitter Strom real time processing ,but not yet opensource/
- More Detail: commonly speaking,you can implement the InputFormat interface to do what ever you want. but be care of the getSplits method. in general , you may think of just implement the InputSplit ,but in the hadoop internal work flow,at certain point,the return value will used to cast as FileInputSplit. so,if it is not,you will certainly fail the job submission. in most of the case,just use the TextInputFormat,it will serve most of your case. a little tuning may apply to the split stage.since the split is base on file size and block size,with larger block size configuration,you *MAY* get less splits and so decrease the task total.
- data writes are pipelines,not until the next node ack the block,will the current node actually write to the disk. Extended Reading: Google File System ,the original theory model of HDFS
- you can have multiple masters,and zookeeper will select one as live,and the other as backup until the current died. one CF per store file. WAL can be disable. in most of our case it worth to do so. 7k insertion in our cluster,when writes to a single region.the region will hold a big lock aim to ensure write consistent,and so block the other current request.make more regions *MAY* increase the throughput
- no benchmark in our cluster. some tuning may worth consideration. IO schedule fs journal mode hadoop read write buffer instance Numbers per node filesystem type branch the 哈
- read the source,and it will tell you why,at least. some consideration can be apply before setting up an cluster.such as separate the disk partitions for hadoop,avoid complex calculation of disk reservation