This document provides tips for tuning Hadoop clusters and jobs. It recommends: 1) Choosing optimal numbers of mappers and reducers per node and oversubscribing CPUs slightly. 2) Adjusting memory allocations for tasks and ensuring they do not exceed total memory available. 3) Increasing buffers for sorting and shuffling, compressing intermediate data, and using combiners to reduce data sent to reducers.

1. How to Debug and Tune Hadoop Alex Rovner Proclivity Systems

3. Tune Your Cluster

5. mapred.tasktracker.reduce.tasks.maximum

6. Oversubscribe the CPU by 20-30% (8 Cores can generally handle 10 slots)

7. Mappers to reducers ratio 4:3

9. Use 80% of available memory

10. Do not oversubscribe memory to avoid swapping

11. Total Memory = Map Slots + Reduce Slots + TT + DN + Other Services + OS

13. Set to 60-70% of Java heap size

15. Install native libraries for performance

16. Use LZO to minimize CPU cycles

17. Set compression to use BLOCK compression

19. Reduces the amount of data sent to the reducers

20. Does not need to be the same class as the reducer

22. Map output bytes will drive how many reducers you need

23. Rule of thumb is 1 Gig per reducer

24. Tune Your Job

25. Tune Your Job 7.4GB = 7 Reducers

26. Tune Your Job

27. Tune Your Job

28. Tune Your Job

29. Demo