Apache Tez is the new data processing framework in the Hadoop ecosystem. It runs on top of YARN - the new compute platform for Hadoop 2. Learn how Tez is built from the ground up to tackle a broad spectrum of data processing scenarios in Hadoop/BigData - ranging from interactive query processing to complex batch processing. With a high degree of automation built-in, and support for extensive customization, Tez aims to work out of the box for good performance and efficiency. Apache Hive and Pig are already adopting Tez as their platform of choice for query execution.
For anyone who has been working on MapReduce, there is this age-old problem around “how do I figure out the correct number of reducers?”. We guess some number at compile-time and usually that turns out to be incorrect at run-time. Let’s see how we can use the Tez model to fix that. So here is this Map Vertex and this Reduce Vertex, which have these tasks running and you have the Vertex Manager running inside the framework …
[CLICK] The Map Tasks can send Data Size Statistics to the Vertex Manager, which can then extrapolate those statistics to figure out “what would be the final size of the data when all of these Maps finish?”. Based on that, it can realize that the data size is actually smaller than expected, and I can actually run two reduce tasks instead of three.
[CLICK] The Vertex Manager sends a Set Paralellism command to the framework which changes the routing information in-between these two tasks and also cancels the last task.
So the new plan looks like this … which is how Tez handles the concurrency problem of plan re-configuration. What about “when do you actually want to launch those tasks?”?
In MapReduce, there is an interesting concept of pre-launching the Reducers, in which even though the reducers are going to read the output from 100 mappers, we may want to launch the Reducers when only 50 mappers have finished – because then they can use the capacity in the cluster to start pre-fetching the data from the completed mappers while the rest of their inputs still complete. How can we do this in Tez? Again, the same Vertex Manager comes into the picture.
[CLICK] As the Map Tasks complete they send Task Completed notifications to the Vertex Manager. There it can decide that enough Map tasks have finished that I need to start one of my reducers so that it can start pre-fetching the data.
[CLICK] It sends a Start Task notification to the framework, which results in one of the Reduce Tasks starting.
query 1: SELECT pageURL, pageRank FROM rankings WHERE pageRank > X