Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Llap: Locality is Dead

679 Aufrufe

Veröffentlicht am

Hive LLAP Locality processing challenges and advantages

Veröffentlicht in: Software
  • Als Erste(r) kommentieren

Llap: Locality is Dead

  1. 1. Page1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved LLAP: Locality is dead (in the cloud) Gopal Vijayaraghavan
  2. 2. Page2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Locality – as usually discussed Disk CPU Memory Network Share-nothing Shared
  3. 3. Page3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Cloud – The Network eats itself Network Processing Memory Network Shared
  4. 4. Page4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Cutaway Demo – LLAP on Cloud TL;DW - repeat LLA+S3 benchmark on HDC 3 LLAP (m4.xlarge) nodes, Fact table has 864,001,869 rows -------------------------------------------------------------------------------- VERTICES: 06/06 [==========================>>] 100% ELAPSED TIME: 1.68 s -------------------------------------------------------------------------------- INFO : Status: DAG finished successfully in 1.63 seconds INFO : Hortonworks Data Cloud LLAP is >25x faster EMR
  5. 5. Page5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved • Wait a minute – is this a new problem? • How well do we handle data locality on-prem? • Fast BI tools, how long can they afford to wait for locality? • We do have non-local readers sometimes, I know it • I mean, that’s why we have HDFS right? Amdahl’s law knocks, who answers?
  6. 6. Page6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Locality – BI tools fight you, even on-prem Disk CPU Memory Network Share-nothing Shared
  7. 7. Page7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Locality – what it looks like (sometimes) HDFS CPU Memory Network Share-nothing Shared
  8. 8. Page8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Evaluation week of cool new BI tool – easy mistakes Rack#1 Mistake #1 – use whole cluster to load sample data (to do it real fast, time is money) Mistake #2 – use whole cluster to test BI tool (let’s really see how fast it can be) Mistake #3 – Use exactly 1 rack (we’re not going to make that one) Rack#2 Rack#3 ☑ ☑ ☒
  9. 9. Page9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Someone says “Data Lake” and sets up a river Rack#1 BI – becomes a 30% tenant Rack#2 Rack#3 Arguments start on how to set it up How about 1 node every rack? We’ll get lots of rack locality All joins can’t be co-located, so shuffle is always cross-rack – SLOW! And you noticed that the Kafka pipeline running on rack #2 is a big noisy neigbhour Fast is what we’re selling, so that won’t do
  10. 10. Page10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved “Noisy network neighbours? Get a dedicated rack” Rack#1 BI – gets its own rack. ETL – gets the other two Rack#2 Rack#3 All files have 3 replicas - but you might still not have rack locality. 3 replicas in HDFS – always uses 2 racks (3rd replica is in-rack to 2nd) replication=10 on 20-node racks, uses 2 racks (1+9 replicas)
  11. 11. Page11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Dedicated rack – your DFS IO is now crossing racks Rack#1 The real victims are now broadcast joins – which scan fact tables over the network. If your ETL is coming from off-rack – 50% probability that your new data has no locality in rack #1 You either have 2 replicas in Rack #1 or none. Rack#2 Rack#3 If you try to fix this with a custom placement policy, the DNs on rack #1 will get extra writes Tail your DFS audit logs folks – there’s so much info there
  12. 12. Page12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Rack#1 However, you realize that you can make a “copy” in rack #1 But for this cluster, until you setrep 7, there’s no way to be sure rack #1 has a copy. Rack#2 Rack#3 Cache!
  13. 13. Page13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Caching efficiently - LLAP’s tricks LLAP’s cache is decentralized, columnar, automatic, additive, packed and layered. When a new column or partition is used, the cache adds to itself incrementally - unlike immutable caches
  14. 14. Page14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved LLAP cache: ACID transactional snapshots LLAP cache is built to handle Hive ACID 2.x data with overlapping read transactions. With failure tolerance across retries Q1Q2 LLAP Partition=1 [txns=<1,2>] Partition=1 [txns=<1>] ✔ ✔ Partition=1 (retry) [txns=<1,2>] Partition=1 (retry) [txns=<1>] ✔ ✔ HIVE-12631 This works with a single cached copy for any rows which are common across the transactions. The retries work even if txn=2 deleted a row which existed in txn=1. Q2 is a txn ahead of Q1 Same partition, different data (in cache)
  15. 15. Page15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Locality is dead, long live cache affinity Rack#1 1 3 2 Split #1 [2,3,1] Split #2 [3,1,2] If node #2 fails or is too busy, scheduler will skip. When a node reboots, it takes up lowest open slot when it comes up A reboot might cause an empty slot, but won’t cause cache misses on others
  16. 16. Page16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions? This presentation represents the work of several folks from the Hive community over several years – Sergey, Gunther, Prasanth, Rajesh, Nita, Ashutosh, Jesus, Deepak, Jason, Sid, Matt, Teddy, Eugene and Vikram.

×