Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Interview questions on Apache spark [part 2]

1.385 Aufrufe

Veröffentlicht am

This is Apache Spark Question & Answer Tutorial.

We provide training on Big Data & Hadoop,Hadoop Admin ,MongoDB,Data Analytics with R, Python..etc

Our Big Data & Hadoop course consists of Introduction of Hadoop and Big Data,HDFS architecture ,MapReduce ,YARN ,PIG Latin ,Hive,HBase,Mahout,Zookeeper,Oozie,Flume,Spark,Nosql with quizzes and assignments.

To watch the video or know more about the course, please visit http://www.knowbigdata.com/page/big-data-spark

Veröffentlicht in: Bildung

Interview questions on Apache spark [part 2]

  1. 1. www.KnowBigData.comBig Data and Hadoop Welcome to Interview Questions on Apache Spark [Part 2] +1 419 665 3276 (US) +91 803 959 1464 (IN) reachus@knowbigdata.com Subscribe to our Youtube channel for latest videos - https://www.youtube.com/channel/UCxugRFe5wETYA7nMH6VGyEA
  2. 2. www.KnowBigData.comBig Data and Hadoop ABOUT INSTRUCTOR - SANDEEP GIRI 2014 KnowBigData Founded 2014 Amazon Built High Throughput Systems for Amazon.com site using in-house NoSql. 2012 2012 InMobi Built Recommender that churns 200 TB 2011 tBits Global Founded tBits Global Built an enterprise grade Document Management System 2006 D.E.Shaw Built the big data systems before the term was coined 2002 2002 IIT Roorkee Finished B.Tech.
  3. 3. www.KnowBigData.comBig Data and Hadoop ❏ Expert Instructors ❏ CloudLabs ❏ Lifetime access to LMS ❏ Presentations ❏ Class Recording ❏ Assignments + Quizzes ❏ Project Work WELCOME - KNOWBIGDATA ❏ Real Life Project ❏ Course Completion Certificate ❏ 24x7 support ❏ KnowsBigData - Alumni ❏ Jobs ❏ Stay Abreast (Updated Content, Complimentary Sessions) ❏ Stay Connected
  4. 4. www.KnowBigData.comBig Data and Hadoop QUESTION 1 Say I have a huge list of numbers in RDD (say myrdd). And I wrote the following code to compute average: def myAvg(x, y): return (x+y)/2.0; avg = myrdd.reduce(myAvg);
  5. 5. www.KnowBigData.comBig Data and Hadoop QUESTION 1 ANSWER: The average function is not commutative and associative; I would simply sum it and then divide by count. def sum(x, y): return x+y; total = myrdd.reduce(sum); avg = total / myrdd.count(); The only problem with the above code is that the total might become very big thus over flow. So, I would rather divide each number by count and then sum in the following way. cnt = myrdd.count(); def devideByCnd(x): return x/cnt; myrdd1 = myrdd.map(devideByCnd); avg = myrdd.reduce(sum);
  6. 6. www.KnowBigData.comBig Data and Hadoop QUESTION 2 Say I have a huge list of numbers in a file in HDFS. Each line has one number. And I want to compute the square root of sum of squares of these numbers. How would you do it?
  7. 7. www.KnowBigData.comBig Data and Hadoop QUESTION 2 ANSWER: There could two approaches: 1. numsAsText = sc.textFile("hdfs://hadoop1.knowbigdata. com/user/student/sgiri/mynumbersfile.txt"); def toSqInt(str): v = int(str); return v*v; 2. nums = numsAsText.map(toInt); total = nums.reduce(sum) import math; print math.sqrt(total);
  8. 8. www.KnowBigData.comBig Data and Hadoop QUESTION 3 Is the following approach correct? Is the sqrtOfSumOfSq a valid reducer? numsAsText = sc.textFile("hdfs://hadoop1.knowbigdata. com/user/student/sgiri/mynumbersfile.txt"); def toInt(str): return int(str); nums = numsAsText.map(toInt); def sqrtOfSumOfSq(x, y): return math.sqrt(x*x+y*y); total = nums.reduce(sum) import math; print math.sqrt(total);
  9. 9. www.KnowBigData.comBig Data and Hadoop QUESTION 3 Is the following approach correct? Is the sqrtOfSumOfSq a valid reducer? ANSWER: Yes. The approach is correct and sqrtOfSumOfSq is a valid reducer.
  10. 10. www.KnowBigData.comBig Data and Hadoop QUESTION 4 Which approach is better? Could you compare the pros and cons of the your approach and my approach?
  11. 11. www.KnowBigData.comBig Data and Hadoop QUESTION 4 Which approach is better? Could you compare the pros and cons of the your approach and my approach? ANSWER: You are doing the square and square root as part of reduce code while I am squaring in map() and summing in reduce in my approach. My approach will be faster because in your case the reducer code is heavy as it is calling math.sqrt() and reducer code is generally executed approximately n-1 times. The only downside of my approach is that there is a huge chance of integer overflow because I am computing the sum of squares as part of map.
  12. 12. www.KnowBigData.comBig Data and Hadoop QUESTION 5 How do you do the word count using Spark?
  13. 13. www.KnowBigData.comBig Data and Hadoop QUESTION 5 How do you do the word count using Spark? ANSWER: lines = sc.textFile("hdfs://hadoop1.knowbigdata.com/user/student/sgiri/bigtextfile.txt");
  14. 14. www.KnowBigData.comBig Data and Hadoop QUESTION 6 In a very huge text file, you want to just check if a particular keyword exists. How would you do this using Spark?
  15. 15. www.KnowBigData.comBig Data and Hadoop QUESTION 6 ANSWER: lines = sc.textFile("hdfs://hadoop1.knowbigdata.com/user/student/sgiri/bigtextfile.txt"); def isFound(line): if line.find(“mykeyword”) > -1: return 1; return 0; foundBits = lines.map(isFound); sum = foundBits.reduce(sum); if sum > 0: print “FOUND”; else: print “NOT FOUND”; In a very huge text file, you want to just check if a particular keyword exists. How would you do this using Spark?
  16. 16. www.KnowBigData.comBig Data and Hadoop QUESTION 7 Can you improve the performance of this code in previous answer?
  17. 17. www.KnowBigData.comBig Data and Hadoop QUESTION 7 ANSWER: Yes. The search is not stopping even after the word we are looking for has been found. Our map code would keep executing on all the nodes which is very inefficient. We could utilize accumulators to report whether the word has been found or not and then stop the job. Something on these line: import thread, threading from time import sleep result = "Not Set" lock = threading.Lock() accum = sc.accumulator(0) Can you improve the performance of this code in previous answer? ….CONTD
  18. 18. www.KnowBigData.comBig Data and Hadoop QUESTION 7 def map_func(line): #introduce delay to emulate the slowness sleep(1); if line.find("Adventures") > -1: accum.add(1); return 1; return 0; def start_job(): global result try: sc.setJobGroup("job_to_cancel", "some description") lines = sc.textFile("hdfs://hadoop1.knowbigdata. com/user/student/sgiri/wordcount/input/big.txt"); ….CONTD
  19. 19. www.KnowBigData.comBig Data and Hadoop QUESTION 7 result = lines.map(map_func); result.take(1); except Exception as e: result = "Cancelled" lock.release() def stop_job(): while accum.value < 3 : sleep(1); sc.cancelJobGroup("job_to_cancel") supress = lock.acquire() supress = thread.start_new_thread(start_job, tuple()) supress = thread.start_new_thread(stop_job, tuple()) supress = lock.acquire()
  20. 20. www.KnowBigData.comBig Data and Hadoop Thank You. +1 419 665 3276 (US) +91 803 959 1464 (IN) reachus@knowbigdata.com Subscribe to our Youtube channel for latest videos - https://www.youtube.com/channel/UCxugRFe5wETYA7nMH6VGyEA

×