Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

EMRでスポットインスタンスの自動入札ツールを作成する

1.388 Aufrufe

Veröffentlicht am

2016/12/12 BigData-JAWS 勉強会#4での、荻原の講演資料になります

Veröffentlicht in: Technologie
  • Login to see the comments

  • Gehören Sie zu den Ersten, denen das gefällt!

EMRでスポットインスタンスの自動入札ツールを作成する

  1. 1. (C) Recruit Technologies Co.,Ltd. All rights reserved. EMR
  2. 2. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department   29    Apple Hadoop
  3. 3. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department 2010 2016 2012 2014 / / / Hadoop / / AWS( EMR)
  4. 4. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department 2010 2016 2012 2014 / / / Hadoop / / AWS( EMR)
  5. 5. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department    
  6. 6. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA DepartmentEMR   ( 1 ) ( )    SQL  EMR   Hadoop   11 100  EMR
  7. 7. (C) Recruit Technologies Co.,Ltd. All rights reserved.
  8. 8. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  EC2   EC2 30 50% )   Hadoop
  9. 9. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department    HIGH  LOW
  10. 10. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department client Amazon EMR Amazon EC2 Amazon EC2 Amazon EC2 ......
  11. 11. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department   30 5
  12. 12. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  5 15 + 15
  13. 13. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  30 5 30
  14. 14. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  5 15 + 15
  15. 15. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  16. 16. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  17. 17. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  18. 18. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  19. 19. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  20. 20. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department   Mapper, Reducer  HDFS
  21. 21. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department Frequency of being outbid (week)
  22. 22. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department Frequency of being outbid (week) . https://aws.amazon.com/jp/ec2/spot/bid-advisor/ ( 2016-12-09)
  23. 23. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department   r3.xlarge  r3.2xlarge  r3.4xlarge  1  2  4 /
  24. 24. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department インスタンスタイプ 取得するスポットインスタン スの値段 値段 r3.xlarge $ 0.1 $ 0.1 / 1 = $ 0.1 r3.2xlarge $ 0.3 $ 0.3 / 2 = $ 0.15 r3.4xlarge $ 0.35 $ 0.35 / 4 = $ 0.0875
  25. 25. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  48      10
  26. 26. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  48  1 2 48
  27. 27. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department
  28. 28. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  48      10
  29. 29. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  or 4  r3.xlarge  r3.2xlarge  r3.4xlarge  r3.8xlarge   e.g. r3.2xlarge オンデマンド価格 スポットインスタンス価格 $ 0.798 0.798 * 10% = $ 0.0798 0.798 * 20% = $ 0.1596 … 0.798 * 90% = $ 0.7182
  30. 30. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department  48      10
  31. 31. (C) Recruit Technologies Co.,Ltd. All rights reserved. BIG DATA Department    WebUI   JSON

×