The document discusses the BIG DATA department of Recruit Technologies. It describes how the department has used Amazon's Elastic MapReduce (EMR) and Elastic Compute Cloud (EC2) services since 2010 to perform big data analytics on datasets using Hadoop. It provides details on how the department configures and manages EMR clusters on EC2 spot instances to perform tasks like log analysis and recommendation algorithms in a cost-effective manner. Various strategies are discussed around optimizing the use of spot instances and EMR to reduce costs while managing reliability.
20. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
Mapper, Reducer
HDFS
21. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
Frequency of being outbid (week)
22. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
Frequency of being outbid (week)
. https://aws.amazon.com/jp/ec2/spot/bid-advisor/ ( 2016-12-09)
23. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
r3.xlarge
r3.2xlarge
r3.4xlarge
1
2
4
/
24. (C) Recruit Technologies Co.,Ltd. All rights reserved.
BIG DATA Department
インスタンスタイプ
取得するスポットインスタン
スの値段
値段
r3.xlarge $ 0.1 $ 0.1 / 1 = $ 0.1
r3.2xlarge $ 0.3 $ 0.3 / 2 = $ 0.15
r3.4xlarge $ 0.35 $ 0.35 / 4 = $ 0.0875