Matt Kimball, AMD Server Solutions Marketing presentation on "Cutting Big Data Down to Size with AMD and Dell" from Dell World.
Learn how “Hadoop” solutions are helping companies overcome growing pressures on IT budgets with an innovative approach to Big Data.
2. WHAT‘S BIG DATA?
BIG DATA
Data sets sized beyond the ability of commonly
used tools to capture, manage, and process the
data within a tolerable time.
BIG DATA Technologies
A new generation of technologies designed to
economically extract value from very large
volumes of a wide variety of data.
Definitions derived from Gartner, IDC, and NIST documents
2 | Cutting Big Data Down to Size | Dell World – December 2012 |
3. DROWNING IN DATA?
Proliferation of
Increasing data
unstructured
complexity
data
Growing Surge in
numbers of external data
volumes 2.5 streams
quintillion
bytes of
data created
daily*
3 | Cutting Big Data Down to Size | Dell World – December 2012 |
4. WHAT‘S HADOOP?
Framework for distributed processing of large data sets across
clusters of computers using a simple programming model
• Scalable, fault-tolerant, distributed system for data storage and processing
• Foundation of rapidly growing ecosystem of supporting projects
• Created by developers to handle intractable problem (Cloudera, Ebay, Facebook,
Hortonworks, Linkedin, Twitter, and Yahoo!)
• Open source project hosted under the Apache Software Foundation
http://hadoop.apache.org/
4 | Cutting Big Data Down to Size | Dell World – December 2012 |
5. HOW IS HADOOP BEING USED?
INDUSTRY TERM VERTICAL INDUSTRY TERM
Social network analysis Web Clickstream sessionization
PROCESSING
Media
ANALYTICS
ADVANCED
Content optimization Clickstream sessionization
DATA
Network analytics Telco Mediation
Loyalty and promotions analysis Retail Data factory
Fraud analysis Financial Trade reconciliation
Entity analysis Federal SIGINT
Source: Cloudera
5 | Cutting Big Data Down to Size | Dell World – December 2012 |
6. WHO IS USING, CONTRIBUTING, AND BUILDING FOR HADOOP?
10Gen Concurrent Klout SnapLogic
Accela Datameer LexisNexis StackIQ
Adobe eBay LinkedIn StumbleUpon
Alibaba.com eHarmony Microsoft SunGard
Amazon.com EMC NetApp Talend
AOL EnterpriseDB Netflix Telenav
AT&T Google Nokia Teradata
Baidu Groupon NTT Communications The New York Times
Bank of America Hortonworks Orbitz Trend Micro
bitly Hulu Pandora Twitter
Booz Allen Hamilton IBM Pervasive Tynt
CBS IIIT Hyderabad Rackspace Hosting Vertica
CBS Interactive Intel Samsung Visa
Cisco Jaspersoft SGI VoltDB
CNET JPMorgan Chase Sky Yahoo
Source: Hortonworks
And of course AMD!
http://wiki.apache.org/hadoop/PoweredBy
6 | Cutting Big Data Down to Size | Dell World – December 2012 |
7. WHY WAS HADOOP CREATED?
Exploding data volumes Dramatic changes in enterprise data
LEADS TO
and types management
DIGITAL
CONTENT
OPERATIONAL • Extract more value from
NEW
WEB DATA more data more cost
OPPORTUNITY
LOGS FOR USING effectively with greater
SOCIAL
FILES MEDIA SMART CREATED DATA flexibility
GRIDS
• Deep analysis
TRANSACTIONAL HARD PROBLEMS • Exhaustive and detailed
DATA • Sophisticated algorithms
AD • Quick results
IMPRESSIONS R&D
DATA • Any type any kind
BIG DATA • From any source
• Structured and unstructured
• It’s difficult to handle data this diverse at this scale • At scale
• Traditional platforms can’t keep pace
Source: Cloudera
7 | Cutting Big Data Down to Size | Dell World – December 2012 |
8. TRENDS AROUND BIG DATA
Enterprise and Government organizations faced with
analyzing large amounts of unstructured data.
Big Data Trends:
• Low acquisition costs
• Servers that can effectively parallelize workloads
• Cost effectively scale out Hadoop systems to
meet expanding data processing and analysis
needs
• Compute density to handle large amounts of data
Data Driven Business Decisions and Actions
8 | Cutting Big Data Down to Size | Dell World – December 2012 |
9. AMD AND DELL FOR YOUR HADOOP CLUSTERS
CPU • Medium clock speeds
Core Density for
Parallelism
Form Factor • Two sockets
Three P
Equilibrium
Power • Concern as cluster grows
Scale Out
Capability
RAM • 48GB to keep CPUs busy
Manageability
Storage • 1 spindle per core – SATA
Cost Per Node
Network • 1GB growing to InfiniBand™
10 | Cutting Big Data Down to Size | Dell World – December 2012 |
10. AMD OPTERON™ SERIES PROCESSORS FOR HADOOP
LOWEST TCO FOR DELL CUSTOMERS
CapEx
Hadoop Deployment Scenario
OpEx TCO
24 TB Data
I/O Bound
10 Users
Core Count
C6105
AMD 4274HE
Scale Out Perf 64GB RAM
12 | Cutting Big Data Down to Size | Dell World – December 2012 |
11. AMD HADOOP SCENARIO TCO
$800,000
$605,120 $705,040
$401,500 Up to 34% Lower
$600,000 CapEx Hardware Costs11
$422,842
Dell C6105
$400,000 HP SL230s
Gen8 Up to 40% Lower*
$200,000
$87,920 CapEx
$0 $12,000
$18,342 $3,000
$0
Server HW cost Server SW cost Rack switch cost Rack + install cost CapEx
$183,977
$200,000
OpEx $142,267
$105,195
Up to 23% Lower
$150,000
$100,000
$24,479
Dell C6105 OpEx*
$19,583 $34,720 HP DL360e
$50,000 $15,773
$12,619 $8,680 Gen8
$0
Server power Overhead Mgmt OH/year Footprint cost OpEx/yr
cost/year power cost/year
Up to 32% lower 3 year TCO - $407,329 savings*
* See slide titled ” Substantiation | Hadoop TCO Calculations” for assumptions and calculations
13 | Cutting Big Data Down to Size | Dell World – December 2012 |
12. HADOOP CONFIGURATION COMPARISON
DELL POWEREDGE C6105 VS HP DL360E GEN8
Dell™ PowerEdge™ C6105 (4 nodes) HP DL360e Gen8
CPU / Node AMD Opteron™ 4274HE (2) CPU Intel Xeon E5-2440 (2)
Memory / Node 64 GB (8x8 1333MHz) Memory 64 GB (8x8 1333MHz)
Storage / Node 3x 2TB SATA Storage 3 x 2TB LFF SATA
Network / Node 1GBe Network 1GBe
PSU / Node 1100W PSU 750W
80 Servers
1 42U rack (C6105) vs. 4 42U racks (DL360e Gen8)
Mellanox SX2016 10GBe switches: 2 for C6105 vs. 8 for DL360e Gen8
Hadoop Projects Software
14 | Cutting Big Data Down to Size | Dell World – December 2012 |
13. EUROPEAN AEROSPACE PROJECT HADOOP SYSTEM
Astronomy Mission Mapping Our Galaxy
• 4 service nodes
• Cluster infrastructure – 5 Nodes
• 94 x C6105 – 742 Sockets
• 4x MB w/ 2x 4274HE, 24GB, 3x 3TB
• Dell|Force10 + Dell switches
2013 Production
15 | Cutting Big Data Down to Size | Dell World – December 2012 |
14. DELL CLOUDERA SOLUTIONS
The Dell | Hadoop Big Data Solution
• Cloudera’s Distribution for Hadoop
• Dell PowerEdge™ C hardware, including the C6105 and C6145
http://dell.cloudera.com/
http://www.mellanox.com/webinars/Accelerating-Big-Data-Over-Hadoop-With-InfiniBand/
16 | Cutting Big Data Down to Size | Dell World – December 2012 |
15. THANK YOU
For more information on AMD Big Data
solutions, please visit;
www.AMD.com/BigData
17 | Cutting Big Data Down to Size | Dell World – December 2012 |
21. HADOOP FOR TELCO
4 x Motherboard system
Hadoop on Each Motherboard:
C6105 • 2 x4274HE Proc.
• 4 x 16GB RAM
• 4 x 2TB HDD
23 | Cutting Big Data Down to Size | Dell World – December 2012 |
22. SUBSTANTIATION AND FOOTNOTES
1. SVR-217 - AMD Opteron™ 6300 Series processors are expected to have up to 37% higher java performance/watt than AMD Opteron 6200 Series processors.. Estimate based on
preliminary measurements of java performance/watt in AMD labs as of August 2012 comparing AMD Opteron™ processor Models 6380 and 6278. SVR-217
2. SVR-168 - AMD Opteron™ 6300 Series processors are expected to have up to 24% higher Java performance than AMD Opteron 6200 Series processors. Estimate based on
preliminary measurements of server side Java performance in AMD labs as of August 30, 2012. 1,199,838 operations per second using 2 x AMD Opteron™ processors Model
6278. 1,489,668 operations per second using 2 x AMD Opteron™ processors Model 6380. SVR-168
3. SVR-205 - AMD Opteron™ 6300 processors are expected to offer up to 8% better integer performance than AMD Opteron™ 6200 processors. Estimate based on preliminary
SPECint®_rate2006 results in AMD labs as of August 2012 comparing AMD Opteron™ processor Models 6380 and 6278.
4. SVR-216 - AMD Opteron™ 6300 processors are expected to offer up to 7% better floating point performance than AMD Opteron™ 6200 processors. Estimates based on
preliminary SPECfp®_rate2006 measurements in AMD labs as of August 2012 comparing AMD Opteron™ processor Models 6380 and 6278. SVR-216
5. SVR-255 - The Dell R815 lowers TCO for virtualization by up to 17% versus the HP DL560 Gen8. TCO calculation based on virtualization scenario with 1600 physical servers
consolidated down to 40 with each physical server running 40 virtualized machines (12GB RAM each). This is supported by two racks of (20) servers with two Mellanox
SX1016 10GBe 64-port switches. The cost per rack with installation is $3,000 based on http://www.intel.com/technology/eep/datacenter.pdf as of 9/10/12 and (2) Mellanox
SX1016 10GBe 64-port switches cost $10,990 each and have a typical power consumption of 113 watts as of 9/10/12 at
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1377. Power costs based on $0.10 per kWhr and using 976 watts per for the Dell R815 server and 588 watts
for the HP DL560p Gen8 server according to the Dell Energy Smart Solution Advisor (ESSA) tool with a memory-intensive workload set at 100% CPU loading and an input
voltage of 220 volts as well as the HP Power Advisor Tool set at 100% CPU loading and an input voltage of 220 volts as of 9/24/12 based on the configurations below. The PUE
is 1.8. Assuming 28 square feet per rack according to http://www.energystar.gov/ia/partners/prod_development/downloads/Additional_FAQs_LightingCooling_Racks.pdf as
of 9/11/12, and the average fully burdened cost of real estate at $310 per square foot per year based on
http://www.vmware.com/technology/whyvmware/calculator/index.php as of 9/10/12. Management overhead yearly cost per server based on $81k salary with 1.25
multiplier for a fully loaded expense (benefits, taxes, etc) of $101,250. $1315 per server derived from dividing the fully loaded expense by 77 servers being managed by IT
administrator, based on page 4 at http://www.vmware.com/files/pdf/vmware-solution-opex-reducing-opex-wp-en.pdf and http://www.indeed.com/salary/Windows-Server-
Administrator.html. VMware vSphere license is $3495 per CPU as of 9/25/12 at www.vmware.com. R815 with four AMD Opteron™ processor Model 6274, 512GB RAM, 2 x
1TB SATA drives, two 10GBe NICs, and 3yr base warranty cost $20140 as of 9/24/12 at www.dell.com. HP DL560p Gen8 with four Intel Xeon processor Model E5-4620, 512GB
RAM, 2 x 1TB SATA drives, two 10GBe NICs, and 3yr base warranty cost $30948 as of 9/24/12 at www.hp.com. The 3yr TCO using AMD Opteron™-based servers is $1,811,446.
The 3yr TCO using Intel Xeon-based server is $2,170,350. Assuming customer already has agreement for client license and run time license, so not including Windows Server
license. Actual results will vary. SVR-255
24 | Cutting Big Data Down to Size | Dell World – December 2012 |
23. SUBSTANTIATION AND FOOTNOTES, CONTINUED
6. SVR-254 - The Dell R815 offers up to a 35% lower server hardware acquisition cost than the HP DL560 Gen8 for virtualization configurations. R815 with four AMD Opteron™ processor Model 6274,
512GB RAM, 2 x 1TB SATA drives, two 10GBe NICs, and 3yr base warranty cost $20140 as of 9/24/12 at www.dell.com. HP DL560 Gen8 with four Intel Xeon processor Model E5-4620, 512GB RAM, 2 x
1TB SATA drives, two 10GBe NICs, and 3yr base warranty cost $30948 as of 9/24/12 at www.hp.com. SVR-254
7. SVR-213 - Virtualized AMD-based servers deliver up to 12% higher performance and up to 23% better price performance than the competition. Testing with VMware ESX in today’s typical virtualized
environments consisting of mid-range processor and memory configurations, reveals that AMD Opteron processors lead in both price as well as price/performance versus Intel-based systems.
Virtualized AMD-based servers deliver up to 12% higher performance and up to 23% better price performance than the competition. Test run VMware ESX with DVD Store, an open source
performance tool that includes web and database servers to emulate an ecommerce environment.. Servers running increasing number of VMs from 2 to 32 with a fixed, utilization rate of ~25% .
Systems include: 2X AMD Opteron™ processors Model 6274 in HP DL385 Gen8 server running 4,233 Operations per Minute (OPM) for 2 VMs to 1,242 OPM for 32 VMs with 96GB (12x8 GB DDR3-
1333) memory, 1x15K 500GB SAS HD, base warranty and costing $7076 for the hardware and $6990 for VMware vSphere Enterprise Plus software; 2 x Intel Xeon E5-2665 processors in a HP DL380p
Gen 8 server running 3,769 OPM for 2 VMs to 1,178 OPM for 32 VMs, 96GB (12x8 GB DDR3-1333) memory, 1x15K 500GB SAS HD, base warranty and costing $9,338 for hardware and $6990 for
VMware vSphere Enterprise Plus software . System prices as of 7/5/2012 http://www.hp.com and software as of 7/5/2012 http://vmware.com. SVR-213
8. SVR-303 - The Dell C6145 lowers TCO for VDI by up to 28% versus the HP SL230s Gen8. For the Dell C6145, the TCO calculation is based on a VDI scenario with 8000 users, 200 VDI sessions per server,
and 2.5GB physical RAM per session, which is supported by one rack of (20) two-node 4P servers and two Mellanox SX1016 10GBe 64-port switches. For the HP SL230s Gen8, the TCO calculation is
based on a VDI scenario with 8000 users, 100 VDI sessions per server, and 2.5GB physical RAM per session, which is supported by one rack of (80) 2P servers and two Mellanox SX1016 10GBe 64-
port switches. The cost per rack with installation is $3,000 based on http://www.intel.com/technology/eep/datacenter.pdf as of 9/10/12 and (2) Mellanox SX1016 10GBe 64-port switches cost
$10,990 each and have a typical power consumption of 113 watts as of 9/10/12 at http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1377. Power costs based on $0.10 per kWhr and
using 1650 watts per for the Dell C6145 server and 417 watts for the HP SL230s server plus 182 watts for the C6500 enclosure according to the Dell Energy Smart Solution Advisor (ESSA) tool with a
memory-intensive workload set at 100% CPU loading and an input voltage of 220 volts as well as the HP Power Advisor Tool set at 100% CPU loading and an input voltage of 220 volts as of 9/26/12
based on the configurations below. The PUE is 1.8. Assuming 28 square feet per rack according to
http://www.energystar.gov/ia/partners/prod_development/downloads/Additional_FAQs_LightingCooling_Racks.pdf as of 9/11/12, and the average fully burdened cost of real estate at $310 per
square foot per year based on http://www.vmware.com/technology/whyvmware/calculator/index.php as of 9/10/12. Management overhead yearly cost per server based on $81k salary with 1.25
multiplier for a fully loaded expense (benefits, taxes, etc) of $101,250. $1315 per server derived from dividing the fully loaded expense by 77 servers being managed by IT administrator, based on
page 4 at http://www.vmware.com/files/pdf/vmware-solution-opex-reducing-opex-wp-en.pdf and http://www.indeed.com/salary/Windows-Server-Administrator.html. XenDesktop VDI software
costs $134.75 per VDI session as of 9/10/12 provided by Sales at Insight. Dell C6145 with two server nodes of four AMD Opteron™ processor Model 6274, 512GB (32x16 1600MHz) RAM, 2 x 1TB
SATA drives, one 10GBe NICs, and 3yr 24x7 warranty cost $43978 as of 10/4/12 according to Dell Sales. HP SL230s Gen8 with two Intel Xeon processor Model E5-2665, 256GB (16x16 1600MHz)
RAM, 2 x 1TB SATA drives, one 10GBe NICs, and 3yr 24x7 warranty cost $19669 plus $2008 for the HP C6500 enclosure as of 10/4/12 at www.hp.com. The 3yr TCO using AMD Opteron™-based
servers is $2,323,545. The 3yr TCO using Intel Xeon-based servers is $3,205,690. Assuming customer already has agreement for client license and run time license, so not including Windows Server
license. Actual results will vary. SVR-303
25 | Cutting Big Data Down to Size | Dell World – December 2012 |
24. SUBSTANTIATION AND FOOTNOTES, CONTINUED
9. SVR-302 - The Dell C6145 offers up to a 44% lower server hardware acquisition cost than the HP SL230s Gen8 for VDI configurations. Dell C6145 with two server nodes of four
AMD Opteron™ processor Model 6274, 512GB (32x16 1600MHz) RAM, 2 x 1TB SATA drives, one 10GBe NICs, and 3yr 24x7 warranty cost $43978 as of 10/4/12 according to
Dell Sales. HP SL230s Gen8 with two Intel Xeon processor Model E5-2665, 256GB (16x16 1600MHz) RAM, 2 x 1TB SATA drives, one 10GBe NICs, and 3yr 24x7 warranty cost
$19669 as of 10/4/12 at www.hp.com. SVR-302
10. SVR-301 - The Dell C6105 lowers TCO for Hadoop by 32% versus the HP DL360e Gen8. TCO calculation based on Hadoop scenario with 24TB dataset being analyzed by 10
users and being processed by an I/O bound application. This is supported by one rack of 79 Dell C6105 slave nodes and 1 Dell C6105 master node along with two Dell
Force10 S60 High-Performance 1/10 GbE switches as well as four racks of 79 HP DL360e Gen8 slave nodes and 1 HP DL360e Gen8 master node along with two Mellanox
SX1016 10GBe 64-port switches per rack. The cost per rack with installation is $3,000 based on http://www.intel.com/technology/eep/datacenter.pdf as of 9/10/12.
Mellanox SX1016 10GBe 64-port switches cost $10,990 each and have a typical power consumption of 113 watts as of 9/10/12 at
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1377. Dell Force10 S60 High-Performance 1/10 GbE switches cost $9171 each and have a typical power
consumption of 156 watts as of 9/26/12 at www.dell.com and http://www.networkcomputing.com/next-gen-network-tech-center/force10s-deep-buffer-switch/229501471.
Power costs based on $0.10 per kWhr and using 889 watts per for the Dell C6105 with four server nodes in one system and 338 watts for the HP DL360e Gen8 server with a
transactional workload set at 100% CPU loading according to the Dell Energy Smart Solution Advisor (ESSA) tool with a transactional workload set at 100% CPU loading and
an input voltage of 220 volts and the HP Power Advisor tool set at 100% CPU loading and an input voltage of 220 volts as of 9/26/12 based on the configurations below. The
PUE is 1.8. Assuming 28 square feet per rack according to
http://www.energystar.gov/ia/partners/prod_development/downloads/Additional_FAQs_LightingCooling_Racks.pdf as of 9/11/12, and the average fully burdened cost of
real estate at $310 per square foot per year based on http://www.vmware.com/technology/whyvmware/calculator/index.php as of 9/10/12. Management overhead yearly
cost per server based on $81k salary with 1.25 multiplier for a fully loaded expense (benefits, taxes, etc) of $101,250. $1315 per server node derived from dividing the fully
loaded expense by 77 servers being managed by IT administrator, based on page 4 at http://www.vmware.com/files/pdf/vmware-solution-opex-reducing-opex-wp-en.pdf
and http://www.indeed.com/salary/Windows-Server-Administrator.html. Open Source software is being used so there is no cost. Each of the four Dell C6105 server nodes
has two AMD Opteron™ processor Model 4274HE, 64GB RAM, 3 x 2TB SATA drives, 1GBe, and 3yr base warranty with a total system cost of $20075 ($5019 each node) as of
10/3/12 from Dell Sales. HP DL360e with two Intel Xeon processor Model E5-2440, 64GB RAM, 3 x 2TB SATA drives, 1GBe, and 3yr base warranty cost $7564 as of 9/21/12
at www.hp.com. The 3yr TCO using AMD Opteron™-based servers is $849,643. The 3yr TCO using Intel Xeon-based servers is $1,256,972. Actual results will vary. SVR-301.
11. SVR-300 - The Dell C6105 offers up to a 34% lower scale-out server hardware acquistion cost than the HP DL360e Gen8 for Hadoop configurations. Each of the four Dell C6105
server nodes has two AMD Opteron™ processor Model 4274HE, 64GB RAM, 3 x 2TB SATA drives, 1GBe, and 3yr base warranty with a total system cost of $20075 ($5019 each
node) as of 10/3/12 from Dell Sales. HP DL360e with two Intel Xeon processor Model E5-2440, 64GB RAM, 3 x 2TB SATA drives, 1GBe, and 3yr base warranty cost $7564 as
of 9/21/12 at www.hp.com. SVR-300
26 | Cutting Big Data Down to Size | Dell World – December 2012 |