A short introduction to Big Data; what it is and what are the implications for performance and capacity management.

1. TRACK 2S
S7 - Big Data
Performance and Capacity Management
Paul Seaton-Smith
RightSize Solutions Limited
www.rightsizesolutions.co.uk
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 1

2. Agenda
 What is Big Data?
 Big Data technologies
 Implications for capacity management
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 2

3. What is Big Data?
 Exceeds the processing capacity of legacy databases
 The data is too big, arrives too fast, or can’t be stored in existing
database architectures
 To leverage this data, we need new ways to store and analyse it
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 3

4. Volume, Velocity, Variety
 Big data spans three dimensions
 Volume
 Velocity
 Variety
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 4

5. Volume
 Every day 2.5 Exabytes (1018 bytes) of data are created
 90% of the world’s data has been created in the last two years
 Historically much of that data has not been able to be processed
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 5

6. Velocity
 Streaming data
 Less sampling or aggregation (leading to greater Volume)
 Big data may be used as soon as it is collected as results are often
time-sensitive
 Real-time or near-time feedback
 Maximize the value to the business
 E.g. Sales data that may have been collated and analysed on a
monthly basis might now be available every hour
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 6

7. Variety
 Unstructured data of all varieties that is hard to store in databases
 Text  Social network feeds
 Audio (e.g. MP3, radio)  Vehicle telemetry
 Video (e.g. CCTV, YouTube)  Financial market data
 Click streams  Web page content
 Log files (e.g. web logs)  GPS trails
 Images (e.g. satellite images)  etc.
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 7

8. Why do we need Big Data?
 Previously 80% of corporate information was stored on paper
 20% was kept in electronic form
 At least 80% of that was held in databases
 Now 80% of corporate information is in electronic form (Volume)
 At least 80% of that is not in a database (Variety)
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 8

9. What is driving Big Data?
 More data is stored as storage prices drop
 Over the last 30 years, storage space per unit cost has doubled
roughly every 14 months
 Commodity hardware, cloud architectures and open source software
 Marketing from large vendors (EMC, IBM, Oracle)
 Change of attitude towards leveraging data
 Tackle complex problems that previously could not be solved
 Monetise your business data
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 9

10. Examples
 Financial services could examine data sources to determine likely
potential sources of financial fraud
 Credit and transaction history
 Social networking behaviour
 Demographics
 Voice recordings
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 10

11. Examples
 Analyse Tweets to understand public sentiment for a new product
 Monitor power meter readings to better predict consumption
 Examine service desk call detail records in real-time to predict
customer churn faster
 Analyse real time traffic information to change traffic lights and ease
congestion
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 11

12. Technology
 Apache Hadoop
 MapReduce
 HDFS
 Further Apache projects to support Hadoop
 3rd party enterprise class distributions built on Hadoop
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 12

13. Apache Hadoop
 Distributed processing of large data sets across clusters of
computers
 Scale to thousands of machines, each offering local computation
and storage
 Designed to detect and handle failures at the application layer
 Delivers a highly-available service on top of a cluster of computers,
each of which may be prone to failures
 Based on MapReduce and HDFS
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 13

14. MapReduce
 MapReduce is a system for distributing computation
 Created by Google in response to the problem of creating web
search indexes
 Take a query over a dataset, divide it, and run it in parallel over
multiple nodes
 Processes and analyses any data type across clusters of
commodity servers
 Distributing the computation solves the problem of having data too
large to fit onto a single node
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 14

15. HDFS
 A file system that spans all the nodes in a cluster for data storage
 Data in a cluster is broken down into smaller blocks and distributed
throughout the cluster
 The map and reduce functions can be ex-ecuted on smaller subsets
of the original data sets
 HDFS links together the file systems on many local nodes to make
them into one big file system
 HDFS assumes nodes will fail, so it achieves reliability by replicating
data across multiple nodes
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 15

16. Apache Hadoop
 Apache Hadoop is an open source MapReduce implementation with
HDFS
 Originally funded by Yahoo
 Hadoop is supplemented by further Apache components to enhance
its usability and functionality
 3rd party distributions have additional functionality and management
tools
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 16

17. NoSQL Database Management Systems
 Do not have to use SQL
 May not guarantee atomicity, consistency, isolation, durability
 Distributed, fault-tolerant architecture
 Various types
 Document store databases
 Graph databases
 Key-value stores
 BigTable implementations
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 17

18. Apache HBase
 HBase is the NoSQL Hadoop database
 Runs on top of HDFS
 Provides BigTable-like support for Hadoop
 Provides random, real-time read/write access to data in Hadoop
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 18

19. Apache Hive
 Built on top of Hadoop enabling it to operate as a data warehouse
 Superimposes structure on data in HDFS on a variety of data
formats
 Enables ad-hoc analytical queries over the data using a SQL-like
syntax
 Tools to enable easy data extract/transform/load
 Best used for batch jobs over large sets of append-only data (such
as web logs)
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 19

20. Further Apache projects
 ZooKeeper is a centralised service for maintaining configuration
information, naming, distributed synchronisation, group services
 Pig is a programming language that simplifies the common tasks of
working with Hadoop
 Sqoop is a tool designed to import data from relational databases
into Hadoop, either directly into HDFS or into Hive
 Flume is designed to import streaming flows of log data directly into
HDFS
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 20

21. Benefits of Hadoop
 Scalability
 Add new nodes as needed
 Add nodes without needing to change data formats, how data is
loaded, how jobs are written, or the applications on top
 Value for money
 Massively parallel computing on commodity servers
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 21

22. Benefits of Hadoop
 Flexibility
 Schema-less
 Can absorb any type of data, structured or not, from any
number of sources
 Fault tolerance
 If you lose a node, the system redirects work to another location
of the data and continues processing
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 22

23. Commercial Hadoop Distributions
 EMC Greenplum HD
 http://www.greenplum.com/products/greenplum-hd/
 Microsoft Big Data solution
 http://www.microsoft.com/sqlserver/en/us/solutions-
technologies/business-intelligence/big-data-solution.aspx
 Oracle Big Data Appliance
 http://www.oracle.com/us/products/database/big-data-
appliance/overview/index.html
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 23

24. Commercial Hadoop Distributions
 Cloudera CDH
 http://www.cloudera.com/hadoop/
 IBM InfoSphere BigInsights
 http://www-01.ibm.com/software/data/infosphere/biginsights/
 Amazon Elastic MapReduce (Amazon EMR)
 http://aws.amazon.com/elasticmapreduce/
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 24

25. Capacity Management Challenges
 Hard to forecast what new data may be available in the future
 Social networks
 Social video
 Social media
 Social location
 Augmented reality
 What next?
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 25

26. Capacity Management Challenges
 Forecasting
 Will data growth of existing types continue exponentially
 What new data types will emerge
 Monitoring
 Cost of agent-based licencing model for monitoring tools
 Lack of standard Hadoop management tools
 Response times
 Velocity means results are time-critical
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 26

27. Capacity Management Challenges
 Technology
 Understanding new technologies, which are still emerging
 Experienced staff principally come from US companies (Google,
Yahoo, Facebook etc.)
 Crowded marketplace means harder to build expertise
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 27

28. Contact
 Email
 paul.seatonsmith@rightsizesolutions.co.uk
 Website
 www.rightsizesolutions.co.uk
 LinkedIn
 http://linkd.in/seatonsmith
 LinkedIn Group
 http://linkd.in/capacitymanagement
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 28

29. References
 http://radar.oreilly.com/2012/01/what-is-big-data.html
 http://radar.oreilly.com/2012/02/what-is-apache-hadoop.html
 http://radar.oreilly.com/2012/01/big-data-ecosystem.html
 http://www.theregister.co.uk/2012/01/16/big_data_study/
 http://www.mkomo.com/cost-per-gigabyte
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 29

30. References
 http://www-01.ibm.com/software/data/bigdata/
 http://en.wikipedia.org/wiki/NoSQL
 http://www.techrepublic.com/blog/cio-insights/big-data-cheat-
sheet/39748353
 http://jimbojw.com/wiki/index.php?title=Understanding_Hbase_and_
BigTable
Copyright © 2012 RightSize Solutions Limited. All rights reserved. 30