As one of Big Data’s Founding Fathers, Google explored the technological changes we faced over the past 10 years and present their solutions to the new data challenges within the Google Cloud ecosystem
5. Confidential & ProprietaryGoogle Cloud Platform 5
Everything You Need To Build And Scale
Compute
From virtual machines with
proven price/performance
advantages to a fully managed
app development platform.
Compute Engine
App Engine
Container Engine
Container Registry
Cloud Functions
Storage and Databases
Scalable, resilient, high
performance object storage and
databases for your applications.
Cloud Storage
Cloud Bigtable
Cloud Datastore
Cloud SQL
Networking
State-of-the-art software-defined
networking products on Google’s
private fiber network.
Cloud Virtual Network
Cloud Load Balancing
Cloud CDN
Cloud Interconnect
Cloud DNS
Management Tools
Monitoring, logging, and diagnostics
and more, all a easy to use web
management console or mobile app.
Stackdriver Overview
Monitoring
Logging
Error Reporting
Debugger
Deployment Manager & More
Big Data
Fully managed data warehousing,
batch and stream processing, data
exploration, Hadoop/Spark, and
reliable messaging.
BigQuery
Cloud Dataflow
Cloud Dataproc
Cloud Datalab
Cloud Pub/Sub
Genomics
Machine Learning
Fast, scalable, easy to use ML
services. Use our pre-trained models
or train custom models on your data.
Cloud Machine Learning Platform
Vision API
Speech API
Translate API
Developer Tools
Develop and deploy your applications
using our command-line interface and
other developer tools.
Cloud SDK
Deployment Manager
Cloud Source Repositories
Cloud Endpoints
Cloud Tools for Android Studio
Cloud Tools for IntelliJ
Google Plugin for Eclipse
Cloud Test Lab
Identity & Security
Control access and visibility to
resources running on a platform
protected by Google’s security model.
Cloud IAM
Cloud Resource Manager
Cloud Security Scanner
Cloud Platform Security Overview
6.
7. Hitting the limits, early on...
The Anatomy of a Large-Scale Hypertextual Web Search Engine
1996, Sergey Brin and Lawrence Page
Computer Science Department, Stanford University, Stanford,
CA 94305
8. Building on Google’s infrastructure
2.8 million
devices activated
Every Day (1.1 billion devices)
10 billion
hours watched
Every Month (100 hours of
new content uploaded
every minute)
43 billion
pages crawled
Every Day
16. Original Launch Target Estimated Worst Case Actual Traffic
Cloud Datastore Transactions Per Second
50X
Actual Traffic
5X
Worst Case
Estimate
1X
Target Traffic
20. Leading Open Source Communities
#1Highest
Engagement
on Github
#2
Highest
Engagement
on Github
Kubernetes Tensorflow
Source: Analyzing GitHub issues and comments with BigQuery
30. Dataproc: Fully managed Hadoop and
Spark w/ industry-leading performance
BigQuery: Fully managed data
warehouse for large-scale analytics
Dataflow: Real-time data pipelines, with
open source SDK via Apache Beam
Big Data with Google
30
32. “Right at the start of the partnership we were able to
reduce time to insight from 96 hours to 30 minutes by using BigQuery”
Gary Sanders
Head of Digital Analytics
33. Music for Everyone
75M+ Users
2B+ Playlists
30M+ Songs
Data is the center
of the Spotify
music experience
With GCP, data
teams get big data
insights in minutes
versus hours
34. “From traditional batch processing to rock-solid event delivery to the nearly
magical abilities of BigQuery, building on Google’s data infrastructure
provides us with a significant advantage where it matters the most.”
Nicholas Harteau
VP of Engineering and Infrastructure
39. Rapidly Accelerating Use of Deep Learning at Google
Number of directories containing model description files
2012 2013 2014 2015
1500
1000
500
0
Used across products:
45. Faces
Faces, facial landmarks, emotions
OCR
Read and extract text, with
support for > 80 languages
Cloud Vision API
Call API from anywhere, with support for embeddable images, and Google Cloud storage
Label
Detect entities from furniture to
transportation
Logos
Identify product logos
Landmarks & Image Properties
Detect landmarks & dominant
color of image
Safe Search
Detect explicit content - adult,
violent, medical and spoof
46. API Usage: Detect Objects in an Image
Image Detected
Items
Vision API
Upload Images
Cloud/On premise
Process
the response
Call the
REST API1 2 3