GDG Heraklion - Architecting for the Google Cloud Platform

Architecting for the
Google Cloud Platform
Márton Kodok / @martonkodok
Google Developer Expert at REEA.net
June 2018 - Heraklion, Greece

● Geek. Hiker. Do-er.
● Among the Top3 romanians on Stackoverflow 120k reputation
● Google Developer Expert on Cloud technologies
● Crafting Web/Mobile backends at REEA.net
● BigQuery/Redis and database engine expert
● Active in mentoring and IT community
Twitter: @martonkodok
StackOverflow: pentium10
Slideshare: martonkodok
GitHub: pentium10
Architecting for the Google Cloud Platform @martonkodok
About me

REEA.net uses GCP
Build on the same infrastructure
that powers Google

Google Cloud Platform (GCP)
Compute Big Data
BigQuery
Cloud
Dataflow
Cloud
Dataproc
Cloud
Datalab
Cloud
Pub/Sub
Genomics
Storage & Databases
Cloud
Storage
Cloud
Bigtable
Cloud
Datastore
Cloud SQL
Cloud
Spanner
Persistent
Disk
Machine Learning
Cloud Machine
Learning
Cloud
Vision API
Cloud
Speech API
Cloud Natural
Language API
Cloud
Translation
API
Cloud
Jobs API
Data
Studio
Cloud
Dataprep
Cloud Video
Intelligence
API
Advanced
Solutions Lab
Compute
Engine
App
Engine
Kubernetes
Engine
GPU
Cloud
Functions
Container-
Optimized OS
Identity & Security
Cloud IAM
Cloud Resource
Manager
Cloud Security
Scanner
Key
Management
Service
BeyondCorp
Data Loss
Prevention API
Identity-Aware
Proxy
Security Key
Enforcement
Internet of Things
Cloud IoT
Core
Transfer
Appliance

Google Cloud Platform (GCP)
Developer Tools
Cloud SDK
Cloud
Deployment
Manager
Cloud Source
Repositories
Cloud
Tools for
Android Studio
Cloud Tools
for IntelliJ
Cloud
Tools for
PowerShell
Cloud
Tools for
Visual Studio
Container
Registry
Google Plug-in
for Eclipse
Cloud Test
Lab
Networking
Virtual
Private Cloud
Cloud Load
Balancing
Cloud
CDN
Cloud
Interconnect
Cloud DNS
Cloud
Network
Cloud
External IP
Addresses
Cloud
Firewall Rules
Cloud
Routes
Cloud VPN
Management Tools
Stackdriver Monitoring Logging
Error
Reporting
Trace
Debugger
Cloud
Deployment
Manager
Cloud
Endpoints
Cloud
Console
Cloud
Shell
Cloud Mobile
App
Cloud
Billing API
Cloud
APIs
Cloud
Router
Dedicated
Interconnect
Container
Builder

1. Virtual Machines
2. Autoscaling - Design for resiliency, scalability, and disaster recovery
3. Serverless
4. Data layer design - Identification of storage needs and mapping
5. Microservices architecture
6. Event sourcing - streaming
7. Cloud Functions
8. Task Queues - Cloud Pub/Sub - DataFlow
9. Data warehouse in the Cloud
10. Practical use cases
Agenda

Virtual Machines (VMs)
Machine Type Options
- Standard machine
- Shared-core machine (F1)
- Shared-core machine (G1)
- High-CPU machine
- High-memory machine
- Custom Virtual Machine
Disk options
PD - Persistent Disk
PD-SSD - Persistent SSD Disk
Local SSD - Local SSD Disk
Standard
machine
High-CPU
machine
High-memory
machine
Shared-core
machine
Machine type
Number of cores
GB of RAM
Custom Virtual
Machine
Shared-core
machine
Autoscaling Preemptable
Virtual Machine

General use > Compute Engine and REST
iOS
Android
Web
Mobile Services REST API
Compute Engine
Flexibility for Every Workload
● High-Performance, Scalable VMs
● Custom Machine Types
● Persistent Disks or SSD (local or
network based)
● Per-second billing
● Networking
● Firewall
● Load balancing
● Compliance & Security

Creating a VM instance steps

Creating a VM instance

Creating a VM instance - Choosing Region, Instance type

Creating a VM instance - OS, boot disk, Network

VM instances

Virtual Private Network - Firewall rules

Preemptible - spot instances
● Much lower price than normal instances
● For fault-tolerant applications
● Can withstand possible instance losses
● Runs for maximum 24 hours
● Might not be always available
● Instance templates
Good for: Nodes (eg: Elasticsearch)
Batch jobs
Worker async tasks
Pricing: ranges 3-4 times lower price

Cloud Endpoints

Cloud Endpoints: Develop, deploy and manage APIs
App
Engine
Mobile Backend API
Cloud Endpoints
Android
Web
Endpoint
Clients
iOS
Manages and protects your APIs
● Works with JSON web tokens, API
keys, integration with Auth0,
Firebase Auth
● Provides insight with Stackdriver
Monitoring, Trace and Logging
● Less than 1ms per call
● Works best with Compute engine,
App Engine, and Kubernetes Engine.
● JSON/Rest/gRPC
Pricing: 0-2M $0.00 2M+ req $3/million

Autoscaling
with Instance Groups

Creating an Instance Group

Setting Autoscaling settings

Example Architecture for Autoscaling

CMS with Autoscaling
ZONE B
Content Server
Compute Engine
Autoscaling
ZONE A
Content Server
Compute Engine
Autoscaling
Static Content
Cloud Storage
Dynamic Content
Cloud SQL
Publisher
Cloud Load
Balancing
Cloud
DNS

DB multiple zones and Master/Slave topology
Zone 1
us-central 1-a
Instance Group
Compute Engine
Web Applications
Master
Cloud SQL
Zone 2
us-central 1-f
Instance Group
Compute Engine
Web Applications
Read Replica
Cloud SQL
ReplicationCloud Load
Balancing

Data layer design

Database Mapping

Serverless

Meet Serverless

Meet Serverless
serverless data center depicted

Serverless is about maximizing elasticity, cost
savings, and agility of cloud computing.
@martonkodok

Serverless types
Platforms Triggered Code

App Engine - Websites > API Hosting
API
Compute Engine
Virtual Machines
API
App Engine
Cloud Endpoints
API
Container Engine
Kubernetes Services
Multiple
Clients
API Requests
Cloud Load
Balancing
● Fully managed serverless application
platform
● Scale your applications seamlessly from
zero to planet scale
● Automatically scales depending on your
application traffic
● Traffic Splitting (app versions, A/B tests,
incremental rollouts)
● Standard vs Flexible environments
Python, Java, Node.js, Go, Ruby, PHP, .NET

App Engine
Instance startup time Minutes Seconds
Maximum request timeout 60 minutes 60 seconds
Background threads Yes Yes, with restrictions
SSH debugging Yes No
Writing to local disk Yes, ephemeral (disk initialized on VM
startup)
No
Modifying the runtime Yes (through Dockerfile) No
Supports installing third-party binaries Yes No
Pricing Based on usage of vCPU, memory, and
persistent disks
Based on instance hours
Flexible Standard

Cloud Functions
On-Premises Servers
ApplicationEvent Sourcing
Frontend
Platform Services
Metrics / Logs/
Streaming
Event
Triggered
Cloud
Functions
Triggered Code
Result

● No servers to provision or manage
● Abstract away the complexity
● Scales with usage (ready every time for viral spikes or #BlackFriday)
● Availability and fault tolerance built in
● No orchestration in code
● Never pay for idle
● Cost savings (ps: we don’t have the same budget for security like GCP or AWS)
● Decoupled: APIs as contracts
● Monitored: Metrics and logging are a universal right
● Think concurrent, stateless, queue, stream based.
Serverlessmeans

Serverless architecture through examples

Email Delivery + Webhook response

Event-driven serverless compute platform
Cloud
Services
Changes in data state
Business logic events
Integrations
Event Router
Gateway
HTTPS
Event Source
Multiple Platforms
Cloud Functions
Task Queues | Pub/Sub
Data Warehouse
Event
Business Value
Application
Task
Analysis

Message Queue systems: Task Queue
Cloud Pub/Sub

Task Queue - Message Queue systems
Static Content
Cloud Storage
Dynamic Content
Cloud SQL
Dynamic Content
Cloud Datastore
Front End App
App Engine
Autoscaling
Task
Queues
Memcache
Batch App
App Engine
Autoscaling
Cloud Load
Balancing
Cloud
DNS
Workers
Compute Engine
Autoscaling

Task Queues
Slides: https://www.slideshare.net/martonkodok
Title: Architectural Patterns - Message Queues
Push queues
● Put with delays
Pull queues
● Ability to “tag”
● Lease multiple eg: gameboard
updated, game id as tag.

Reporting & Analytics moved to the Cloud

Legacy Reporting System
App
Cloud Load
Balancing
NGINX
Compute Engine
10GB PD
2 1
Database Service (Master/Slave)
Compute Engine
10GB PD
4 1
Compute Engine
10GB PD
4 1
Compute Engine
10GB PD
4 1
Report & Share
Business Analysis
Scheduled
Tasks
Batch Processing
Compute Engine
Multiple Instances

Serverless Reporting System
App
Cloud Load
Balancing
NGINX
Compute Engine
10GB PD
2 1
Database Service (Master/Slave)
Compute Engine
10GB PD
4 1
Compute Engine
10GB PD
4 1
Compute Engine
10GB PD
4 1
Report & Share
Business Analysis
Scheduled
Tasks
Batch Processing
Compute Engine
Multiple Instances
BigQuery Data Studio
Report & Share
Business Analysis

Analytics-as-a-Service - Data Warehouse in the Cloud
Scales into Petabytes on Managed Google Infrastructure (US or EU zone)
SQL 2011 + Javascript UDF (User Defined Functions)
Familiar DB Structure (table, views, struct, nested, JSON)
Integrates with Google Sheets + Cloud Storage + Pub/Sub connectors
Decent pricing (queries $5/TB, storage: $20/TB cold: $10/TB) *June 2018
Open Interfaces (Web UI, BQ command line tool, REST, ODBC)
What is BigQuery?

Columnar storage (max 10 000 columns in table)
Large files for loading: 5TB (CSV or JSON)
UDF in Javascript or SQL
Append-only tables prefered (DML syntax available)
Day column partitioned tables (select * from t where day=’2018-01-01’)
Rich SQL 2011: JSON,IP,Math,RegExp,Geocode,Window functions
Modern data types: Record, Nested, Struct, Array
BigQuery: Convenience of SQL

“ Data needs to be processed in
multiple services.
How can we pipe to multiple places?

Architecting for The Cloud
On-Premises Servers
Event Sourcing
Frontend
Platform Services
Analyze
Metrics / Logs/
Streaming
Cloud Storage
Cloud
Dataflow
Process
BigQuery
Cloud SQL
Stream
Batch
Data
Studio
Third-Party
Tools

“ We have our app outside of GCP.
How can we use the benefits of BigQuery?

Data Pipeline Integration at REEA.net
Analytics Backend
BigQuery
On-Premises Servers
Pipelines
FluentD
Event Sourcing
Frontend
Platform Services
Metrics / Logs/
Streaming
Development
Team
Data Analysts
Report & Share
Business Analysis
Tools
Tableau
QlikView
Data Studio
Internal
Dashboard
Database
SQL
Application
ServersServers
Cloud Storage
archive
Load
Export
Replay
Standard
Devices
HTTPS

● SQL language to run BigData queries
● run raw ad-hoc queries (either by analysts/sales or Devs)
● no more throwing away-, expiring-, aggregating old data
● no provisioning/deploy
● no running out of resources
● no more focus on large scale execution plan
Our benefits using BigQuery

Architecture: Retail > Real-Time Inventory
Ingest
App Engine
Capture Changes
Events
Cloud Bigtable
Processed Changes
Aggregates
Cloud SQL
Inventory Counts
Analytics
BigQuery
Data Warehouse
Messaging
Cloud Pub/Sub
Streaming Changes
Processing
Cloud Dataflow
Inventory Streams
Services
App Engine
Inventory APIs
Backoffice
Business Applications
Integrations
Google Services
Retail Store
Inventory Changes

Architecture: Retail > Recommendation Engines
Data Sources
Machine Learning
Applications
Browsing
Compute Engine
Shopping Cart
App Engine
Outreach
Container Engine
Purchases
Cloud Datastore
Wish Lists
Cloud SQL
Reviews
Cloud Datastore
Inventory Data
Cloud SQL
Analytics
BigQuery
Spark MLlib
Cloud Dataproc
ETL
Cloud Dataflow
Transform Data
Normalized Data
Cloud Storage
Combined Sources
Hosted Models
Cloud Machine Learning
Real-Time
Prediction API

Thank you.
Slides available on: slideshare.net/martonkodok
Reea.net - Integrated web solutions driven by creativity to deliver
projects.

GDG Heraklion - Architecting for the Google Cloud Platform

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie GDG Heraklion - Architecting for the Google Cloud Platform

Ähnlich wie GDG Heraklion - Architecting for the Google Cloud Platform (20)

Mehr von Márton Kodok

Mehr von Márton Kodok (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

GDG Heraklion - Architecting for the Google Cloud Platform