Best of re:Invent 2016 meetup presentation

Best of re:Invent 2016| Jan. 2017
Best of
2016
Lahav Savir
Co-founder and CTO
AllCloud

Agenda
● Intro - Lahav Savir
● CI / CD News - Johanan Liebermann
● Serverless News - Yakir Eliyahu
● Security News - Hemed Gur Ary
● Data News - Shay Gury
● Q&A Panel - AllCloud’ers
Pizza &
Beer
allowed at all
tim
es

A Global Leader in Cloud
Transformation and Adoption
with expertise across the cloud
stack, Infrastructure, Platform,
and Software-as-a-Service.

The AllCloud Solutions Framework
Infrastructure-as-a-Service
Platform-as-a-Service
Software-as-a-Service
Solutions Partners
Enablement
Advisory

“AWS Managed Service Partners
are skilled at cloud infrastructure
and application migration, and
deliver value to customers by
offering proactive monitoring,
automation, and management of
their customer’s environment.”
https://aws.amazon.com/partners/msp/
http://www.emind.co/msp
AWS Next-Gen (v3) Managed Service Partner (MSP)

www.allcloud.io

Democracy of Infrastructure and Services
Infrastructure-as-a-Service
Platform-as-a-Service
Software-as-a-Service

www.allcloud.io
● 32,000 Attendees
● +400 Technical Sessions
● One-of-a-kind re:Play party
Largest Cloud
Conference in History

New Services
● 22 New Services
● 22 New Features
● 5 New Instance Types
● 2 Price Reductions
● And a massive Snowmobile

www.allcloud.io
AWS completes and enhances its product pipelines
to offer fully managed services with minimal pain points.

www.allcloud.io
It’s the big player times

Where is it Heading?
16 Regions
+80 Services

CI/CD News
AWS CodeBuild, AWS OpsWorks for Chef Automate
Johanan Liebermann,
Software Developer, AllCloud

● Tiny startups can challenge giants
● Delivering a product to the masses is easy
● Release cycle takes days, not months
● Time to market can make or break your product
Today’s Software Moves Fast!

● Automate everything!
○ Source code management, builds, testing, deployment, configuration,
documentation, infrastructure...
● Operations time < Development time
● Do more with less people
● Short release cycle
How Can You Move Fast?

You Need Tools!

AWS Code* Services
AWS CodePipeline AWS CodeDeploy AWS CodeCommit

AWS Code* Services
Commit Build Test Production

AWS Code* Services
AWS CodeCommit

AWS Code* Services
Third-Party
Tooling
Third-Party
Tooling

AWS Code* Services
AWS CodeDeploy

AWS Code* Services
AWS CodePipeline

AWS Code* Services
AWS CodeCommit
Third-Party
Tooling
Third-Party
Tooling
AWS CodeDeploy
AWS CodePipeline

● Build and test code in the
cloud
● Scales continuously
● Charged by the minute
● Integrates with other
AWS services like
CodePipeline to automate
building and testing code
AWS CodeBuild

Deploying Code is Not Enough!

● Configuration
management service
● Fully-managed Chef
servers on AWS
● Includes premium
features of Chef
Automate
OpsWorks for
Chef Automate

Serverless News
AWS Step Functions
Yakir Eliyahu,
Cloud Architect, AllCloud

What is
Serverless?
Build and run applications
without thinking about servers

Let’s take a look at the evolution of computing
Physical servers
in datacenters
Virtual servers
in datacenters
Virtual servers
in the cloud

Each progressive step was better
Physical Servers
Datacenters
Virtual Servers
Datacenters
• Higher utilization
• Faster provisioning speed
• Improved uptime
• Disaster recovery
• Hardware independence
• Trade CAPEX for OPEX
• More scale
• Elastic resources
• Faster speed and agility
• Reduced maintenance
• Better availability and fault
tolerance
Virtual servers
in the cloud

But there are still limitations
Physical Servers
Datacenters
Virtual Servers
Datacenters
• Trade CAPEX for OPEX
• More scale
• Elastic resources
• Faster speed and agility
• Reduced maintenance
• Better availability and fault
tolerance
• Still need to administer
virtual servers
• Still need to manage
capacity and utilization
• Still need to size
workloads
• Still need to manage
availability, fault tolerance
• Still expensive to run
intermittent jobs
Virtual servers
in the cloud

Evolving
to Serverless
SERVERLESS
Virtual servers
in the cloud
Physical servers
in datacenters
Virtual servers
in datacenters

No server is easier to manage than no server
All of these responsibilities
go away
Provisioning and utilization
Availability and fault tolerance
Scaling
Operations and management

EVENT DRIVEN CONTINUOUS SCALING PAY BY USAGE
Deliver on demand, never pay for idle

Building blocks for serverless applications
AWS Lambda Amazon DynamoDB
Amazon SNS
Amazon API
Gateway
Amazon SQS
Amazon
Kinesis
Amazon S3
Orchestration and State Management
API Proxy Messaging and Queues Analytics
Monitoring and Debugging
Compute Storage Database
AWS X-Ray
AWS Step
Functions

Serverless changes how you deliver
Speeds up
time to market
Dedicated time
to innovation
Increases developer
productivity
Eliminates operational
complexity

Orchestration and state management

AWS Step
Functions
● Utilizes AWS Lambda which manages
the operations and underlying
infrastructure to help ensure your
application is available at any scale
● Easy to coordinate the components of
distributed applications and
microservices using visual workflows
● Building applications from individual
components that each perform a
discrete function lets you scale and
change applications quickly
● Makes it simple to build and run
multi-step applications
● Can use an Activity (process on a
VM/Contianer) for long-running tasks

1. A concept used by
CompSci profs for
torturing undergrads,
full of arcane math
2. A practical way to build
and manage modern
serverless apps
“State machine” (noun)

“I want to sequence functions”
“I want to call functions based on data”
“I want more control over retries”
“I want try/catch/finally”
“I have a workflow that runs for hours”
“I want to run functions in parallel”
Orchestration and state management

Introducing AWS Step Functions
Run cloud state machines
Coordinate components of
multi-step apps
Visualize application as a series of
steps
Handles thousands of workflows
and millions of simultaneous steps

Define your Application Visually as a Series of Steps

Verify Everything is Operating as Intended

Run and Scale Applications Reliably

Security News
AWS Shield
Hemed Gur Ary,
Security DevOps Engineer, AllCloud

AWS Shield
● Managed DDoS
Protection that
safeguards web
applications running on
AWS
● Provides always-on
detection and automatic
inline mitigations that
minimize application
downtime and latency

What is DDoS?
Distributed Denial Of Service

DDoS attack trends
65%
Volumetric
17%
State exhaustion
18%
Application layer

Examples of DDoS attacks
HTTP GET flood
TCP SYN flood
UDP reflection attacks

Challenges in mitigating DDoS attacks

Difficult to enable
Complex set-up Provision bandwidth
capacity
Application re-architecture

Manual involvement
Operator involvement to
initiate mitigation
Re-route traffic via distant
scrubbing location
Increased time to
mitigate
Traditional
Datacenter

Traffic re-routing = Increased latency for users
Traditional
Datacenter

Expensive to use

AWS approach to DDoS protection

DDoS protections built into AWS
✓ Protection against most common
infrastructure attacks
✓ SYN/ACK Floods, UDP Floods,
Reflection attacks etc.
✓ No additional cost
DDoS mitigation
systems
DDoS Attack
Users

AWS Shield
AWS Integration
DDoS protection
without infrastructure
changes
Affordable
Don’t force unnecessary
trade-offs between cost and
availability
Flexible
Customize protections
for your applications
Always-On Detection
and Mitigation
Minimize impact on application
latency
Four key pillars…

AWS Shield
A Managed DDoS Protection Service

AWS Shield
Standard Protection Advanced Protection
Available to ALL AWS customers at
No Additional Cost
Paid service that provides additional
protections, features and benefits.

AWS Shield Standard
Layer 3/4 protection
✓ Automatic detection & mitigation
✓ Protection from most common
attacks (SYN/UDP Floods, Reflection
Attacks, etc.)
✓ Built into AWS services
Layer 7 protection
✓ AWS WAF for Layer 7 DDoS attack
mitigation
✓ Self-service & pay-as-you-go

AWS Shield Standard
Better protection than ever for your applications running on AWS
• Improved mitigations using proprietary BlackWatch systems
• Additional mitigation capacity
• Commitment to continuously improve detection and mitigation
• Still at no additional cost

AWS Shield Advanced
Managed DDoS Protection

AWS Shield Advanced
Application Load Balancer Classic Load Balancer Amazon CloudFront Amazon Route 53
Available today on …

AWS Shield Advanced
Available today in …
US East (N. Virginia) us-east-1
US West (Oregon) us-west-2
EU (Ireland) eu-west-1
Asia Pacific (Tokyo) ap-northeast-1

AWS Shield Advanced
Always-on monitoring &
detection
Advanced L3/4 & L7 DDoS
protection
Attack notification and
reporting
24x7 access to DDoS
Response Team
AWS bill protection

Always-on monitoring and detection
Network flow monitoring Application traffic monitoring

Always-on monitoring and detection
Signature based detection
Heuristics-based
anomaly detection
Baselining

Advanced DDoS protection
Layer 7
application
protection
Layer 3/4
infrastructure
protection

Low suspicion attributes
Normal packet or request header
Traffic composition and volume is typical
given its source
Traffic valid for its destination
High suspicion attributes
• Suspicious packet or request headers
• Entropy in traffic by header attribute
• Entropy in traffic source and volume
• Traffic source has a poor reputation
• Traffic invalid for its destination
• Request with cache-busting attributes
Layer 3/4 infrastructure protection
Traffic prioritization based on scoring

Layer 3/4 infrastructure protection
• Inline inspection and scoring
• Preferentially discard lower priority (attack) traffic
• False positives are avoided and legitimate viewers are protected
Traffic prioritization based on scoring
High-suspicion
packets dropped
Low-suspicion
packets retained

AWS WAF – Layer 7 application protection
Web traffic filtering
with custom rules
Malicious request
blocking
Active monitoring
and tuning

Three modes of operation
Self-service Engage DDoS experts Proactive DRT engagement

AWS WAF included at no additional
cost
Self-service

1. You engage the AWS DDoS Response Team (DRT)
2. DRT triages attack
3. DRT assists you with creating AWS WAF rules
Engage DDoS experts

1. Always-on monitoring engages the AWS DDoS
Response Team (DRT)
2. DRT proactively triages DDoS attack
3. DRT creates AWS WAF rules (prior
authorization required)
Proactive DRT engagement

Attack notification and reporting
Attack monitoring
and detection
• Real-time notification of attacks via Amazon CloudWatch
• Near real-time metrics and packet captures for attack forensics
• Historical attack reports

24x7 access to DDoS Response Team
Before Attack
Proactive consultation and
best practice guidance
During Attack
Attack mitigation
After Attack
Post-mortem
analysis

AWS cost protection
AWS absorbs scaling cost due to DDoS attack
• Amazon CloudFront
• Elastic Load Balancer
• Application Load Balancer
• Amazon Route 53

For protection against most
common DDoS attacks, and
access to tools and best
practices to build a DDoS
resilient architecture on AWS.
AWS DDoS Shield: How to choose
For additional protection against
larger and more sophisticated
attacks, visibility into attacks,
AWS cost protection, Layer 7
mitigations, and 24X7 access to
DDoS experts for complex cases.
Standard Protection Advanced Protection

Data News
AWS Athena, AWS QuickSight
Shay Gury,
BI and Big Data Engineer, AllCloud

AWS Athena
● An interactive query service that
makes it easy to analyze data in
Amazon S3 using standard SQL
● Serverless, so there is no
infrastructure to manage, and you pay
only for the queries that you run.
● No need for complex ETL jobs to
prepare your data for analysis. This
makes it easy for anyone with SQL
skills to quickly analyze large-scale
datasets.

Introducing Amazon Athena
Amazon Athena is an interactive query service
that makes it easy to analyze data in Amazon S3
using standard SQL. Athena is serverless, so
there is no infrastructure to manage, and you
pay only for the queries that you run.

Amazon Athena is Easy to Use
• Log into the Console
• Create a table
• Type in a Hive DDL
Statement
• Use the console Add Table
wizard
• Start querying

Query Data Directly from Amazon S3
• No ETL required. No loading of data. Query data where it
lives.
• Query data in its raw format
• Text, CSV, JSON, weblogs, AWS service logs
• ORC or Parquet
• Querying using ANSI SQL
• Stream data directly from Amazon S3
• Take advantage of Amazon S3 durability and availability

Familiar Technologies Under the Covers
Used for SQL Queries
In-memory distributed query engine
ANSI-SQL compatible with extensions
Used for DDL functionality
Complex data types
Multitude of formats
Supports data partitioning

Creating Tables - Concepts
• Create Table Statements (or DDL) are written in Hive
• Flexible
• Schema on Read
• External tables and partitioning of data
• JSON, TXT, CSV, TSV,
• Parquet and ORC (via Serdes)
• Data is stored in Amazon S3
• Metadata is stored in an a metadata store

Amazon Athena is Fast
• Tuned for performance
• Automatically parallelizes queries Athena uses warm
compute pools across multiple Availability Zones
• Results also stored in S3
• Improve Query performance
• Compress your data
• Use columnar formats
• Partitions

Amazon Athena is Cost Effective
• Pay per query
• $5 per TB scanned from S3
• DDL Queries and failed queries are free
• Save by using
• compression, columnar formats, partitions

Big Query vs Athena
AWS Athena Google Big Query
Data Formats *SV, JSON,Parquet,ORC External (*SV,JSON,AVRO)/Native
ANSI SQL Support Yes Yes
DDL Support Only CREATE/ALTER DROP Create/Update/Delete
Underlying Technology FB presto Google Dremel
Caching No Yes
UDF Not Yet Yes
Data Partition On any key By Day
Pricing 5$/TB (scanned) plus S3 ops
Don’t Pay For Canceled/Failed Queries
5$/TB (Scanned) less cached data

PARQUET AND ORC
• Columnar format
• Schema segregated into footer
• Column major format
• All data is pushed to the leaf
• Integrated compression and
indexes, and stats
• Support for predicate
pushdown
Columnar Formats

Converting to ORC and PARQUET
• You can use Spark to convert the file into PARQUET / ORC
• 20 lines of Pyspark code, running on EMR
• Converts 1TB of text data into 130 GB of Parquet with snappy conversion
• Total cost $5
https://github.com/awslabs/aws-big-data-blog/tree/master/aws-blog-spark-parquet-conversion

Pay by the Query - $5/TB Scanned
• Pay by the amount of data scanned per query
• Ways to save costs
• Compress
• Convert to Columnar format
• Use partitioning
• Free: DDL Queries, Failed Queries
Dataset Size on Amazon S3 Query Run time Data Scanned Cost
Logs stored as Text
files
1 TB 237 seconds 1.15TB $5.75
Logs stored in
Apache Parquet
format*
130 GB 5.13 seconds 2.69 GB $0.013
Savings 87% less with Parquet 34x faster 99% less data scanned 99.7% cheaper

AWS
QuickSight
● A fast, cloud-powered business
analytics service that makes it easy to
build visualizations, perform ad-hoc
analysis, and quickly get business
insights from your data.
● Easily connect to your data, perform
advanced analysis, and create
stunning visualizations and rich
dashboards that can be accessed from
any browser or mobile device.
● 1/10th of the cost of traditional BI
solutions with no upfront investments,
no expensive hardware to purchase or
infrastructure to manage, and no
additional license or maintenance fees.

QuickSight allows you to connect to data from a wide variety of AWS, third-party, and
on-premises sources including Amazon Athena
Amazon RDS
Amazon S3
Amazon Redshift
Amazon Athena
Using Amazon Athena with Amazon QuickSight

QuickSight SPICE
SPICE = Super-fast, Parallel, In-memory Calculation Engine.
• combination of columnar storage,
in-memory technologies
• Data in SPICE is persisted until it is
explicitly deleted by the user.
• SPICE automatically replicates data
for high availability

AWS and GCP Division Overview | Nov. 2016
© 2016 AllCloud
www.allcloud.io
Join Our Fastlane to a
Successful Cloud Deployment
Contact us: info@allcloud.io

You’re Invited
What: AllCloud's Launch Event
When: Jan. 30th, 2017
Where: “Sheva” 19 Hatzfira St., Tel Aviv
Join Us to Experience the Cloud Journey in a Whole
Different Way.

Best of re:Invent 2016 meetup presentation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Best of re:Invent 2016 meetup presentation

Similar to Best of re:Invent 2016 meetup presentation (20)

More from Lahav Savir

More from Lahav Savir (11)

Recently uploaded

Recently uploaded (20)

Best of re:Invent 2016 meetup presentation