This document summarizes a workshop on building comprehensive big data architectures using AWS services and solutions from the AWS Marketplace. The workshop goals are to show the benefits of integrating AWS and demonstrate how to build a real-time data pipeline using Amazon Redshift and perform predictive analytics using machine learning. The workshop consists of demonstrations of extending on-premises data to AWS, transforming and analyzing data on Redshift, and visualizing predictions, as well as exercises where attendees build a streaming data pipeline and perform predictive modeling. The document also describes the AWS Marketplace Intelligent Analytical System solution and highlights takeaways about combining on-premises and cloud data, using predictive analytics, and the benefits of a modular big data architecture

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Comprehensive Big Data Architecture Made Easy:
The AWS Marketplace Intelligent Analytical System
Luis Daniel Soto, AWS.
@luisdans
AWS Re:INVENT HANDS-ON WORKSHOP
Kim Schmidt, President & CIO/Dataleader.io
@dataleader
GPSWKS301
November 28, 2017
AWS re:INVENT

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Big Data Workshop Audience and Goals
Workshop Audience:
• APN Consulting Partners
• Organizations of all sizes
Workshop Goals:
• Show you the benefits of integrating AWS services and solutions from AWS
Marketplace
• Exercise 1: Build a data pipeline that will stream live data into Amazon
Redshift
• Exercise 2: Leverage machine learning to generate predictive analytics
• Learn how to avoid common errors when building a Big Data Architecture

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What we’ll be doing today
1. Introduction (5 minutes)
2. Collect and Store Data (60 minutes)
• DEMO #1: Extending on-premise data load to AWS
• COLLABORATIVE TEAM EXERCISE: Building a data pipeline
3. Transform & Analyze (55 minutes)
• DEMO #2: Orchestrate, transform and aggregate data on Amazon Redshift
• COLLABORATIVE TEAM EXERCISE: Predictive Analytics with Machine Learning
• DEMO #3: Visualization of prediction output for real-time
4. Operations Management (25 minutes)
• PRESENTATION: AWS Marketplace Intelligent Analytical System
• DISCUSSION: Other challenges on building a end-to-end Big Data architecture
5. Workshop Wrap-up (5 minutes)
Collect & Store Transform
& Analyze
Operations
Management

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GENERATE COLLECT STORE ANALYZE/PROCESS CONSUME
Increasing variety of
sources
Little visibility into
new data
Myopic view of
existing data
Difficulty
consolidating data
across different
sources and locations
Challenges
normalizing/
transforming/
aggregating data into
a standardized
format
Unable to capture
and/or process data
as quickly as it is
being generated
Unfamiliarity with
modern data
management
techniques
Lack of necessary skills
to implement and
maintain new
technologies
Scaling IT
infrastructure
Inability to process
data in a timely
manner once its
needed
Limited resources
and capabilities to
experiment and
iterate
Processing all data in
various formats
Predicting future
required capacity
Make more intelligent
business decisions
Limited adoption due
to rigidity and
inflexibility of legacy
BI tools
Be able to run queries
quickly
Get to data-driven
results faster
What we hear from our customers
Big Data Architecture

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Immediate availability
Broad & deep capabilities
Trusted & secure
Large partner ecosystem
http://aws.amazon.com/mp
Big Data on AWS iAS

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DEMO 1: EXTEND YOUR ON-PREMISES
DATA MANAGEMENT TO AWS
Transform
& Analyze
Operations
Management
AWS MARKETPLACE: INTELLIGENT ANALYTICAL SYSTEM
Collect & Store

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SoftNAS Cloud High-Performance Cloud NAS

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Software-only NAS virtual appliance
built for the cloud
• Full protocol support: CIFS, NFS, AFP,
iSCSI
• High Availability
• Snapshots / Rollbacks
• Replication
• Deduplication
• Compression
• Price and Performance Tunable
• Active Directory / LDAP integration
• Scales from Gigabytes to Petabytes
SoftNAS Cloud Virtual NAS Overview

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Transform
& Analyze
Operations
Management
Collect & Store
WORKSHOP ACTIVITY 1: CREATING A
REAL-TIME STREAMING DATA PIPELINE
AWS MARKETPLACE: INTELLIGENT ANALYTICAL SYSTEM

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. See how to use serverless AWS Lambda to transform streaming data from a
sample weblog generator
2. Define an Amazon Kinesis Firehose Delivery Stream
3. Set up Amazon Kinesis Streams
4. Monitor the live stream using Amazon CloudWatch
5. Query the data in Amazon Redshift via a customized command line
Collaborative Team Exercise #1

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operations
Management
Collect & Store
DEMO 2: MPP DATA TRANSFORMATION
& ORCHESTRATION
AWS MARKETPLACE: INTELLIGENT ANALYTICAL SYSTEM
Transform
& Analyze

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operations
Management
Collect & Store
WORKSHOP ACTIVITY 2:
PREDICTIVE ANALYTICS
AWS MARKETPLACE: INTELLIGENT ANALYTICAL SYSTEM
Transform
& Analyze

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. Query data in Amazon S3
2. Design a predictive scoring formula using PredicSis.ai
3. Compute predictive models to apply to real-time customer data that can
be used to discover outliers who are ready right now to buy a Tesla
Collaborative Team Exercise #2

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operations
Management
Collect & Store
DEMO 3: REAL-TIME DATA
VISUALIZATIONS AND ALERTS
AWS MARKETPLACE: INTELLIGENT ANALYTICAL SYSTEM
Transform
& Analyze

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Collect & Store
OPERATIONS MANAGEMENT
AWS MARKETPLACE: INTELLIGENT ANALYTICAL SYSTEM
Transform
& Analyze
Operations
Management

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
iAS: System Architecture and Design

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GROUP DISCUSSION
AWS MARKETPLACE: INTELLIGENT ANALYTICAL SYSTEM
• What challenges is my customer/organization facing
in building a Big Data Architecture?
• What capabilities is my organization missing?
• What parts would I want to further customize?
Collect & Store Transform
& Analyze
Operations
Management

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key Takeaways
There is an increase of importance on new Big Data capabilities including:
• Securely combining data residing on-premises with Cloud applications
• Abandon historical-only business analysis to include predictive and other modern analytics to
immediately act upon key business insights
• Break down data silos and harness data in real time from all over the globe
• The importance of building a modular Big Data architecture
AWS Marketplace enables agility and experimentation
• Combining AWS services with solutions in AWS Marketplace are “pieces of the puzzle” that
can be replaced whenever newer products and services are released
• Easily evaluate innovative software solutions, pay only for what you use
Download the step-by-step guide for the Intelligent Analytical System (iAS)
• https://aws.amazon.com/mp/mp_solution
• https://aws-kimsshmidt.com

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
@ l u i s d a n s
@ d a t a l e a d e r
http://aws.amazon.com/mp