The document discusses using serverless data and analytics on AWS for operations. It describes ingesting data from various sources into an AWS data lake for storage, processing, analysis and visualization. This allows operational data to be combined and analyzed to improve response by leveraging serverless AWS services like S3, Glue, Athena and QuickSight in a cost-effective way. A demo shows how different data sources can be ingested and analyzed using a data lake approach.

1. A T L A N T A | M A Y 1 9 T H 2 0 2 2

2. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Serverless data and analytics on
AWS for operations
Patrick Hannah (he/him/his)
D E V 3 0 1
CTO
CloudHesive

3. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• A moment of appreciation for Serverless
• Why do I care? What is it (the data)?
• What do I do with it (Data Lifecycle Stages)?
• What are our potential approaches?
• Data lake detail; tying it all together
• In action (demo)
• Conclusion

4. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Serverless’ Flavors
High Level Abstractions
SaaS (Connect)
Hybrid Abstractions
PaaS (DynamoDB)
Low Level Abstractions
IaaS (Lambda)

5. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud Workload Lifecycle Management
• Workload (What is it?)
• Architecture (How does it run?)
• Monitoring (How do I know something has gone wrong?)
• Automation (If something goes wrong, do I need to fix it?)
• Processes (How do I keep it alive)?

6. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workload + Architecture Drives Service Selection
• Virtual Machines
 AMI
 Patching
 Multi-threaded/Multi-task
 Hours to Months
 Per VM/Per Hour
• Containers
 Container File
 Versioning
 Multi-threaded/Single-task
 Minutes to Days
 Per VM/Per Hour
• Functions/Services
 Code
 Versioning
 Single-threaded/Single-task
 Microseconds to Seconds
 Per Memory/Second/Per Request

7. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Serverless Data Stores and Analytics - The Easy Button
• S3 Query – Query objects in S3, through S3
• Athena (and S3 and Glue) – Query objects in S3, Presto
• Quicksight

8. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step Functions Optimized Integrations
• Lambda
• Batch
• DynamoDB
• ECS/Fargate
• SNS
• SQS
• Glue, DataBrew
• SageMaker
• EMR
• CodeBuild
• Athena
• EKS
• API Gateway
• EventBridge
• Step Functions

9. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Options for Event Buses/Messaging/Queuing
• DynamoDB > Triggers
• CloudWatch Logs > Metrics > Alarms / Lambda
• CloudWatch Metrics > Destination
• Kinesis > Lambda
• Event Bridge (DLQ Support) > Lambda
• SQS (DLQ Support) > Lambda
• SNS (DLQ Support) > Lambda
• (DLQ Support) Lambda

10. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Serverless Data Stores and Analytics
• Purpose Built
 DynamoDB – Key/Value
 Timescale – Time Series
 Keyspaces – Cassandra
 QLDB – Ledger
 Aurora – Relational
 Prometheus/Grafana
• Solves a specific problem
 AppFlow – Data Integration Platform
– Profiles
– Wisdom
– Tasks
 HoneyCode – No Code Platform
 MWAA – Airflow

11. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why do I care?
• Business Operational
 What does it cost?
 What value is it providing?
• Technical Operational
 Things important to my business
• Security Operational
 Things important to my business
• Product
 What does it cost?
 What value is it providing?
• Development
 Things important to my business

12. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is it (the data)?
• Inventory – what do we have?
• Cost/usage – how much of it do we have?
• Best practices – what did we miss?
• Governance/risk/compliance – are we covered?
• Metrics/events/traces/streams – what is the system telling us?
• Tickets – where can we improve?

13. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What do I do with it? (Data Lifecycle Stages)
• Generation
• Collection (ingest)
• Processing (filter)
• Storage
• Management
• Analysis (alert) – groupings, advanced functions, AI/ML
• Visualization
• Interpretation (respond)

14. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What are our potential approaches?
• Purpose built
 Software and SaaS based solutions
• Native
 Individual AWS services (e.g. Amazon CloudWatch)
• Amazon OpenSearch Service
 Some effort required to use
• Data lake (build your own, AWS data lake variants)
 Closest to native while providing the option to unify

15. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Lake Detail
• Non AWS native (Third Party SaaS, Hosts, Databases)
• Amazon EventBridge > S3
• Amazon CloudWatch > S3
• Amazon Kinesis > S3
• Amazon S3
• AWS Glue
• Amazon Athena
• Amazon QuickSight
• Amazon EventBridge/AWS Step Functions (what?!)

16. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What do I do with it? (Data Lifecycle Stages)
• Generation > S3 destined data
• Collection (ingest) > Glue
• Processing (filter) > Glue
• Storage > S3
• Management > S3
• Analysis (alert) – groupings, advanced functions, AI/ML > QuickSight
• Visualization > QuickSight
• Interpretation (respond) > Step Functions

17. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo

18. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What did we observe?
• We are able to ingest a variety of sources (native and non native).
• Individually these sources have value.
• Combined, these sources tell a lifecycle driven story.
• We let the data lake services do the heavy lifting for us.
• It wasn’t that difficult to get started.

19. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Conclusion
• Leveraging a data lake style approach for our technology operations
data, we can be more agile in improving our response to the what the
data tells us.
• There is no one size fits all approach—some organizations may
choose a commercial solution, a native solution, an OpenSearch/data
lake style solution, or a combination of the above.
• Deciding factors are the desired end state versus the value of the
data versus the cost to operate.

20. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deepen your skills with digital
learning on demand
Access 500+ free digital courses
and Learning Plans
Earn an industry-recognized
credential
AWS Skill Builder AWS Certifications
Explore resources with a variety
of skill levels and 16+ languages
to meet your learning needs
Join the AWS Certified community
and get exclusive benefits
Receive Ffoundational,
Associate, Professional,
and Specialty certifications
Train now
Access new
exam guides
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Learn in-demand AWS Cloud skills

21. Thank you!
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Patrick Hannah
LinkedIn: patrickjhannah
Twitter: pjhannah1
Polywork: phannah

22. Please complete
the session survey
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.