Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Wild Rydes with Big Data/Kinesis focus: AWS Serverless Workshop

55 Aufrufe

Veröffentlicht am

This is a hands-on workshop where every participant will not only learn how to architect and implement a serverless application on Amazon Web Services using nothing but serverless resources for all layers in theory, but actually do it in practice, with all the necessary support from the speakers. Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers. You can build them for nearly any type of application or backend service, and everything required to run and scale your application with high availability is handled for you. Building serverless applications means that developers can focus on their core product instead of worrying about managing and operating servers or runtimes. This reduced overhead lets developers reclaim time and energy that can be spent on developing great products which scale and that are reliable.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Wild Rydes with Big Data/Kinesis focus: AWS Serverless Workshop

  1. 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Markus Kaiser, Solutions Architect Daniel Geske, Solutions Architect Wild Rydes: Dawn of a new unicorn Serverless data processing on AWS
  2. 2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark What to expect 1. Serverless architectures 2. Wild Rydes scenario 3. Modules and relevant AWS services • Setup • Real-time data streaming • Stream aggregation • Stream processing • Data lake • Extra credit (optional) 4. Summary and clean-up
  3. 3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Serverless Build and run applications and services without thinking of servers Fully Managed Developer Productivity Continuous Scaling
  4. 4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
  5. 5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Help Wild Rydes Disrupt Transportation! So how does this magic work?
  6. 6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Wild Rydes is Backed by Leading Investors THE BARN ACCELERATOR TENDERLOIN CAPITAL PENGLAI COMMUNICATIONS AND POST NEW CENTURY TECHNOLOGY CORP LIMITED
  7. 7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Your Task: Process and visualize data in real-time Welcome to Wild Rydes Inc., Employee #3!
  8. 8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Serverless data processing on AWS http://github.com/aws -samples/aws- serverless-workshops
  9. 9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Setup Configure your AWS Cloud9 IDE and setup pre-requisites like an AWS Account.
  10. 10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Real-time Streaming Data Create a stream in Kinesis and write to and read from the stream to track Wild Rydes unicorns on a live map. In this module you’ll also create an Amazon Cognito identity pool to grant the live map access to your stream. https://dataprocessing.wildrydes.com/streaming-data.html
  11. 11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Kinesis Data Streams • Easy administration & low cost • Secure, durable storage • Build real time applications with framework of choice
  12. 12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Kinesis Data Streams Producers Consumers Kinesis Agent Apache Kafka AWS SDK LOG4J Flume Fluentd AWS Mobile SDK Kinesis Producer Library Get* APIs Kinesis Client Library + Connector Library Apache Storm Amazon EMR AWS Lambda Apache Spark Amazon Kinesis
  13. 13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Stream Aggregation Build an Kinesis Data Analytics application to read from the stream and aggregate metrics like unicorn health and distance traveled each minute. https://dataprocessing.wildrydes.com/streaming-aggregation.html
  14. 14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Kinesis Analytics • Connect to streaming source • Write SQL code to process streaming data • Continuously deliver SQL results • Fully managed, automatic elasticity
  15. 15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Stream Processing Persist aggregate data from application to a backend database stored in DynamoDB and run queries against those data. https://dataprocessing.wildrydes.com/stream-processing.html
  16. 16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Data Lake Use Kinesis Data Firehose to flush the raw sensor data to an S3 bucket for archival purposes. Using Athena, you’ll run SQL queries against the raw data for ad-hoc analysis. https://dataprocessing.wildrydes.com/data-lake.html
  17. 17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Kinesis Data Firehose • Zero administration and seamless elasticity • Direct-to-data store integration • Serverless, continuous data transformations
  18. 18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Amazon Athena No loading of data, analyze data directly from Amazon S3 Standard SQL • Support for complex data types (arrays, structs) • Support for partitioning of data by any key Query data in its raw format • Athena supports multiple data formats: Text, CSV, TSV, JSON, weblogs, AWS service logs • or convert to an optimized form like ORC or Parquet for the best performance and lowest cost
  19. 19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Extra credit AWS SDK Amazon Kinesis Data Analytics AWS Lambda Amazon Athena
  20. 20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Summary
  21. 21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Serverless Data Processing Architecture
  22. 22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Pricing AWS Service Price* Example Kinesis Firehose Data Volume: $0.033 per GB 1350 GB: $44.55 S3 Data Volume: $0.0245 per GB PUT Requests: $0.0054 per 1.000 27 GB: $0.66 45.000 PUT: $0.243 Athena Scanned Data: $5 per TB 5.4 TB: $27.00 AWS Service Price* Example (Monthly) Kinesis Streams Shard: $0.018 per hour PUT Units: $0.0175 per Mio 2 x Shard: $27.00 270M PUTs: $4.725 Kinesis Analystics KPU: $0.132 per hour 1 x KPU: $99 Lambda Requests: $0.20 per Mio Execution: $0,00001667 per GB/s (negligible) 1.000 Unicorns, 10s Status Msg, Msg Size 1KB eu-central-1 30 days retention, 1:10 compression, 200 queries
  23. 23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Clean up 1. Amazon Athena 2. Amazon Kinesis Data Analytics 3. Amazon Kinesis Data Firehose 4. Amazon Kinesis Data Streams 5. Amazon S3 6. AWS Lambda 7. Amazon DynamoDB 8. AWS IAM 9. Amazon Cognito 10.Cloud9 IDE
  24. 24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Go build Markus Kaiser, Solutions Architect @markuskaiser Daniel Geske, Solutions Architect @btx94

×