Modern Data Architectures for Real Time Analytics & Engagement
•
2 gefällt mir•603 views
The AWS Workshop Series Online is a series of live webinars designed for IT professionals who are looking to leverage the AWS Cloud to build and transform their business, are new to the AWS Cloud or looking to further expand their skills and expertise. In this series, we will cover: 'Modern Data Architectures for Real-time Analytics and Engagement'.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie Modern Data Architectures for Real Time Analytics & Engagement
Ähnlich wie Modern Data Architectures for Real Time Analytics & Engagement (20)
Mehr von Amazon Web Services
Mehr von Amazon Web Services (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Modern Data Architectures for Real Time Analytics & Engagement
- 2. Modern Data Architectures for Real-Time Analytics & Engagement Surawut Phornthabthong Solutions Architect, Thailand Amazon Web Services
- 3. Surawut Phornthabthong Solutions Architect, Thailand Amazon Web Services
- 4. SCALABLE FLEXIBLE MANAGEABLE COST EFFECTIVE Modern Data Architecture
- 5. Ingest Serving Speed (Real-time) Scale (Batch) Data analysts Data scientists Business users Engagement platforms Automation / events Sources Modern Data Architecture
- 6. Ingest Serving Speed (Real-time) Scale (Batch) Data analysts Data scientists Business users Engagement platforms Automation / events Sources Real-time Pipeline Amazon Kinesis Machines Devices Mobile Clickstream
- 7. Amazon Kinesis Streams Amazon Kinesis Firehose Amazon Kinesis Analytics Kinesis Family
- 8. Availability Zone Availability Zone Availability Zone Amazon Kinesis Stream AWS Lambda KCL App Amazon EMR Streaming Logs Alerts Analysis Dashboards Predictions
- 9. Amazon Kinesis Stream SHARD1000 TPS or 1MB 5 TPS or 2MB SHARD 2000 TPS or 2MB 10 TPS or 4MB SHARD 3000 TPS or 3MB 15 TPS or 6MB Retention: 24 hours to 7 Days
- 10. Creating a Kinesis Stream
- 12. • Writes to one or more Amazon Kinesis Streams • Retry Mechanism • Uses PutRecords • Aggregates • Integrates with Amazon KCL to de-aggregate • Submits Amazon CloudWatch metrics Kinesis Producer Library
- 13. Kinesis Agent • Monitors files and sends new data records to your delivery stream • Handles file rotation, checkpointing, and retry upon failures • Delivers all data in a reliable, timely, and simple manner • Emits AWS CloudWatch metrics
- 14. Availability Zone Availability Zone Availability Zone Amazon Kinesis Stream AWS Lambda KCL App Amazon EMR Streaming Logs Alerts Analysis Dashboards Predictions
- 15. Kinesis Data Out – Kinesis Client Library SHARD 1 SHARD 2 SHARD 3 SHARD N EC2 Instance Worker 1 Worker 2 EC2 Instance Worker 3 Worker N KCL: Java, Node.js, Python, .NET, Ruby
- 17. twitter-trends.com The solution: Local Top 10 My top-10 My top-10 My top-10 Global top-10
- 18. K I N E S I S twitter-trends.com Challenges using the Kinesis API directly Kinesis application Manual creation of workers and assignment to shards How many workers per EC2 instance? How many EC2 instances?
- 19. K I N E S I S twitter-trends.com Using the Kinesis Client Library Kinesis application Shard mgmt table
- 20. K I N E S I S twitter-trends.com Elasticity and load balancing Shard mgmt table Auto scaling Group
- 21. K I N E S I S twitter-trends.com Fault tolerance support in KCL Shard mgmt table X Availability Zone 1 Availability Zone 3
- 22. Checkpoint, replay design pattern Kinesis 1417182123 Shard-i 235810 Shard ID Lock Seq num Shard-i Host A Host B Shard ID Local top-10 Shard-i 0 10 18 X2 3 5 8 10 14 17 18 21 23 0 310 Host AHost B {#Movies: 10235, #Weather: 9835, …}{#Movies: 10235, #Weather: 9910, …} 1023 14 17 18 21 23
- 23. Availability Zone Availability Zone Availability Zone Amazon Kinesis Stream AWS Lambda KCL App Amazon EMR Streaming Logs Alerts Analysis Dashboards Predictions
- 24. Kinesis & Lambda SHARD 1 SHARD 2 SHARD 3 SHARD N AWS Lambda: Node.js, Java, Python, C# AWS Lambda
- 26. Availability Zone Availability Zone Availability Zone Amazon Kinesis Stream AWS Lambda KCL App Amazon EMR Streaming Logs Alerts Analysis Dashboards Predictions
- 31. Data Prep PredictionModel Train Test Split 70% 30% Near Real-time Data Training Data SQL ML
- 32. Ingest Serving Speed (Real-time) Scale (Batch) Data analysts Data scientists Business users Engagement platforms Automation / events Sources Amazon Kinesis AWS Lambda Application Amazon EMR Streaming S3 (Log) Amazon ElasticSearch (Dashboard) Real-time Pipeline
- 33. Amazon Elasticsearch • Search and Analytics • Scalable • Fully Managed • Integrated – Logstash, Kibana
- 35. Ingest Serving Speed (Real-time) Scale (Batch) Data analysts Data scientists Business users Engagement platforms Automation / events Sources Amazon Kinesis AWS Lambda Application Amazon EMR Streaming S3 (Logs) Amazon ElasticSearch (Dashboards) Amazon EMR (Predictions) ML Amazon SNS (Alerts) Real-time Pipeline Amazon Redshift (Analytics)
- 36. Amazon Kinesis Streams Amazon Kinesis Firehose Amazon Kinesis Analytics Kinesis Family
- 37. S3 Redshift Elasticsearch Amazon Kinesis Firehose Auto provisioning Auto partition keys End to End Elastic Batch Compress Encrypt
- 38. Amazon Kinesis Streams Amazon Kinesis Firehose Amazon Kinesis Analytics Kinesis Family
- 39. Kinesis Analytics Stream or Firehose Kinesis Analytics Data OutData In SQL Stream or Firehose
- 42. Sonos
- 43. New X1 Instance - Tons of Memory • Large-scale, in-memory applications • Intel® Xeon® E7 8880 v3 Haswell processors • Up to 2TB of memory • Up to 128 vCPUs per instance
- 44. Intel® Processor Technologies Intel® AVX – Dramatically increases performance for highly parallel HPC workloads such as life science engineering, data mining, financial analysis, media processing Intel® AES-NI – Enhances security with new encryption instructions that reduce the performance penalty associated with encrypting/decrypting data Intel® Turbo Boost Technology – Increases computing power with performance that adapts to spikes in workloads Intel Transactional Synchronization (TSX) Extensions – Enables execution of transactions that are independent to accelerate throughput P state & C state control – provides granular performance tuning for cores and sleep states to improve overall application performance
- 46. twitter.com/awsawscloudseasia aws-asean-marketing@amazon.com facebook.com/amazonwebservices/ youtube.com/user/AmazonWebServices slideshare.net/amazonwebservices Thank you for joining us today. Please complete the survey & let us know what you think of the webinar.
- 47. REGISTER NOW http://amzn.to/2jFt11N Complimentary labs are available only till 31 March 2017 Get hands on experience working with the AWS Technology. Access the complimentary Big Data on AWS self-paced labs
- 48. Q&A
Hinweis der Redaktion
- The second option for writing to Kinesis Streams is to use the Kinesis Producer Library (KPL). This has a number of advantages over writing directly to the endpoint, the KPL, Writes to one or more Amazon Kinesis Streams with an automatic and configurable Retry Mechanism. Uses PutRecords to write multiple records to multiple shards per Request. Aggregates user records to increase payload size and improve throughput. Integrates seamlessly with the Amazon KCL to de-aggregate Batched Records. Submits Amazon CloudWatch metrics on your behalf to provide visibility into producer performance.
- The second option for getting data into Firehose is to use the Kinesis Agent. This allows you to easily get log files into Firehose by monitoring those files.
- The Kinesis Client library simplifies parallel processing of streaming big data by allowing customers to write simple applications for processing records. The application with the Kinesis library is called a Kinesis Worker. Kinesis’s library notifies the customer’s processing code when there is new data to process. Kinesis Workers will often process data and write output to an external data store such as DynamoDB, EMR, Redshift, or S3. Kinesis ensures that every record within a Kinesis Data stream is processed at least once. The KCL takes care of ensuring that there is always a worker process for each shard. You can put the EC2 instances into an autoscaling group to deal with increased load.
- The desired functionality: twitter-trends.com website that displays top-10 current twitter trends. On AWS you might start out by simply creating an ElasticBeanstalk website.
- Unfortunately the Twitter Firehose feed is quickly becoming too big to process on a single machine. We will need to somehow divide up the work so that it can be done in parallel on multiple machines and then coalesced into a output that can be vended by our Beanstalk web server.
- What we’d like to do is compute a local “top-10” value for all the tweets coming into each orange processing box and then combine all the local top-10 lists into a global top-10 list at the web server. The problem is that, without some sort of sorting/grouping function, each processing box will receive a randomly selection of tweets, making it impossible for any one box to know how many tweets are occurring for any given topic. We can solve this problem by taking a page from the map/reduce world. If we were computing the top-10 tweet topics for a stand-alone batch of tweets then what we would do is treat the topic of each tweet as a partition key. Each map task would take in a fraction of all the tweets, group by topic, and send the count for each topic to the appropriate reduce task responsible for that topic. We can do something equivalent by introducing an intermediate stage that takes in the stream of tweets and orders them by topic. Each processing box would then pull only the tweets for the set of topics that it is responsible for. This would enable each box to be the authoritative counter of tweets for some subset of all currently active tweet topics. This is essentially the streaming equivalent of the map/reduce processing paradigm.
- You need to manually do your own shard management and manually scale-out to multiple EC2 instances.
- The Kinesis client library automatically load balances the number of shards acquired by an EC2 instance across all the participating EC2 instances. If we create an auto-scaling group for our instances then we can manage the number of EC2 instances required automatically. We do this by defining a metric in Cloudwatch that the ASG can use to determine when to add or remove EC2 instances from the ASG. Suppose our initial estimate of how many EC2 instances we needed was low, or suppose that we get an increase in traffic. The Cloudwatch metric will breach the load level at which the ASG will spin up another EC2 instance. When the client library code on each EC2 instance next queries the shard management table in DDB each instance will notice that there is an underloaded EC2 instance and some of the instances will relinquish their lock on one of the shards they own. The result will be that the new EC2 instance will then be able to acquire the lock on one or more shards and thereby take over some of the processing of the stream.
- Kinesis Client library simplifies parallel processing of streaming big data by allowing customers to write simple applications for processing records. Kinesis is a Java library that customers compile into their data processing application. The application with the Kinesis library is called a Kinesis Worker. Kinesis’s library notifies the customer’s processing code when there is new data to process. Kinesis Workers will often process data and write output to an external data store such as DynamoDB, EMR, Redshift, or S3. Kinesis guarantees that every record within a Kinesis Data stream is processed at least once The grey box on the top right – will be EC2 instances in your space. Within there is the Kinesis worker process in green. That is your specific application and business logic. That logic leverages our libraries that connect back to Kinesis (in pink on the left) . There is also another VIP on the GET side.
- Amazon Elasticsearch is a managed service for Elasticsearch. Use the AWS Management Console or simple API calls to access a production-ready Amazon Elasticsearch cluster in minutes without worrying about infrastructure provisioning, or installing and maintaining Elasticsearch software. Amazon Elasticsearch Service simplifies time-consuming management tasks --such as ensuring high availability, patch management, failure detection and node replacement, backups, and monitoring-
- With Firehose, you do not provision shards. Instead, Firehose will automatically scale up to meet demand. Firehose will also aggregate, compress and encrypt the messages before writing to either S3, Redshift or Elasticsearch.
- Kinesis Analytics allows you to query a Kinesis Stream or Firehose using SQL. The output of this can then be sent to another Stream or Firehose.
- https://aws.amazon.com/solutions/case-studies/hearst-data-anayltics/
- Firmware Device Logs Application Telemetry Music Service Usage Metrics Cloud Applications Logs
- More : https://aws.amazon.com/blogs/aws/ec2-instance-update-x1-sap-hana-t2-nano-websites/