Introduction to Real-time, Streaming Data and Amazon Kinesis
•
3 gefällt mir•430 views
by Ray Zhu, Sr. Product Manager, AWS
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Introduction to Real-time, Streaming Data and Amazon Kinesis
Ähnlich wie Introduction to Real-time, Streaming Data and Amazon Kinesis (20)
Mehr von Amazon Web Services
Mehr von Amazon Web Services (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (17)
Introduction to Real-time, Streaming Data and Amazon Kinesis
- 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ray Zhu, Sr. Product Manager, Amazon Kinesis 10/11/2016 Streaming Data: Introduction to Amazon Kinesis
- 2. Agenda • Streaming Scenarios • Amazon Kinesis Streams Overview • Amazon Kinesis Firehose Overview • Amazon Kinesis Analytics Overview • Kinesis Firehose Demo and Walkthrough
- 3. Time Value of Money Data
- 4. Scenarios Accelerated Ingest- Transform-Load Continual Metrics Generation Responsive Data Analysis Data Types IT logs, applications logs, social media / clickstreams, sensor or device data, market data Ad/Marketing Tech Publisher, bidder data aggregation Advertising metrics like coverage, yield, conversion Analytics on user engagement with ads, optimized bid / buy engines IoT Sensor, device telemetry data ingestion IT operational metrics dashboards Sensor operational intelligence, alerts, and notifications Gaming Online customer engagement data aggregation Consumer engagement metrics for level success, transition rates, CTR Clickstream analytics, leaderboard generation, player-skill match engines Consumer Engagement Online customer engagement data aggregation Consumer engagement metrics like page views, CTR Clickstream analytics, recommendation engines 1 2 3 Streaming Data Scenarios
- 5. Kinesis Streams Stores data as a continuous replayable stream for custom applications Kinesis Firehose Load streaming data into Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service Kinesis Analytics Analyze data streams using standard SQL queries Amazon Kinesis
- 6. Match the Services and Scenarios 1 2 3 Kinesis StreamsKinesis Firehose Kinesis Analytics Accelerated Ingest- Transform-Load Continual Metrics Generation Responsive Data Analysis
- 10. Kinesis Firehose
- 13. Kinesis Firehose Demo and Walkthrough
- 14. Data Flow to S3
- 15. Step 1 Set Up Firehose Delivery Stream and Configure Data Transformation
- 16. Destination
- 17. Configuration
- 18. Configuration
- 19. Review
- 20. Step 2 Send Data to Firehose Delivery Stream
- 21. Sample Data 219.134.32.117 - - [16/Feb/2017:09:38:20 -0800] "GET /wp-content HTTP/1.1" 200 4521 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.1; .NET CLR 3.8.23015.5)" 95.169.41.62 - - [16/Feb/2017:09:38:20 -0800] "PUT /app/main/posts HTTP/1.1" 200 3883 "-" "Mozilla/5.0 (Windows NT 6.2; Trident/7.0; rv:11.0) like Gecko" 221.147.191.247 - - [16/Feb/2017:09:38:20 -0800] "GET /explore HTTP/1.1" 200 6579 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1) AppleWebKit/538.0.1 (KHTML, like Gecko) Chrome/38.0.895.0 Safari/538.0.1" 179.96.123.130 - - [16/Feb/2017:09:38:20 -0800] "GET /list HTTP/1.1" 200 560 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:5.4) Gecko/20100101 Firefox/5.4.6" 132.119.12.76 - - [16/Feb/2017:09:38:20 -0800] "PUT /explore HTTP/1.1" 200 3131 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_0 rv:5.0; AZ) AppleWebKit/535.1.0 (KHTML, like Gecko) Version/4.0.3 Safari/535.1.0" 74.113.56.92 - - [16/Feb/2017:09:38:20 -0800] "DELETE /app/main/posts HTTP/1.1" 200 7069 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_9) AppleWebKit/532.1.0 (KHTML, like Gecko) Chrome/15.0.877.0 Safari/532.1.0"
- 22. After Data Transformation {"host":"26.56.11.130","ident":"-","authuser":"-","request":"GET /wp-content HTTP/1.1","response":200,"bytes":4582,"verb":"GET","@timestamp":"2017-04- 04T11:32:29.000Z","timezone":"-0700","@timestamp_utc":"2017-04-04T18:32:29.000Z"} {"host":"180.153.215.216","ident":"-","authuser":"-","request":"PUT /search/tag/list HTTP/1.1","response":200,"bytes":1461,"verb":"PUT","@timestamp":"2017-04- 04T11:32:29.000Z","timezone":"-0700","@timestamp_utc":"2017-04-04T18:32:29.000Z"} {"host":"155.233.163.37","ident":"-","authuser":"-","request":"GET /explore HTTP/1.1","response":500,"bytes":326,"verb":"GET","@timestamp":"2017-04- 04T11:32:29.000Z","timezone":"-0700","@timestamp_utc":"2017-04-04T18:32:29.000Z"} {"host":"189.176.106.5","ident":"-","authuser":"-","request":"POST /search/tag/list HTTP/1.1","response":200,"bytes":3059,"verb":"POST","@timestamp":"2017-04- 04T11:32:29.000Z","timezone":"-0700","@timestamp_utc":"2017-04-04T18:32:29.000Z"}
- 23. Send Data
- 24. Step 3 Check Results in S3
- 26. Step 4 Monitor Streaming Data Pipeline
- 27. Monitor with CloudWatch Metrics
- 28. Monitor with CloudWatch Logs
- 29. Firehose Pricing
- 30. Pricing
- 32. New Pricing
- 33. Q & A
- 34. Thank you!
Hinweis der Redaktion
- csv timeformat ‘auto’
- 'use strict'; console.log('Loading function'); /* Combined Apache Log format parser */ const parser = /^([\d.]+) (\S+) (\S+) \[([\w:\/]+\s[+\-]\d{4})\] \"(.+?)\" (\d{3}) (\d+) \"([^\"]+)\" \"([^\"]+)\"/; exports.handler = (event, context, callback) => { let success = 0; // Number of valid entries found let failure = 0; // Number of invalid entries found /* Process the list of records and transform them */ const output = event.records.map((record) => { const entry = (new Buffer(record.data, 'base64')).toString('utf8'); const match = parser.exec(entry); if (match) { /* Prepare CSV version from Apache log data */ const requestParts = match[5].split(' '); const result = `${match[1]},${match[4]},${requestParts[0]},${requestParts[1]},${requestParts[2]},${match[6]},${match[7]},${match[8]},"${match[9]}"\n`; const payload = (new Buffer(result, 'utf8')).toString('base64'); success++; return { recordId: record.recordId, result: 'Ok', data: payload, }; } else { /* Failed event, notify the error and leave the record intact */ failure++; return { recordId: record.recordId, result: 'ProcessingFailed', data: record.data, }; } }); console.log(`Processing completed. Successful records ${success}, Failed records ${failure}.`); callback(null, { records: output }); };
- 'use strict'; console.log('Loading function'); /* Combined Apache Log format parser */ const parser = /^([\d.]+) (\S+) (\S+) \[([\w:\/]+\s[+\-]\d{4})\] \"(.+?)\" (\d{3}) (\d+) \"([^\"]+)\" \"([^\"]+)\"/; exports.handler = (event, context, callback) => { let success = 0; // Number of valid entries found let failure = 0; // Number of invalid entries found /* Process the list of records and transform them */ const output = event.records.map((record) => { const entry = (new Buffer(record.data, 'base64')).toString('utf8'); const match = parser.exec(entry); if (match) { /* Prepare CSV version from Apache log data */ const requestParts = match[5].split(' '); const result = `${match[1]},${match[4]},${requestParts[0]},${requestParts[1]},${requestParts[2]},${match[6]},${match[7]},${match[8]},"${match[9]}"\n`; const payload = (new Buffer(result, 'utf8')).toString('base64'); success++; return { recordId: record.recordId, result: 'Ok', data: payload, }; } else { /* Failed event, notify the error and leave the record intact */ failure++; return { recordId: record.recordId, result: 'ProcessingFailed', data: record.data, }; } }); console.log(`Processing completed. Successful records ${success}, Failed records ${failure}.`); callback(null, { records: output }); };
- {{internet.ip}} - - [{{date.now("DD/MMM/YYYY:HH:mm:ss ZZ")}}] "{{random.weightedArrayElement({"weights":[0.6,0.1,0.1,0.2],"data":["GET","POST","DELETE","PUT"]})}} {{random.arrayElement(["/list","/wp-content","/wp-admin","/explore","/search/tag/list","/app/main/posts","/posts/posts/explore"])}} HTTP/1.1" {{random.weightedArrayElement({"weights": [0.9,0.04,0.02,0.04], "data":["200","404","500","301"]})}} {{random.number(10000)}} "-" "{{internet.userAgent}}" AKIAJOZZKMI6NBZDQIAQ nQU9Lx4f9Z07VA1bW4oDe1OQDZ1JsjRNYiREbVVH