Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, InfluxData

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 29 Anzeige

Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, InfluxData

Herunterladen, um offline zu lesen

Time series data is everywhere -- connected IoT devices, application monitoring & observability platforms, and more. What makes time series datastreams challenging is that they often have orders of magnitude more data than other workloads, with millions of time series datapoints being quite common. Given its ability to ingest high volumes of data, Kafka is a natural part of any data architecture handling large volumes of time series telemetry, specifically as an intermediate buffer before that data is persisted in InfluxDB for processing, analysis, and use in other applications. In this session, we will show you how you can stream time series data to your IoT application using Kafka queues and InfluxDB, drawing upon deployments done at Hulu and Wayfair that allow both to ingest 1 million metrics per second. Once this session is complete, you’ll be able to connect a Kafka queue to an InfluxDB instance as the beginning of your own time series data pipeline.

Time series data is everywhere -- connected IoT devices, application monitoring & observability platforms, and more. What makes time series datastreams challenging is that they often have orders of magnitude more data than other workloads, with millions of time series datapoints being quite common. Given its ability to ingest high volumes of data, Kafka is a natural part of any data architecture handling large volumes of time series telemetry, specifically as an intermediate buffer before that data is persisted in InfluxDB for processing, analysis, and use in other applications. In this session, we will show you how you can stream time series data to your IoT application using Kafka queues and InfluxDB, drawing upon deployments done at Hulu and Wayfair that allow both to ingest 1 million metrics per second. Once this session is complete, you’ll be able to connect a Kafka queue to an InfluxDB instance as the beginning of your own time series data pipeline.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, InfluxData (20)

Anzeige

Weitere von HostedbyConfluent (20)

Aktuellste (20)

Anzeige

Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, InfluxData

  1. 1. Al Sargent Sr. Director, Product InfluxData April 2021 Stream processing IoT time series data with Kafka & InfluxDB Mirek Malecha Product Manager Bonitoo April 2021
  2. 2. © 2021 InfluxData. All rights reserved. 3 What is time series data?
  3. 3. © 2021 InfluxData. All rights reserved. 4 Time series data is ‒ Sequence of data points. ‒ Successive measurements made from same source over a time interval. ‒ Plot the points on a graph and your X axis is time.
  4. 4. © 2021 InfluxData. All rights reserved. 5 Example: Weather conditions
  5. 5. © 2021 InfluxData. All rights reserved. 6 Example: Server monitoring
  6. 6. © 2021 InfluxData. All rights reserved. 7 Example: Healthcare
  7. 7. © 2021 InfluxData. All rights reserved. 8 What is a time series database?
  8. 8. © 2021 InfluxData. All rights reserved. 9 Characteristics of the Data – All Time-stamped data – Generated in regular (Metric) and irregular (Event) time periods – Huge volumes of data – Real-time and time sensitive
  9. 9. © 2021 InfluxData. All rights reserved. 10 Time Series databases: optimized for collecting, storing, retrieving & processing of time series data Compare this to: – Document databases – Optimized for JSON documents – Search databases – Optimized for full-text searches – Traditional relational – Databases optimized for the tabular storage of related data in rows & columns
  10. 10. © 2021 InfluxData. All rights reserved. 11 Time-series use cases
  11. 11. © 2021 InfluxData. All rights reserved. 12 IoT DevOps Monitoring Operations Analytics Industrial devices: factories, energy, vehicles, buildings, agriculture, etc. Consumer sensors: wearables, home appliances, health monitors Monitor the performance of servers, virtual machines, containers, and applications Track business, customer, financial and scientific metrics Primary Use Cases for time series data
  12. 12. Delivered an Industrial IoT data historian that • provides automation and control of their oil drilling platforms • provides edge and centralized insight into emerging patterns by storing and analyzing all the sensor data derived from energy production sensors Internet of Things (IoT) IoT monitoring
  13. 13. © 2020 InfluxData. All rights reserved. 15 InfluxData Time Series Data Platform Accumulate Act Analyze Flux joins MySQL, Postgres, RDS, etc. Telegraf plugins 200+ plugins Client libraries Python, Javascript, Java, Go... Prometheus scrapers FluentD 700+ plugins AWS, Google, Azure InfluxDB Cloud InfluxDB Enterprise InfluxDB Purpose-built Time Series Database Realtime Data Stream Processing Visualization & Dashboarding Data Analysis & Anomaly Detection Alerting & Notifications Metering & Billing InfluxDB Open Source (OSS) On-prem, self-managed cloud Alerting Systems PagerDuty Slack Webhooks Grafana Client Libraries CLI REST API
  14. 14. © 2021 InfluxData. All rights reserved. 16 Simple IoT Solution
  15. 15. © 2021 InfluxData. All rights reserved. 17 IoT Center • Simple demo application • Register, configure and monitor IoT devices • How to integrate InfluxDB • Write measurements • Query database • Manage InfluxDB • Based on Node.js and React • Use Kafka to write measurements • Virtual Device - emulates real devices • Help with links to source code - GIT • InfluxDB Cloud - No DB installation
  16. 16. © 2021 InfluxData. All rights reserved. 18 IoT Center Functions ‒ Device Registration - register new IoT Devices ‒ Virtual Device - emulate IoT device ‒ Dashboard - show measured data from the IoT Devices
  17. 17. © 2021 InfluxData. All rights reserved. 19 IoT Devices Registration Left menu ● Devices Registration
  18. 18. © 2021 InfluxData. All rights reserved. 20 Virtual Device - Generate demo data Left menu ● Virtual Device Top screen ● Button with pencil
  19. 19. © 2021 InfluxData. All rights reserved. 21 Visualisation Left Menu ● Dashboard Filters ● Device ● Time
  20. 20. © 2021 InfluxData. All rights reserved. 22 IoT Center Architecture Configuration InfluxDB Cloud Time Series Database (or standalone InfluxDB) IoT Center IoT Devices InfluxDB JS Client Virtual Device Web Browser Measurements (temp, humidity, pressure, GPS, …) UI Kafka topic: iot Telegraf IoT Devices registration, Queries kafkajs
  21. 21. © 2021 InfluxData. All rights reserved. 23 Arduino Python Java .NET Kafka Scalable component decoupling IoT layer from DB/Analytics layer IoT Center - virtual device ‒ KafkaJS client IoT devices - data flow const producer = kafka.producer() await producer.connect() await producer.send({ key: 'mykey', topic: KAFKA_TOPIC, messages: [{value: influxLineProtocolData}], compression: CompressionTypes.GZIP, }) IoT Device Kafka Telegraf MQTT Broker . . . 1 2 3
  22. 22. © 2021 InfluxData. All rights reserved. 24 Telegraf A plugin-driven server agent collecting messages from Kafka into InfluxDB. 1. Download telegraf (Docker, Linux, Windows, macOS, ARM, ...) https://portal.influxdata.com/downloads/ 2. Update configuration file telegraf.conf (see the next slide) 3. Start it # telegraf --config telegraf.conf Kafka Telegraf IoT Devices
  23. 23. © 2021 InfluxData. All rights reserved. 25 Telegraf.conf file Read data from Kafka https://github.com/influxdata/telegraf/tree/master/plugins/inputs/kafka_consumer Write to InfluxDB (replace fields by your account) Format options https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md ‒ Values, CSV, JSON, XML, InfluxDB Line Protocol, Collectd, Dropwizard, Graphite, Grok, Logfmt, Nagios, Prometheus, PrometheusRemoteWrite, Wavefront [[outputs.influxdb_v2]] urls = ["https://us-west-2-1.aws.cloud2.influxdata.com"] token = "heRbZX2n4kc8Q_jYPddwjkv3dAZRorNQnN67pMwKs1lGgbMW8vWRjAi7VvkUitQMii2XwJM9qX3cnK4oAZDIjg==" organization = "iot-center@bonitoo.io" bucket = "iot_center" [[inputs.kafka_consumer]] brokers = ["localhost:9092"] topics = ["iot"] data_format = "influx"
  24. 24. © 2021 InfluxData. All rights reserved. 26 InfluxDB FREE Account https://cloud2.influxdata.com ‒ Data Explorer ‒ Dashboards ‒ Alerts ‒ Configuration Kafka Telegraf IoT Devices
  25. 25. © 2021 InfluxData. All rights reserved. 27 Demo Kafka Telegraf Virtual Device
  26. 26. © 2021 InfluxData. All rights reserved. 28 IoT Center - Source code https://github.com/bonitoo-io/iot-center-v2 ESP8266 and ESP32 devices Arduino Raspberry Pi Python Mobile application Flutter/Dart
  27. 27. © 2021 InfluxData. All rights reserved. 29 Summary ‒ Time series data ‒ InfluxDB ‒ Highly scalable database ‒ Store & Analyze data ‒ CLI, API, Dashboards, Tasks, etc. ‒ Free cloud time-unlimited account ‒ Telegraf ‒ Reads Kafka messages ‒ Stores data into InfluxDB ‒ 200+ data inputs ‒ Configuration file, only 9 lines
  28. 28. © 2021 InfluxData. All rights reserved. 30 influxdata.com/cloud
  29. 29. Thank You

×