Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, InfluxData

1. Al Sargent Sr. Director, Product InfluxData April 2021 Stream processing IoT time series data with Kafka & InfluxDB Mirek Malecha Product Manager Bonitoo April 2021

3. © 2021 InfluxData. All rights reserved. 4 Time series data is ‒ Sequence of data points. ‒ Successive measurements made from same source over a time interval. ‒ Plot the points on a graph and your X axis is time.

8. © 2021 InfluxData. All rights reserved. 9 Characteristics of the Data – All Time-stamped data – Generated in regular (Metric) and irregular (Event) time periods – Huge volumes of data – Real-time and time sensitive

9. © 2021 InfluxData. All rights reserved. 10 Time Series databases: optimized for collecting, storing, retrieving & processing of time series data Compare this to: – Document databases – Optimized for JSON documents – Search databases – Optimized for full-text searches – Traditional relational – Databases optimized for the tabular storage of related data in rows & columns

11. © 2021 InfluxData. All rights reserved. 12 IoT DevOps Monitoring Operations Analytics Industrial devices: factories, energy, vehicles, buildings, agriculture, etc. Consumer sensors: wearables, home appliances, health monitors Monitor the performance of servers, virtual machines, containers, and applications Track business, customer, financial and scientific metrics Primary Use Cases for time series data

12. Delivered an Industrial IoT data historian that • provides automation and control of their oil drilling platforms • provides edge and centralized insight into emerging patterns by storing and analyzing all the sensor data derived from energy production sensors Internet of Things (IoT) IoT monitoring

13. © 2020 InfluxData. All rights reserved. 15 InfluxData Time Series Data Platform Accumulate Act Analyze Flux joins MySQL, Postgres, RDS, etc. Telegraf plugins 200+ plugins Client libraries Python, Javascript, Java, Go... Prometheus scrapers FluentD 700+ plugins AWS, Google, Azure InfluxDB Cloud InfluxDB Enterprise InfluxDB Purpose-built Time Series Database Realtime Data Stream Processing Visualization & Dashboarding Data Analysis & Anomaly Detection Alerting & Notifications Metering & Billing InfluxDB Open Source (OSS) On-prem, self-managed cloud Alerting Systems PagerDuty Slack Webhooks Grafana Client Libraries CLI REST API

15. © 2021 InfluxData. All rights reserved. 17 IoT Center • Simple demo application • Register, configure and monitor IoT devices • How to integrate InfluxDB • Write measurements • Query database • Manage InfluxDB • Based on Node.js and React • Use Kafka to write measurements • Virtual Device - emulates real devices • Help with links to source code - GIT • InfluxDB Cloud - No DB installation

16. © 2021 InfluxData. All rights reserved. 18 IoT Center Functions ‒ Device Registration - register new IoT Devices ‒ Virtual Device - emulate IoT device ‒ Dashboard - show measured data from the IoT Devices

20. © 2021 InfluxData. All rights reserved. 22 IoT Center Architecture Configuration InfluxDB Cloud Time Series Database (or standalone InfluxDB) IoT Center IoT Devices InfluxDB JS Client Virtual Device Web Browser Measurements (temp, humidity, pressure, GPS, …) UI Kafka topic: iot Telegraf IoT Devices registration, Queries kafkajs

21. © 2021 InfluxData. All rights reserved. 23 Arduino Python Java .NET Kafka Scalable component decoupling IoT layer from DB/Analytics layer IoT Center - virtual device ‒ KafkaJS client IoT devices - data flow const producer = kafka.producer() await producer.connect() await producer.send({ key: 'mykey', topic: KAFKA_TOPIC, messages: [{value: influxLineProtocolData}], compression: CompressionTypes.GZIP, }) IoT Device Kafka Telegraf MQTT Broker . . . 1 2 3

22. © 2021 InfluxData. All rights reserved. 24 Telegraf A plugin-driven server agent collecting messages from Kafka into InfluxDB. 1. Download telegraf (Docker, Linux, Windows, macOS, ARM, ...) https://portal.influxdata.com/downloads/ 2. Update configuration file telegraf.conf (see the next slide) 3. Start it # telegraf --config telegraf.conf Kafka Telegraf IoT Devices

23. © 2021 InfluxData. All rights reserved. 25 Telegraf.conf file Read data from Kafka https://github.com/influxdata/telegraf/tree/master/plugins/inputs/kafka_consumer Write to InfluxDB (replace fields by your account) Format options https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md ‒ Values, CSV, JSON, XML, InfluxDB Line Protocol, Collectd, Dropwizard, Graphite, Grok, Logfmt, Nagios, Prometheus, PrometheusRemoteWrite, Wavefront [[outputs.influxdb_v2]] urls = ["https://us-west-2-1.aws.cloud2.influxdata.com"] token = "heRbZX2n4kc8Q_jYPddwjkv3dAZRorNQnN67pMwKs1lGgbMW8vWRjAi7VvkUitQMii2XwJM9qX3cnK4oAZDIjg==" organization = "iot-center@bonitoo.io" bucket = "iot_center" [[inputs.kafka_consumer]] brokers = ["localhost:9092"] topics = ["iot"] data_format = "influx"

26. © 2021 InfluxData. All rights reserved. 28 IoT Center - Source code https://github.com/bonitoo-io/iot-center-v2 ESP8266 and ESP32 devices Arduino Raspberry Pi Python Mobile application Flutter/Dart

27. © 2021 InfluxData. All rights reserved. 29 Summary ‒ Time series data ‒ InfluxDB ‒ Highly scalable database ‒ Store & Analyze data ‒ CLI, API, Dashboards, Tasks, etc. ‒ Free cloud time-unlimited account ‒ Telegraf ‒ Reads Kafka messages ‒ Stores data into InfluxDB ‒ 200+ data inputs ‒ Configuration file, only 9 lines

29. Thank You

Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, InfluxData

Du liest eine Vorschau.

Hier ansehen

Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, InfluxData

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, InfluxData (20)

Weitere von HostedbyConfluent (20)

Aktuellste (20)