More Related Content Similar to Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021 Similar to Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021 (20) More from InfluxData (20) Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021 1. Ryan Betts, VP Engineering
Tim Hall, VP Products
InfluxDB Roadmap &
Engineering Update
2. © 2021 InfluxData. All rights reserved. 2
| Agenda
● InfluxDB Enterprise
● InfluxData Community
● InfluxDB Cloud
4. © 2021 InfluxData. All rights reserved. 4
|
● 2x Growth
● Management Depth
● Deployment & Build
Automation
Team Growth and Security Posture
● SOC 2 Type II
● Penetration Testing
● Code Signing
● Digital Supply Chain
5. © 2021 InfluxData. All rights reserved. 5
| These Engineering Investments Enable...
Delivery of new data access &
processing while
not significantly altering
current deployment
Continue the success
of deployments
managed by you
6. © 2021 InfluxData. All rights reserved. 6
|InfluxDB Enterprise 1.9
● Flux
○ Upgraded from 0.65 -> 0.113 including
feature advancements and improvements
already available in InfluxDB Cloud
○ Query management controls
■ Place limits on resource usage
■ Log queries which exceed these limits
○ Performance Improvements
■ last(), min(), max()
● Expanded metrics for managing
InfluxDB Enterprise at scale
○ data size on disk per measurement (also in
1.8.5)
○ cardinality per measurement
○ data written metrics by measurement and/or
username
○ series creation metrics
● InfluxQL
○ Add WITH KEY syntax to show tag keys
○ Performance improvement for sorted merge
iterator
● Other
○ Updated prometheus remote protocol
○ Enable security integration with Kapacitor via LDAP
○ Node labels: add key/value meta data to either meta
nodes and/or data nodes.
Note: There is no companion InfluxDB 1.9 OSS
release paired with InfluxDB Enterprise 1.9
Also, 32-bit support is also no longer available.
7. © 2021 InfluxData. All rights reserved. 7
|
Key Theme:
Continue the success of your on-prem deployments
● Leverage Node Labels for shard ownership/placement
● Streamline backup process including object store targets
● Prevent shard creation outside designated time range
● Upgrade internal hashing algorithm to support FedRAMP/FIPS 140-2
● Reduce frequency of OOM conditions via InfluxQL
○ Ex: Require time range
Need something? Let’s Talk
InfluxDB Enterprise 1.10 and beyond
8. © 2021 InfluxData. All rights reserved. 8
|Kapacitor 1.6
● Goal: enable a path for Kapacitor users to continue to leverage
that investment with InfluxDB 2.x
● Introduce Flux task engine to Kapacitor
○ Allow definition, deployment, scheduling and management
of Flux-based tasks
○ Leverage the v2 API for tasks
○ Flux helper functions - ease conversion of TICKscript to
Flux tasks
○ Add queryFlux node for TICKscripts
○ Alert Handler additions: Zenoss
● Backport Kapacitor Enterprise Security feature to OSS
○ Allows Kapacitor, when paired with InfluxDB Enterprise to
leverage the same security store.
9. © 2021 InfluxData. All rights reserved. 9
|Chronograf 1.9
• Data Explorer
– Set database context for meta
queries such as DELETE/DROP
– Filter fields
• Dashboards
– Flux-based template variables
• LogViewer Improvements
• Eliminate phantom query refresh
• Enhance data precision via time in
visualization
• Enable time selector
• Improve load performance
• Option to hide histogram panel
• Kapacitor Integration
– Support configuration for
ServiceNow, Big Pandas, and
Zenoss, retire HipChat
– Add pagination for TICKscripts
• Chronograf HA
– Allow for migration script to be
run against secure etcd cluster
– Improved integration with
InfluxDB Enterprise for
administrative functions
11. © 2021 InfluxData. All rights reserved. 11
|
● Due to limited address space, large databases cause InfluxDB to crash.
● What’s large? > 3GB
● Typically retention policies can be used to restrict the amount of data
stored on these smaller footprint instances.
● Community to the rescue
○ A number of community members have collaborated on addressing this and
building a 32-bit edition of InfluxDB to address this.
● Thanks to @fluffynukeit and @simonvetter
● Builds: https:/
/github.com/simonvetter/influxdb/
We do not plan on producing a 32-bit build for InfluxDB 2.x
Community can build for this platform, if desired.
InfluxDB 1.x OSS on 32-bit Systems
12. © 2021 InfluxData. All rights reserved. 12
| OSS 1.x → OSS 2.0
How to take the limo ride?
IN-PLACE UPGRADE
● Updates Configuration
● Write Data Into 2.0 Buckets
● Create DBRP Mappings
● Migrate Users, Metadata
● Build TSI (Time-series index)
● Includes Docker-based installs as of 2.0.4
https:/
/docs.influxdata.com/influxdb/v2.0/upgrade/v1-to-v2/
INFLUXDB OSS 2.0
13. © 2021 InfluxData. All rights reserved. 13
|
Continuous Queries
(CQ)
Replace with Flux
Tasks. Leverage docs
1.x Supported
Protocols
(Graphite, CollectD,
OpenTSDB, UDP, Prometheus
Remote Write)
Use Telegraf
Kapacitor 1.x
(Subscription API)
Dual write to
InfluxDB 2.0 and
Kapacitor 1.x
Chronograf / Grafana
(Dashboards)
Existing dashboards
will continue to work
with InfluxQL
compatibility API
Authentication
Auth required by
default
Admin users
Re-assess and
manually create
admin users in OSS
2.0
Platform Support
Docker, Windows,
ARM builds are
available now!
A Few Things To Watch For...
INFLUXDB OSS 2.0
https:/
/docs.influxdata.com/influxdb/v2.0/upgrade/v1-to-v2/
14. © 2021 InfluxData. All rights reserved. 14
| Flux: TICKscript Compatibility Package
● Most TICKscript functions have similar counterparts in Flux.
● The tickscript package simplifies conversion of
TICKscripts to InfluxDB 2.x tasks.
● import ts "contrib/bonitoo-io/tickscript"
● Provides a set of functions making conversion easier.
● Allows for creation of custom checks executed as Flux
tasks and triggering notifications in InfluxDB v2.x
15. © 2021 InfluxData. All rights reserved. 15
|
2.0 OSS Alpha
2.1
● User Interface
○ Add Notebooks support
○ Add Dashboard
annotations
● Flux performance improvements
○ first/last
○ count/sum
○ window grouping
○ and more...
● Operational Improvements
○ Additional Query Log
Levels
● Update to Go 1.16
Q3
● API Improvement
○ Add/Standardize Pagination
across APIs
● Functions: Flux
● User Interface
○ Editing Tasks w/ Notebooks
○ Telegraf Config
enhancements
○ Static legends in Graphs
○ Secrets management
○ Custom base path
● Connection to InfluxDB Cloud
○ Status
● Continue Flux performance
improvements based on feedback
2.x
Q4
● User Interface
○ InfluxDB Templates
integration
○ Alert building w/ Notebooks
● Connection to InfluxDB Cloud
○ Data Replication
○ Remote Management
○ Single Login
InfluxDB OSS 2.x
2022
16. © 2021 InfluxData. All rights reserved. 16
|
In Nov 2020, we invited 3 community members to participate
in PR reviews and more.
● Sven Rebhan - @srebhan, @DocLambda
● Thomas Casteleyn - @Hipska
● Paweł Żak - @zak-pawel, @p-zak
Their contributions have been invaluable in terms of helping
move the Telegraf project forward! Thank you!!!
Telegraf Community Contributions
17. © 2021 InfluxData. All rights reserved. 17
| Telegraf Enhancements in 2020
– Dialect of Python
Specification has details about the
syntax and available functions
– Execution cannot access file system,
network, system resources
– Use Cases:
• Math operations
• Logic operations
• Some string operations
• Add or remove tags/fields/metrics
External Plugins via Execd Starlark Processor
Telegraf
Inputs
Processors
Aggregators
Outputs
Execd
Execd
Execd
Execd
External Input
External Processor
External Aggregator
External Output
18. © 2021 InfluxData. All rights reserved. 18
|Telegraf 1.18.0
Released - March 17, 2021
New Inputs
● Beat Input Plugin
● CS:GO Input Plugin
● Directory Monitoring Input Plugin*
● RavenDB Input Plugin
● NFS Input Plugin*
New Outputs
● Google BigQuery Output Plugin
● Grafana Loki Output Plugin
● Sensu Output Plugin
● SignalFX Output Plugin
New External Plugins
● GeoIP Processor Plugin
● Plex Webhook Input Plugin
● SMCIPMITool Input Plugin
New Aggregators
● Derivative Aggregator Plugin
● Quantile Aggregator Plugin
New Processors
● AWS EC2 Metadata Processor
Plugin
New Parsers
● XML Parser Plugin
New Serializers
● MessagePack Serializer
Plugin
*community requests
New Features
● Updated to Go 1.16.2
● Code Signing for Windows & Mac
● More SNMP v3 authentication
protocols including SHA-512
● 140+ commits from 78
contributors
19. © 2021 InfluxData. All rights reserved. 19
|
JSON Parsing
Improvements
Telegraf: What’s Next?
Prometheus Remote
Write Parsing
Dynamic
Configuration
21. © 2021 InfluxData. All rights reserved. 21
| Global Distribution
Run where you are / Pay how you want
Multi-Cloud +
Global
Distribution
Now:
US (Oregon)
US(Virginia)
EU(Frankfurt)
US (Iowa)
EU (Belgium)
US (Virginia)
EU
(Netherlands)
22. © 2021 InfluxData. All rights reserved. 22
| Global Distribution
Run where you are / Pay how you want
Next:
• Continue expansion across all
super regions (N. America,
Europe, APAC)
– Next: AWS AP (Sydney)
– Azure Europe (North)
– AWS Europe (Ireland)
– Other APAC*
Multi-Cloud +
Global
Distribution
Now:
US (Oregon)
US(Virginia)
EU(Frankfurt)
US (Iowa)
EU (Belgium)
US (Virginia)
EU
(Netherlands)
23. © 2021 InfluxData. All rights reserved. 23
|
• Usage-based billing
• Social sign-on
integration with Google
and Microsoft (Azure AD)
• Eligible to retire CSP
committed consumption
Billing Options
Cloud Service Provider Marketplace Offers
Now:
Later: Annual contracts
24. © 2021 InfluxData. All rights reserved. 24
| IoT: Geotemporal
Available Now!
Telegraf Plugins
S2 Geo Processor: adds tags
with an S2 cell ID token of a
specified cell level. Specify lat
and lon field values with
WGS-84 coordinates in
decimal degrees.
GeoIP Processor: looks up IP
addresses in the MaxMind
GeoLite2 database and adds
the respective ISO country
code, city name, latitude and
longitude.
25. © 2021 InfluxData. All rights reserved. 25
| IoT: Geotemporal
Available Now!
Filtering
• filterRows()
• gridFilter()
• strictFilter()
Aggregate
• groupByArea()
• asTracks()
Transformation
• s2CellIDToken()
• toRows()
Supported Shapes
• box - defined by: minLat, maxLat,
minLon, maxLon
• circle - defined by: lat, lon, radius
• polygon - array of points: lat, lon
Telegraf Plugins
S2 Geo Processor: adds tags
with an S2 cell ID token of a
specified cell level. Specify lat
and lon field values with
WGS-84 coordinates in
decimal degrees.
GeoIP Processor: looks up IP
addresses in the MaxMind
GeoLite2 database and adds
the respective ISO country
code, city name, latitude and
longitude.
26. © 2021 InfluxData. All rights reserved. 26
| IoT: Geotemporal
Available Now!
Filtering
• filterRows()
• gridFilter()
• strictFilter()
Aggregate
• groupByArea()
• asTracks()
Transformation
• s2CellIDToken()
• toRows()
Supported Shapes
• box - defined by: minLat, maxLat,
minLon, maxLon
• circle - defined by: lat, lon, radius
• polygon - array of points: lat, lon
Telegraf Plugins
S2 Geo Processor: adds tags
with an S2 cell ID token of a
specified cell level. Specify lat
and lon field values with
WGS-84 coordinates in
decimal degrees.
GeoIP Processor: looks up IP
addresses in the MaxMind
GeoLite2 database and adds
the respective ISO country
code, city name, latitude and
longitude.
Map Visualisation
27. © 2021 InfluxData. All rights reserved. 27
| Key Themes in 2021
Teams & Collaboration
Tenant Observability
Functions & Reuse
Ease Data Movement
28. © 2021 InfluxData. All rights reserved. 28
| Notebooks
Teams & Collaboration
● Goal is to provide a single
consistent building
experience in our platform
● Designed to allow teams to
incrementally build queries
or background processing
tasks together
● Flexible enough to guide
new users and unblock
power users
● View multiple related
resources in a single view
Now
29. © 2021 InfluxData. All rights reserved. 29
| Dashboard Annotations
Teams & Collaboration
● Goal is share context in
dashboards
● Add, edit, delete as
needed
● Annotations never expire
● Available in your InfluxDB
Cloud accounts soon
● Ranged annotations,
custom colors, support
for more graph types
coming soon
Now
30. © 2021 InfluxData. All rights reserved. 30
| Tenant Observability
● Goal is to provide
self-service access to
usage and operations:
○ Usage
○ Cardinality
○ Rate limit events
○ Write errors
○ Query Log
● User configurable
templates for dashboards
& alerts
● Available on all cloud
accounts shortly
Next
31. © 2021 InfluxData. All rights reserved. 31
| Evolution of Tasks
Schedule
Tasks: Flux
Now
32. © 2021 InfluxData. All rights reserved. 32
| Evolution of Tasks
Functions
Schedule
Trigger
(via Events)
Tasks: Flux
Parameterized /
Reusable
Remote Invocation
Task
(Via Schedule)
Web hook
(via API)
Flux-based...to Start
Next
33. © 2021 InfluxData. All rights reserved. 33
| Functions: Flux
● Goal is code reusability
○ Define Flux code once
○ Register as a Function with
defined parameters
○ Invoke, as desired, with
different values supplied as
arguments
● Power tasks, triggers, dashboard
cells with common functions
Next
Task = Function (Schedule, Arguments)
34. © 2021 InfluxData. All rights reserved. 34
| Triggers
● Goal is to use events to invoke
functions
● Events can be “write to a bucket”
● Use cases:
○ Late arrivals of data
○ Streamlined alerting
Next
QUERY
ACT
Scheduled Task / Alert check
Notificatio
n systems
WRITE
Notificatio
n systems
WRITE
WRITE
ACT
Triggers
35. © 2021 InfluxData. All rights reserved. 35
| Functions: Polyglot Programming
● Goal is make the platform do more
for you
● Two main use cases:
○ Pre-process data before writing
into InfluxDB
○ Post-process rows returned
from querying InfluxDB
● MVP - Run your python code as
function that can be invoked
remotely
Next
Python code as a
Function
36. © 2021 InfluxData. All rights reserved. 36
|Make data movement easier
Enable users to define object store locations within their CSP account.
Bulk Import
Line protocol
CSV
TSM files
Parquet
Next
37. © 2021 InfluxData. All rights reserved. 37
|Make data movement easier
Enable users to define object store locations within their CSP account.
Bulk Import
Line protocol
CSV
TSM files
Parquet
Self-Service
Backup
Bucket backup
(restore to OSS or Cloud)
-- or --
Export to Parquet
Next
38. © 2021 InfluxData. All rights reserved. 38
|Make data movement easier
Enable users to define object store locations within their CSP account.
Bulk Import
Line protocol
CSV
TSM files
Parquet
Self-Service
Backup
Bucket backup
(restore to OSS or Cloud)
-- or --
Export to Parquet
Next
Cloud Native
Ingest
Connect with cloud
native messaging
services
• MQTT,
• AWS Kinesis
• Azure EventHub
39. © 2021 InfluxData. All rights reserved. 39
|Bonus: Explicit Schema
Define Buckets with Explicit Schema
● Blocks non-compliant writes
● Allow for additions
Next
40. © 2021 InfluxData. All rights reserved. 40
|
InfluxDB Enterprise
○ Continue the success of deployments managed by you
○ Deliver latest improvements for data access & processing
InfluxData Community
○ Limo door is open...and ready to take you on the ride to InfluxDB 2.x
○ Telegraf advancements makes it the most powerful data acquisition tool ever
InfluxDB Cloud
○ Teams & Collaboration
○ Functions & Reuse
○ Tenant Observability
○ Ease Data Movement
Summary