Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021

Ryan Betts, VP Engineering
Tim Hall, VP Products
InﬂuxDB Roadmap &
Engineering Update

© 2021 InfluxData. All rights reserved. 2
| Agenda
● InfluxDB Enterprise
● InfluxData Community
● InfluxDB Cloud

Managed by you...
InﬂuxDB Enterprise

|
● 2x Growth
● Management Depth
● Deployment & Build
Automation
Team Growth and Security Posture
● SOC 2 Type II
● Penetration Testing
● Code Signing
● Digital Supply Chain

| These Engineering Investments Enable...
Delivery of new data access &
processing while
not signiﬁcantly altering
current deployment
Continue the success
of deployments
managed by you

|InfluxDB Enterprise 1.9
● Flux
○ Upgraded from 0.65 -> 0.113 including
feature advancements and improvements
already available in InfluxDB Cloud
○ Query management controls
■ Place limits on resource usage
■ Log queries which exceed these limits
○ Performance Improvements
■ last(), min(), max()
● Expanded metrics for managing
InfluxDB Enterprise at scale
○ data size on disk per measurement (also in
1.8.5)
○ cardinality per measurement
○ data written metrics by measurement and/or
username
○ series creation metrics
● InfluxQL
○ Add WITH KEY syntax to show tag keys
○ Performance improvement for sorted merge
iterator
● Other
○ Updated prometheus remote protocol
○ Enable security integration with Kapacitor via LDAP
○ Node labels: add key/value meta data to either meta
nodes and/or data nodes.
Note: There is no companion InfluxDB 1.9 OSS
release paired with InfluxDB Enterprise 1.9
Also, 32-bit support is also no longer available.

|
Key Theme:
Continue the success of your on-prem deployments
● Leverage Node Labels for shard ownership/placement
● Streamline backup process including object store targets
● Prevent shard creation outside designated time range
● Upgrade internal hashing algorithm to support FedRAMP/FIPS 140-2
● Reduce frequency of OOM conditions via InﬂuxQL
○ Ex: Require time range
Need something? Let’s Talk
InﬂuxDB Enterprise 1.10 and beyond

|Kapacitor 1.6
● Goal: enable a path for Kapacitor users to continue to leverage
that investment with InfluxDB 2.x
● Introduce Flux task engine to Kapacitor
○ Allow definition, deployment, scheduling and management
of Flux-based tasks
○ Leverage the v2 API for tasks
○ Flux helper functions - ease conversion of TICKscript to
Flux tasks
○ Add queryFlux node for TICKscripts
○ Alert Handler additions: Zenoss
● Backport Kapacitor Enterprise Security feature to OSS
○ Allows Kapacitor, when paired with InfluxDB Enterprise to
leverage the same security store.

|Chronograf 1.9
• Data Explorer
– Set database context for meta
queries such as DELETE/DROP
– Filter fields
• Dashboards
– Flux-based template variables
• LogViewer Improvements
• Eliminate phantom query refresh
• Enhance data precision via time in
visualization
• Enable time selector
• Improve load performance
• Option to hide histogram panel
• Kapacitor Integration
– Support configuration for
ServiceNow, Big Pandas, and
Zenoss, retire HipChat
– Add pagination for TICKscripts
• Chronograf HA
– Allow for migration script to be
run against secure etcd cluster
– Improved integration with
InfluxDB Enterprise for
administrative functions

|
● Due to limited address space, large databases cause InfluxDB to crash.
● What’s large? > 3GB
● Typically retention policies can be used to restrict the amount of data
stored on these smaller footprint instances.
● Community to the rescue
○ A number of community members have collaborated on addressing this and
building a 32-bit edition of InfluxDB to address this.
● Thanks to @fluffynukeit and @simonvetter
● Builds: https:/
/github.com/simonvetter/influxdb/
We do not plan on producing a 32-bit build for InfluxDB 2.x
Community can build for this platform, if desired.
InfluxDB 1.x OSS on 32-bit Systems

| OSS 1.x → OSS 2.0
How to take the limo ride?
IN-PLACE UPGRADE
● Updates Configuration
● Write Data Into 2.0 Buckets
● Create DBRP Mappings
● Migrate Users, Metadata
● Build TSI (Time-series index)
● Includes Docker-based installs as of 2.0.4
https:/
/docs.influxdata.com/influxdb/v2.0/upgrade/v1-to-v2/
INFLUXDB OSS 2.0

|
Continuous Queries
(CQ)
Replace with Flux
Tasks. Leverage docs
1.x Supported
Protocols
(Graphite, CollectD,
OpenTSDB, UDP, Prometheus
Remote Write)
Use Telegraf
Kapacitor 1.x
(Subscription API)
Dual write to
InfluxDB 2.0 and
Kapacitor 1.x
Chronograf / Grafana
(Dashboards)
Existing dashboards
will continue to work
with InfluxQL
compatibility API
Authentication
Auth required by
default
Admin users
Re-assess and
manually create
admin users in OSS
2.0
Platform Support
Docker, Windows,
ARM builds are
available now!
A Few Things To Watch For...
INFLUXDB OSS 2.0
https:/
/docs.influxdata.com/influxdb/v2.0/upgrade/v1-to-v2/

| Flux: TICKscript Compatibility Package
● Most TICKscript functions have similar counterparts in Flux.
● The tickscript package simplifies conversion of
TICKscripts to InfluxDB 2.x tasks.
● import ts "contrib/bonitoo-io/tickscript"
● Provides a set of functions making conversion easier.
● Allows for creation of custom checks executed as Flux
tasks and triggering notifications in InfluxDB v2.x

|
2.0 OSS Alpha
2.1
● User Interface
○ Add Notebooks support
○ Add Dashboard
annotations
● Flux performance improvements
○ first/last
○ count/sum
○ window grouping
○ and more...
● Operational Improvements
○ Additional Query Log
Levels
● Update to Go 1.16
Q3
● API Improvement
○ Add/Standardize Pagination
across APIs
● Functions: Flux
● User Interface
○ Editing Tasks w/ Notebooks
○ Telegraf Config
enhancements
○ Static legends in Graphs
○ Secrets management
○ Custom base path
● Connection to InfluxDB Cloud
○ Status
● Continue Flux performance
improvements based on feedback
2.x
Q4
● User Interface
○ InfluxDB Templates
integration
○ Alert building w/ Notebooks
● Connection to InfluxDB Cloud
○ Data Replication
○ Remote Management
○ Single Login
InfluxDB OSS 2.x
2022

|
In Nov 2020, we invited 3 community members to participate
in PR reviews and more.
● Sven Rebhan - @srebhan, @DocLambda
● Thomas Casteleyn - @Hipska
● Paweł Żak - @zak-pawel, @p-zak
Their contributions have been invaluable in terms of helping
move the Telegraf project forward! Thank you!!!
Telegraf Community Contributions

| Telegraf Enhancements in 2020
– Dialect of Python
Specification has details about the
syntax and available functions
– Execution cannot access file system,
network, system resources
– Use Cases:
• Math operations
• Logic operations
• Some string operations
• Add or remove tags/fields/metrics
External Plugins via Execd Starlark Processor
Telegraf
Inputs
Processors
Aggregators
Outputs
Execd
Execd
Execd
Execd
External Input
External Processor
External Aggregator
External Output

|Telegraf 1.18.0
Released - March 17, 2021
New Inputs
● Beat Input Plugin
● CS:GO Input Plugin
● Directory Monitoring Input Plugin*
● RavenDB Input Plugin
● NFS Input Plugin*
New Outputs
● Google BigQuery Output Plugin
● Grafana Loki Output Plugin
● Sensu Output Plugin
● SignalFX Output Plugin
New External Plugins
● GeoIP Processor Plugin
● Plex Webhook Input Plugin
● SMCIPMITool Input Plugin
New Aggregators
● Derivative Aggregator Plugin
● Quantile Aggregator Plugin
New Processors
● AWS EC2 Metadata Processor
Plugin
New Parsers
● XML Parser Plugin
New Serializers
● MessagePack Serializer
Plugin
*community requests
New Features
● Updated to Go 1.16.2
● Code Signing for Windows & Mac
● More SNMP v3 authentication
protocols including SHA-512
● 140+ commits from 78
contributors

|
JSON Parsing
Improvements
Telegraf: What’s Next?
Prometheus Remote
Write Parsing
Dynamic
Conﬁguration

| Global Distribution
Run where you are / Pay how you want
Multi-Cloud +
Global
Distribution
Now:
US (Oregon)
US(Virginia)
EU(Frankfurt)
US (Iowa)
EU (Belgium)
US (Virginia)
EU
(Netherlands)

| Global Distribution
Run where you are / Pay how you want
Next:
• Continue expansion across all
super regions (N. America,
Europe, APAC)
– Next: AWS AP (Sydney)
– Azure Europe (North)
– AWS Europe (Ireland)
– Other APAC*
Multi-Cloud +
Global
Distribution
Now:
US (Oregon)
US(Virginia)
EU(Frankfurt)
US (Iowa)
EU (Belgium)
US (Virginia)
EU
(Netherlands)

|
• Usage-based billing
• Social sign-on
integration with Google
and Microsoft (Azure AD)
• Eligible to retire CSP
committed consumption
Billing Options
Cloud Service Provider Marketplace Oﬀers
Now:
Later: Annual contracts

| IoT: Geotemporal
Available Now!
Telegraf Plugins
S2 Geo Processor: adds tags
with an S2 cell ID token of a
speciﬁed cell level. Specify lat
and lon ﬁeld values with
WGS-84 coordinates in
decimal degrees.
GeoIP Processor: looks up IP
addresses in the MaxMind
GeoLite2 database and adds
the respective ISO country
code, city name, latitude and
longitude.

| IoT: Geotemporal
Available Now!
Filtering
• filterRows()
• gridFilter()
• strictFilter()
Aggregate
• groupByArea()
• asTracks()
Transformation
• s2CellIDToken()
• toRows()
Supported Shapes
• box - defined by: minLat, maxLat,
minLon, maxLon
• circle - defined by: lat, lon, radius
• polygon - array of points: lat, lon
Telegraf Plugins
decimal degrees.
longitude.

| IoT: Geotemporal
Available Now!
Filtering
• filterRows()
• gridFilter()
• strictFilter()
Aggregate
• groupByArea()
• asTracks()
Transformation
• s2CellIDToken()
• toRows()
Supported Shapes
• box - defined by: minLat, maxLat,
minLon, maxLon
• circle - defined by: lat, lon, radius
• polygon - array of points: lat, lon
Telegraf Plugins
decimal degrees.
longitude.
Map Visualisation

| Key Themes in 2021
Teams & Collaboration
Tenant Observability
Functions & Reuse
Ease Data Movement

| Notebooks
● Goal is to provide a single
consistent building
experience in our platform
● Designed to allow teams to
incrementally build queries
or background processing
tasks together
● Flexible enough to guide
new users and unblock
power users
● View multiple related
resources in a single view
Now

| Dashboard Annotations
● Goal is share context in
dashboards
● Add, edit, delete as
needed
● Annotations never expire
● Available in your InﬂuxDB
Cloud accounts soon
● Ranged annotations,
custom colors, support
for more graph types
coming soon
Now

| Tenant Observability
● Goal is to provide
self-service access to
usage and operations:
○ Usage
○ Cardinality
○ Rate limit events
○ Write errors
○ Query Log
● User conﬁgurable
templates for dashboards
& alerts
● Available on all cloud
accounts shortly
Next

| Evolution of Tasks
Schedule
Tasks: Flux
Now

| Evolution of Tasks
Functions
Schedule
Trigger
(via Events)
Tasks: Flux
Parameterized /
Reusable
Remote Invocation
Task
(Via Schedule)
Web hook
(via API)
Flux-based...to Start
Next

| Functions: Flux
● Goal is code reusability
○ Define Flux code once
○ Register as a Function with
defined parameters
○ Invoke, as desired, with
different values supplied as
arguments
● Power tasks, triggers, dashboard
cells with common functions
Next
Task = Function (Schedule, Arguments)

| Triggers
● Goal is to use events to invoke
functions
● Events can be “write to a bucket”
● Use cases:
○ Late arrivals of data
○ Streamlined alerting
Next
QUERY
ACT
Scheduled Task / Alert check
Notiﬁcatio
n systems
WRITE
Notiﬁcatio
n systems
WRITE
WRITE
ACT
Triggers

| Functions: Polyglot Programming
● Goal is make the platform do more
for you
● Two main use cases:
○ Pre-process data before writing
into InﬂuxDB
○ Post-process rows returned
from querying InﬂuxDB
● MVP - Run your python code as
function that can be invoked
remotely
Next
Python code as a
Function

|Make data movement easier
Enable users to deﬁne object store locations within their CSP account.
Bulk Import
Line protocol
CSV
TSM ﬁles
Parquet
Next

Bulk Import
Line protocol
CSV
TSM ﬁles
Parquet
Self-Service
Backup
Bucket backup
(restore to OSS or Cloud)
-- or --
Export to Parquet
Next

Bulk Import
Line protocol
CSV
TSM ﬁles
Parquet
Self-Service
Backup
Bucket backup
(restore to OSS or Cloud)
-- or --
Export to Parquet
Next
Cloud Native
Ingest
Connect with cloud
native messaging
services
• MQTT,
• AWS Kinesis
• Azure EventHub

|Bonus: Explicit Schema
Deﬁne Buckets with Explicit Schema
● Blocks non-compliant writes
● Allow for additions
Next

|
InfluxDB Enterprise
○ Continue the success of deployments managed by you
○ Deliver latest improvements for data access & processing
InfluxData Community
○ Limo door is open...and ready to take you on the ride to InfluxDB 2.x
○ Telegraf advancements makes it the most powerful data acquisition tool ever
InfluxDB Cloud
○ Teams & Collaboration
○ Functions & Reuse
○ Tenant Observability
○ Ease Data Movement
Summary

Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021

Similar to Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021 (20)

More from InfluxData

More from InfluxData (20)

Recently uploaded

Recently uploaded (20)

Tim Hall and Ryan Betts [InfluxData] | InfluxDB Roadmap and Engineering Update | InfluxDays EMEA 2021