Safer Commutes & Streaming Data | George Padavick, Ohio Department of Transportation (sponsored by Confluent)

•

0 gefällt mir•516 views

The Ohio Department of Transportation has adopted Confluent as the event driven enabler of DriveOhio, a modern Intelligent Transportation System. DriveOhio digitally links sensors, cameras, speed monitoring equipment, and smart highway assets in real time, to dynamically adjust the surface road network to maximize the safety and efficiency for travelers. Over the past 24 months the team has increased the number and types of devices within the DriveOhio environment, while also working to see their vendors adopt Kafka to better participate in data sharing.

Technologie

EVENT STREAMING PLATFORM (ESP)
George Padavick, ODOT ESP Manager

INTRODUCING OUR ESP TEAM
George Padavick, ODOT ESP Manager
Mihail Chirita & Jeff Lutz, Architects
Tom Keller, Project Manager
Keshav Ramakkagari, Developer
Mahes Nayak, DBA
Ed Mack, Business Analysist

CHALLENGE
o Processing large amounts in real time
o Volume increasing significantly
End goal: increase safety and efficiency
DATA
FROM WIDE VARIETY OF EDGE DEVICES & SYSTEMS

EVENT STREAMING PLATFORM
o Ingesting, processing & disseminating
o Allowing Public & Private entities
to share, publish or consume
GOALS
A REAL-TIME TRANSPORTATION DATA PLATFORM
Supporting Drive Ohio Initiatives &
ODOT operational needs

5
ESP - TECHNOLOGY STACK - CLOUD COMPONENTS
Hosting/Infrast
ructure
Containers
kubectl
deployer
Persistence
Observability
Streaming
Infrastructure
KSQL
Clusters
Connect
Clusters
Schema
Registry
Confluent
Operator
Confluent Control
Center
Applications
Inputs
Pull|EDGE|Push
Processing
KSQL|Kafka Streams|Custom
Visualizations
Kibana|Grafana|Kepler GL|Custom
Outputs
Push|REST|Stream

WHAT WAS BUILT
o Two principles
o Modular - can be used independently
o Generic - can be used for many use cases
PLATFORM
AKA “THE SHOPPING MALL”
ACCELERATOR FOR BUILDING AND OPERATING APPLICATIONS
Applications aka "the stores" to solve
specific business needs

APPLICATIONS
o Inputs - publishing data
o Outputs - consuming the data
o Processing - business logic
o KSQL, Kafka Stream or Custom programs
o Visualizations - exploring data
o Kibana, Grafana, Kepler GL, or Custom

INGESTION OF DATA
1) Pull
Data Producer
• Support multiple
protocols.
• No code - entirely
configurable
EdgeX +
MQTT Cloud +
MQTT Connector
Minimal
requirements form
publishing data for
both internal and
external users
2) Push
3) Push via Rest API

DATA OUTPUT
1) Push - Connectors
Data Consumer
• Support multiple protocols.
• No code - entirely configurable
Expose secure REST API to external consumers
Allow external entities to subscribe to streams
2) Pull - REST API
3) Stream - REST API

10
APPLICATIONS - CLOUD PROCESSING
o KSQL
olow coding
oextensible language
ostill evolving/missing features
oKafka Streams
opowerful library for streaming applications
oneed to code in Java or JVM language
oCustom
omost flexibility
orequires coding - any language

11
APPLICATIONS - EDGE PROCESSING
oEdgeX/Kuiper
ouse sql to define processing
oconfigurable mechanism to send data to MQTT
broker

12
APPLICATIONS - VISUALIZATIONS
o Kibana/Grafana
o drag and drop data visualizations/dashboards
o some geo-spatial support
o KeplerGL
o geospatial analytic visualizations and insights
o open sourced by Uber
o Custom
o build custom UI functionality

13
SUMMARY
Edge Devices via
EdgeX/Kupier
MQTT

Weitere ähnliche Inhalte

Was ist angesagt?

Many organizations have chosen to go with a hybrid cloud architecture to give them the best of both worlds: the scalability and ease of deployment of cloud, and the security, latency & egress benefits of local storage. Persistence of data on such an architecture can follow a write-back mode, where data is first written to local storage, and then uploaded to cloud asynchronously. However, this means that the applications cannot utilize the availability and durability guarantees of cloud, and the availability of storage is the availability SLA of on-premise storage, which is almost always less than the availability SLA of Cloud. By switching the order, i.e. performing uploads to cloud, and then hydrating on-premise storage, applications get the benefit of availability SLAs of cloud. In our case, this allowed us to move from three 9’s of availability (99.9%) of local storage to four 9’s (99.99%). Instead of uploading in write-back mode, we duplicated the incoming stream to upload to both cloud and on-premise. For on-premise uploads that failed, we leveraged Kafka’s event processing to queue up objects that need to be egressed out of Cloud into the local storage. This architecture allowed us to hydrate the local storage with objects uploaded to Cloud. Furthermore, since local storage space is limited, we periodically purged data out of local storage and created a secondary copy of the data on cloud by leveraging Kafka event processing.

Going from three nines to four nines using Kafka | Tejas Chopra, Netflix

HostedbyConfluent

Transaction Banking from Goldman Sachs is a high volume, latency sensitive digital banking platform offering. We have chosen an event driven architecture to build highly decoupled and independent microservices in a cloud native manner and are designed to meet the objectives of Security, Availability Latency and Scalability. Kafka was a natural choice – to decouple producers and consumers and to scale easily for high volume processing. However, there are certain aspects that require careful consideration – handling errors and partial failures, managing downtime of consumers, secure communication between brokers and producers / consumers. In this session, we will present the patterns and best practices that helped us build robust event driven applications. We will also present our solution approach that has been reused across multiple application domains. We hope that by sharing our experience, we can establish a reference implementation that application developers can benefit from.

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...

HostedbyConfluent

In this talk, we'll discuss how VillageMD is able to use Kafka topic compaction for rapidly scaling our reprocessing pipelines to encompass hundreds of feeds. Within healthcare data ecosystems, privacy and data minimalism are key design priorities. Being able to handle data deletion in a reliable, timely manner within event-driven architectures is becoming more and more necessary with key governance frameworks like the GDPR and HIPAA. We'll be giving an overview of the building and governance of dead-letter queues for streaming data processing. We'll discuss: 1. How to architect a data sink for failed records. 2. How topic compaction can reduce duplicate data and enable idempotency. 3. Building a tombstoning system for removing successfully reprocessed records from the queues. 4. Considerations for monitoring a reprocessing system in production -- what metrics, dataops, and SLAs are useful?

Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...

HostedbyConfluent

Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Kafka Streams and Apache Flink.

It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko, Shopify

HostedbyConfluent

To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

HostedbyConfluent

Speaker: Yupeng Fu, Staff Engineer, Uber High availability and reliability are important requirements to Uber services, and the services shall tolerate datacenter failures in a region and fail over to another region. In this talk, we will present the active-active Apache Kafka® at Uber and how it facilitates disaster discovery across regions for Uber services. In particular, we will highlight the key components including topic replication, topic aggregation, offsets sync and then walk through several use cases of their disaster recovery strategy using active-active Kafka. Lastly, we will present several interesting challenges and the future work planned. Yupeng Fu is a staff engineer in Uber Data Org leading the streaming data platform. Previously, he worked at Alluxio and Palantir, building distributed data analysis and storage platforms. Yupeng holds a B.S. and an M.S. from Tsinghua University and did his Ph.D. research on databases at UCSD.

Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber

confluent

Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highl...

confluent

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Ankur Bansal

Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.

One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...

HostedbyConfluent

Watch this talk here: https://www.confluent.io/online-talks/siem-modernization-build-a-situationally-aware-organization-with-apache-kafka Of all security breaches, 85% are conducted with compromised credentials, often at the administration level or higher. A lot of IT groups think “security” means authentication, authorization and encryption (AAE), but these are often tick-boxes that rarely stop breaches. The internal threat surfaces of data streams or disk drives in a raidset in a data centre are not the threat surface of interest. Cyber or Threat organizations must conduct internal investigations of IT, subcontractors and supply chains without implicating the innocent. Therefore, they are organizationally air-gapped from IT. Some surveys indicate up to 10% of IT is under investigation at any given time. Deploying a signal processing platform, such as Confluent Platform, allows organizations to evaluate data as soon as it becomes available enabling them to assess and mitigate risk before it arises. In Cyber or Threat Intelligence, events can be considered signals, and when analysts are hunting for threat actors, these don't appear as a single needle in a haystack, but as a series of needles. In this paradigm, streams of signals aggregate into signatures. This session shows how various sub-systems in Apache Kafka can be used to aggregate, integrate and attribute these signals into signatures of interest. In this talk you will learn: -The current threat landscape -The difference between Security and Threat Intelligence -The value of Confluent platform as an ideal complement to hardware endpoint detection systems and batch-based SIEM warehouses

SIEM Modernization: Build a Situationally Aware Organization with Apache Kafka®

confluent

Core banking systems are batch oriented: typically with heavy overnight batch cycles before business opens each morning. In this talk I will explain some of the common interface points between core-banking infrastructure and event streaming systems. Then I will focus on how to do stream processing using ksqlDB for core-banking shaped data: showing how to do common operation using various ksqlDB functions. The key features are avro-record keys and multi-key joins (ksqlDB 0.15), schema management and state store planning.

Use ksqlDB to migrate core-banking processing from batch to streaming | Mark ...

HostedbyConfluent

This session will dive into our most successful (and unsuccessful!) multi-cluster event replication patterns. An x-ray of the cross cluster distribution model that powers our globally distributed APIs will touch on the benefits that this model has provided in terms of client API experience, delivery agility and developer experience. We will focus on recipes for effective use of Mirror Maker event replication to power platform distribution including the challenges of managing a 'fan in' event replication workflow - pulling events created in satellite clusters back to a mothership cluster for processing. We will introduce the elegant technique of replication event multiplexing - which can be used to simplify the burden of managing a 'fan-in' replication topology by eliminating regional awareness from the application domain and improving replication health monitoring & observability.

Fan-out, fan-in & the multiplexer: Replication recipes for global platform di...

HostedbyConfluent

Mark Teehan, Principal Solutions Engineer, Confluent Use the Debezium CDC connector to capture database changes from a Postgres database - or MySQL or Oracle; streaming into Kafka topics and onwards to an external data store. Examine how to setup this pipeline using Docker Compose and Confluent Cloud; and how to use various payload formats, such as avro, protobuf and json-schema. https://www.meetup.com/Singapore-Kafka-Meetup/events/276822852/

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

confluent

Couchbase Cloud No Equal (Rick Jacobs, Couchbase) Kafka Summit 2020

HostedbyConfluent

When all your stores are closed, e-commerce becomes your bigger store… and the most challenging. That means a myriad of systems orchestrated to make it happen, all of them scaling out accordingly and implementing Observability and SRE practices to support this growth, preserving stability and reliability. How can we detect problems, root causes and react? How can we predict those problems? HOLMES is the adidas solution to accelerate problem detection, giving a holistic view of technical systems through metrics and logs democratisation. In this talk, we'll show how Kafka technology stack allows adidas to support the ingestion and offload of all logs and metrics of the company. A platform which adoption has skyrocketed during 2020, supporting 100 Billion messages per day. The main takeaway will be the explanation of a cutting-edge solution based on kafka technology stack (kafka, Kafka Streams and Kafka Connect) for demanding throughput ecosystem.

Navigating the obdervability storm with Kafka | Jose Manuel Cristobal, Adidas

HostedbyConfluent

dA Platform is a production-ready platform for stream processing with Apache Flink®. The Platform includes open source Apache Flink, a stateful stream processing and event-driven application framework, and dA Application Manager, a central deployment and management component. dA Platform schedules clusters on Kubernetes, deploys stateful Flink applications, and controls these applications and their state.

dA Platform Overview

Robert Metzger

At Stripe, we operate a general ledger modeled as double-entry bookkeeping for all financial transactions. Warehousing such data is challenging due to its high volume and high cardinality of unique accounts. aFurthermore, it is financially critical to get up-to-date, accurate analytics over all records. Due to the changing nature of real time transactions, it is impossible to pre-compute the analytics as a fixed time series. We have overcome the challenge by creating a real time key-value store inside Pinot that can sustain half million QPS with all the financial transactions. We will talk about the details of our solution and the interesting technical challenges faced.

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...

HostedbyConfluent

A stream processing platform is not an island unto itself; it must be connected to all of your existing data systems, applications, and sources. In this talk we will provide different options for integrating systems and applications with Apache Kafka, with a focus on the Kafka Connect framework and the ecosystem of Kafka connectors. We will discuss the intended use cases for Kafka Connect and share our experience and best practices for building large-scale data pipelines using Apache Kafka.

Data integration with Apache Kafka

confluent

Was ist angesagt? (20)