Real-Time Market Data Analytics Using Kafka Streams
•
3 gefällt mir•11,022 views
(Lei Chen, Bloomberg, L.P.) Kafka Summit SF 2018 At Bloomberg, we are building a streaming platform with Apache Kafka, Kafka Streams and Spark Streaming to handle high volume, real-time processing with rapid derivative market data. In this talk, we’ll share the experience of how we utilize Kafka Streams Processor API to build pipelines that are capable of handling millions of market movements per second with ultra-low latency, as well as performing complex analytics like outlier detection, source confidence evaluation (scoring), arbitrage detection and other financial-related processing. We’ll cover: -Our system architecture -Best practices of using the Processor API and State Store API -Dynamic gap session implementation -Historical data re-processing practice in KStreams app -Chaining multiple KStreams apps with Spark Streaming job
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Real-Time Market Data Analytics Using Kafka Streams
Ähnlich wie Real-Time Market Data Analytics Using Kafka Streams (20)
Mehr von confluent
Mehr von confluent (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Real-Time Market Data Analytics Using Kafka Streams
- 1. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Lei Chen Software Engineer, Team Lead lchen576@bloomberg.net Real-time* Market Data Processing Using Kafka Streams * Actually, just low-latency
- 2. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Agenda • Use cases and challenges • Why Kafka Streams • Deep dive into our implementation • Some other tips & tricks
- 3. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Who we are & What we do • Derivative market data group in Bloomberg • Builds market data pipelines • Apply big data & AI technologies in financial domain
- 4. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Streaming use cases Market movement (bid/ask/trade) Composite price (Bloomberg Generated Market Indicator) Option price (Calculate option price using calculated volatilities)
- 5. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Challenges and why Kafka Streams • Zero data loss • Ultra-Low latency • Huge data volume • Large state • Corporate DR compliance • Maintenance ‘Exactly once’ delivery Super fast Highly scalable State store Fault tolerant Minimal management overhead *Data already in Kafka!
- 6. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Deep dive into our implementation
- 7. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Key takeaways • transform()/process() combines the best of both DSL and PAPI • Kryo for state serialization/deserialization • Think stream/table duality • Avoid unnecessary DSL call by accessing state directly • Use Kubernetes as runtime • Monitoring is important (leverage built-in metrics)
- 8. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. DSL vs Processor API • Declarative vs Imperative • Usability vs Flexibility • High-level API vs low-level programming model
- 9. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Monitoring • Built-in webserver • Queryable state • Metrics • Internal topics • Lags (topic/partition)
- 10. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Hot load configuration KStream KTable Stream & Table Join Joined Stream KStream GlobalKTable Joined Stream Stream & Table Join Conf. topic Data topic Comdb2 Kafka Connect Conf. topic Data topic DB Kafka Connect KStream KTable state
- 11. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Community Involvement • KIP-362 - Dynamic Gap Session Window — Versus fixed-gap session window session1 session2 session3 gap3gap2gap1
- 12. © 2018 Bloomberg Finance L.P. All rights reserved. Some other tips & tricks
- 13. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Thread model and depth first topology APP KAFKA STREAMS topology
- 14. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Trick 1 - Batch Processing In Kafka Streams 10/15/18 Micro batch? Possible! Batch? Harder! watermark State App2 App1
- 15. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Trick 2 - Chain Multiple Kafka Streams App • Kafka as message bus • Compose pipeline using multiple Kafka Streams apps • Could leverage third-party pipeline framework – Spring Cloud Data Flow, etc. KSTREAMS APP1 topology KSTREAMS APPN topology KSTREAMS APP2 topology
- 16. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Our Streaming Platform
- 17. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. We are hiring! Questions? https://tinyurl.com/y7bepre9