Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jonathan Schabowsky, Solace) Kafka Summit 2020

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 14 Anzeige

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jonathan Schabowsky, Solace) Kafka Summit 2020

Herunterladen, um offline zu lesen

As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jonathan Schabowsky, Solace) Kafka Summit 2020 (20)

Anzeige

Weitere von HostedbyConfluent (20)

Aktuellste (20)

Anzeige

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jonathan Schabowsky, Solace) Kafka Summit 2020

  1. 1. How to Discover, Visualize, Catalog, Share and Reuse your Kafka Event Streams Jonathan Schabowsky Senior Architect, CTO, Solace
  2. 2. Agenda 2 • Your growing Kafka event streams • Flying blind • Event portal: an API portal for event streams • Demo: How to discover, visualize, catalog, share and reuse your Kafka event streams • Summary © Solace 2
  3. 3. Does Your Organization Look Like this? 3 © Solace Data Stream Flow Legacy Application Infrastructure …. ? ? ? JSON, XML, Plain Text… IoT/Device Management Producers Consumers Connectors Stream Processors Avro
  4. 4. You’re Flying Blind… Where do you discover event streams for reuse? Why does a given event stream exist? What topic do you use to subscribe to it? What changed? How do you determine the data structure of its payload? Who should have access to it? Who made the latest revision? Who can tell you more about an event? Is your change backward compatible? Who is impacted by your change? Does it comply with security policy? 4 © Solace
  5. 5. Some Try Solutions Not Intended for Event Stream Management • It's a pain… – No single place to discover and share/reuse event streams – No best practices guidance • You’re breaking systems and don’t know who changed what??? – Suffer from changes with unintended consequences – Hard to audit changes when data is stored in different tools Brittle, complex, manual solutions Custom Developed Catalog PowerPoint API Management Spread sheets GitConfluence 5 © Solace
  6. 6. API Management Solved this for REST 6 © Solace
  7. 7. They answer Who, What, When, Where, Why and How For RESTful APIs… but what about event streams? Inspired by API Management Platforms 7 © Solace API Gateway Documentation • Discover Registration • Govern Analysis • Improve • Monetize Community • Collaborate Architects and Developers (Internal, Partner and Third-Party) Productivity • Generate Code API Portal
  8. 8. Sinks “While most organizations have basic event processing infrastructure - such as notification services, message buses, and event brokers - many don’t have the high-level productivity tooling that helps developers design, develop, test and manage event-centric applications. Nor do they have tools for governing, publishing and managing event-based interfaces. The market for these tools is sparse.” Gartner “Top 3 Trends in Application Architecture That Enable Digital Business” Anne Thomas, Yefim Natis, Mark O’Neill, 28 Oct 2019 8 © Solace Event Broker Security Persistence Protocol Translation One-to-Many Distribution Source Runtime We Need an Event Portal for Asynchronous Interactions Design, Develop and Manage Event Portal Documentation • Discover Registration • Govern Analysis • Improve • Monetize Community • Collaborate Productivity • Generate Code (code gen) (spec)
  9. 9. 9 © Solace Single place to design, create, catalog, visualize, discover, share, secure and manage all events within your ecosystem Introducing Event Portal Event Portal Mainframe/ SAP Distributed Apps IoTMicro services Server Less Event Brokers
  10. 10. Event Portal How Users Interact with Event Portal 10 Event Management (Design Time) © Solace Your Event-Driven Enterprise B2B/B2C Ecosystem IoTMicro- services Server- less Generate Code REST API Existing Tools • Schema registries • Data governance • CI/CD pipelines Customers & Partners • Discover and consume events in real-time Architects • Discover, create, update, review apps, events and payload schemas • Map payload schemas to events; and events to apps • Govern event consumption across app domain boundaries Developers • Collaborate with architects on design • Export AsyncAPI app contracts for each app • Generate and implement event-driven apps that provide business value Data Scientists • Understand event-driven data • Discover new insights by combining events
  11. 11. Demo 11 © Solace
  12. 12. Benefits of an Event Portal 12 Solace Your business can… • Understand who is consuming which events, how much, for what • Create value-added services and gain insights by combining events • Make it easy for users across app teams, LoBs and partner ecosystem to find and reuse events • Monetize popular or particular high-value event streams Reuse/Sharing Architects & developers can: • Collaborate on events and event-driven apps • Visually see event-driven interactions to understand impact of changes • Low Code: Generate app code by exporting AsyncAPI definitions • Benefit from best practices & consistently apply conventions Productivity CDO/data governance can: • Understand where data is coming from (lineage) and going • Track changes and audit for deviations while promoting apps and dependencies though Dev, Test, QA, and Prod • Use application/event relationships to create security policies and validate compliance with schema Control
  13. 13. So, if you… Have a lot of event- driven applications 13 © Solace You need an event portal! Have (or need to define) a lot of event streams Want to manage events as easily as you manage your APIs
  14. 14. Event Portal = API Portal for Events/Streams One tool to design, create, catalog, visualize, discover, share, secure and manage all of your events and streams (including Kafka and any other event broker) 14 © Solace Sign up for free! • About data and apps, NOT managing Kafka infrastructure • High-level productivity tooling to help you fully embrace EDA • Governance of apps, event and data

×