Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Work is a Stream of Applications (Audun Strand, NAV) Kafka Summit London 2019

96 Aufrufe

Veröffentlicht am

NAV handles a lot of applications for our benefits(sickness, unemployment etc.). We are rewriting the systems handling these applications to a stream based approach, placing Kafka-streams in the centre of our architecture. This talk will go through our approach to building these systems. We are mostly using Kotling as our programming language, and are running our applications on Kubernetes. We use Change-data-capture to extract data from our legacy systems, to ease the pain of migration. By treating our data, and our applications as streams we get a lot of benefits. We can rerun our applications with different rules, to simulate the effect of policy changes. We can do real time evaluation of applications, to improve the user experience. We also get better robustness, as we remove the runtime dependency to our core systems, and can receive applications even when other parts of the systems are down. The talk will also present the architecural rules we are implementing for NAV, using streams as the main communication pattern between organization units, and some learnings from our current work on a data platform to support Life is a stream of events.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Work is a Stream of Applications (Audun Strand, NAV) Kafka Summit London 2019

  1. 1. Work is a stream of applications Audun Fauchald Strand Principal Engineer Norwegian Work and Welfare Administration @audunstrand
  2. 2. The very first Kafka Summit
  3. 3. Agenda NAV Legacy architecture “The Fred George Way” The moderates Data in NAV
  4. 4. Biggest organisation in the norwegian welfare system 19 000 employees Pays out a third of the national budget in Norway Responsibilities similar to 15-20 agencies in the US - Pensions - Unemployment benefits - Parental Leave - Sickness Benefits
  5. 5. Life is a stream of events
  6. 6. LEGACY Old systems (up to 40 years), that are critical infrastructure for Norway Huge potential in digitalisation - More efficient - Personalization - Faster and fairer case-management - Simulations - (Big) Data
  7. 7. Traditional flow Paper based Applications Mainframe for manual automation Electrical paper (PDF) Some Web Applications with richer interface Data Security Authorization is very important
  8. 8. Data Architecture Data is grouped because they “belong together” Common Databases holds data from other agencies Separate teams owns those databases, but data is used by others IBM MQ for eventing Service Bus for … Big Separate Analytics operation, traditional DWH Slow pace, lots of coordination
  9. 9. Introducing Kafka Increase development pace Use more data better On Premise installation Support from Confluent Simple small cluster
  10. 10. Early Use Cases Replacing MQ for some events Change Data Capture on old systems - Golden gate from Oracle - InfoSphere from IBM Small POCs - Asynchronous microservices with streams - Pipelines into analytics with connect
  11. 11. Security Teams owns their data and thus the topics and approves consumers Some issues with Connect and Streams at first Azure AD not supported, built our own autorisation plugin
  12. 12. Change developer behavior Training developers Changing mindsets Making the right behaviour easier Focus on producing data, not use cases
  13. 13. Fred George - fredgeorge@acm.org
  14. 14. The Fred George Way Rapids and Rivers Optimize for Speed Need-pattern Tiny microservices
  15. 15. Basic Architecture One topic for everything Publish a packet All consumers listen for all packets and filters based on content Choreography based on packet content Append data to packet
  16. 16. Divide system into subdomains River Rapids
  17. 17. Example Unemployment benefit Strangulation of old legacy system Building new stuff on the outside
  18. 18. Packet PROBLEM Integer READ_COUNT Timestamp START NEED Rule 1 - data Rule 2 - data
  19. 19. The moderates Asynchronous microservices kafka-streams for implementing the services Data flows through the system
  20. 20. Comparison The Fred George Way Smaller autonomous components Can easily add new features Difficult to understand the whole picture The Traditional way Easier to reason about Good fit for kafka streams Shareable data for free Tighter coupling between services Bigger services
  21. 21. Life is a stream of events
  22. 22. Software is policy Implementing the rules from the books is the easy part Getting the data is the difficult part Traditional integration architecture works against speed Software with changeability is a political asset.
  23. 23. Inverting the (central) databases Where People work What they earn Person Data Master data as streams Applications populate minimal database for their use of the data
  24. 24. Sharing all decisions Events are created in a distributed fashion Applications received Decisions decisions made Undecided: - Common format? - Producer or consumer responsible?
  25. 25. Reverse Conway Maneuver NAV was built for projects Product Areas Changing the organization - Move central registries to domain teams - Group some domains into areas Conway's law. Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations. The law is based on the reasoning that in order for a software module to function, multiple authors must communicate frequently with each other.
  26. 26. Sharing data across service areas Service areas are a collection of domains Freedom inside an service area Only Communication between service areas using streams Avro and unlimited retention Master Data distributed as streams Data on the inside, data on the outside
  27. 27. Commands vs Events Events Describing what has happened Publish/Subscribe - N consumers “Application Received” Command Expects a change Probably only one consumer Seldom across areas “Execute Payment”
  28. 28. Building a Data Platform Common solutions to problems Dedicated experts Catalysts for change Make the platform a product
  29. 29. GDPR Data Catalogue Consumers must have a legal reason for reading data Document this reason outside code, but automatic Logic on the consumer side to filter out message types Log Compaction for deletion of user-data.
  30. 30. Sharing data on a national level Data originating in one organisation can be important for another organisation Kafka, as it is now, is not really suited for communication between organisations Alternatives exists like atom-feeds over http Future: Separate team that hunts life events, and makes them available as streams for everyone
  31. 31. CentralPerson register Atom feeds Owned by Norwegian Tax Authorities, a separate organisation Data used by almost every public organisation, and many private Slow rate of change - suitable for caching
  32. 32. Next step
  33. 33. Key points Moving from synchronous to asynchronous is not easy - Change of mindset! - Expose the data! Streams work on both application and system level Make it a platfor
  34. 34. audunstrand@gmail.com @audunstrand Questions?