This document summarizes a presentation about implementing a digital government messaging platform in the Netherlands using microservices and an event-driven architecture. The key points are: - The Dutch government implemented a centralized messaging platform called BBO to digitally deliver notices from various government agencies and utilities to citizens. - BBO uses a microservices architecture with independent services for receiving, validating, sorting, notifying and retrieving messages. It also leverages event-driven design with Kafka to decouple services. - The system is highly scalable, secure, available and auditable. It processes over 5 million messages per day from over 600 senders to 9 million users. - Implementing an event-driven micro

1. You’ve got Mail!
Michaël van der Haven

2. © 2023 CGI Inc. Public
Government Goes Digital!
All communications for citizens and organizations
from Goverment (central and decentralized)
and Public Utilities is made available through digital means
2

3. © 2023 CGI Inc. Public 3
• Founded 1976
• Consultants and professionals 90,250
• Locations worldwide 400
• F2022 revenue C$12.87B
• End-to-end services clients globally 5,500
• Clients through our IP-based solutions 50,000
• Stock symbols:
– NYSE: GIB
– TSX: GIB.A

4. © 2023 CGI Inc. Public 4
About me
Michaël
van der Haven
Vice President Consulting Expert
(Renewable) Energy
CGI

5. © 2023 CGI Inc. Public
Digital Government: In Practice
• The Blue Envelope
– The most infamous letter in The Netherlands
– Invite to pay taxes and all other associated tax
correspondations
• But also
– Pension notices
– Compensation schemes
– Reminders to mandatory car inspections
– Vaccination invites
– Local municipality maintenance notices
– Etc.
5
• 600+ ‘senders’
• 9+ million users
• 5+ million messages sent
per day
• 4+ million messages requested
per day

6. © 2023 CGI Inc. Public
This excuse does not work anymore
6

7. © 2023 CGI Inc. Public
BBO: The Messaging Platform
Berichtenbox voor Burgers en Ondernemers
(message-box for citizens and organizations)
• Stakeholders send their messages
to a central location: BBO
– Receives
– Validates
– Sorts
– Notifies
– Makes Available
• Manages all legislation topics
– Retention (archival law)
– Locking (legal disputes)
– Audit
– Etc.
7

8. © 2023 CGI Inc. Public
Microservices
8
Requirements
(non
functional)
Highly secure
Allways on
Auditable
Allways Available à No impact
by performance demands
Scalable Horizontal
Vertical
Reuse of resources
Monitoring
Self-healing

9. © 2023 CGI Inc. Public
Still Microservices?
Prevent congestion at all costs
• Receive
– Senders don’t want to wait until
you’re done sorting each message
including that of other senders
• Process
– Validation and sorting can be at
a different speed: messages often
have a future publication date
• Serve Requests
– Citizen doesn’t want to be impacted
by others or messages that are
being sent
9

10. © 2023 CGI Inc. Public
Yes: Microservices, not nano-services
• Different Service Levels
• Different Kinds of services
• Completely independent (i.e. not the Amazon Prime use-case)
10
Receive
Receive
Receive
Validate
Sort
Validate
Sort
Notify
Retrieve
Retrieve
BBO

11. © 2023 CGI Inc. Public
© 2023 CGI Inc. 11
Event Driven
The Kafka Case
• Microservices with different scaling
requirements
• More than one interested
consumer
• Microservices should only ‘act’
when something becomes relevant

12. © 2023 CGI Inc. Public
Zooming in
1. A receiving service handles a message or
batch of messages
a) Stores the message in S3 (on-premise MinIO)
b) Post an event that initial reception is done
2. RPO=0 Requirement
Buckets in multiple data centers
3. Publish S3 Replication Events
4. Kafka Stream combines to Received
RPO=0 Guaranteed
5. Acknowledgement to sender
(your message is safe! We will handle the rest)
6. Other consumers
(Validated, Reporting, Audit Log)
will consume the same stream and follow with their
own business actions
12
Receive Validate Report Audit
Inbox
Initial Reception
Stored and
Replicated
Received
1
1a
1b
2
3
4
5 6

13. © 2023 CGI Inc. Public
Event Driven: Self Healing System
Utilizing Event Driven Concepts (and a lot of Kubernetes)
to automatically recover from failure
Event Driven
• Store the state of events
• Idempotent
• Each process results in
exactly the same result
Replay
• Process events again
• Anything new ‘after’ the
snapshot is processed
• Idempotency
• Downstream services ‘know’
that the event does not have
to be processed again
Kubernetes
• State store snapshots
• Persistent Volumes
• Snapshots
• Recover from an older
snapshot in case of
detected failure
13

14. © 2023 CGI Inc. Public
The proof of the pudding is in eating
• Replay as part of deployment
– Z-Shaped deployments
– Prove that a solution can always be
rolled back
– Deploy into representative environment
– Test on actual data
– Roll back
– Play again and show that the complete
system still works
14

15. © 2023 CGI Inc. Public
Replay and Snapshots
• Kubernetes & MongoDb state database
– State:
§ MongoDb to store which events have been processed
(yes the collection is very simple)
– Kubernetes Snapshots
§ Each snapshot virtually points to a location on a Kafka
topic
– Kubernetes Hydrating a new PV or existing PV
§ The new MongoDb server has state data that points to
a location on the topic that the original service had not
processed yet
– Microservice & Kafka à Replay
§ Deploy a new service and let it work on the new
database
§ The service will automatically ‘replay’ everything on the
topic
15
t
t(0) t(1) t(n)
t(n-1)

16. © 2023 CGI Inc. Public
© 2023 CGI Inc.
• Yes, you can do everything yourself
• It is a lot of Ops work though
• Managed platform
• Axual
• Main Drivers:
– Topic and Connection Overview
– Integration with Keyvault systems
– Programmable Part of CI/CD
– Clear DTAP management
• Disadvantage: Vendor specific
(and this is a problem with all commercial Kafka providers)
• Operators
• KSQL types of languages
• Schema management
• Etc
Managing Kafka
16
DIY or Managed solution?
• Normal Topics (BBO: 10+)
• Streams (BBO: 20+)
• Operational tasks and challenges
• Security (20+ services)
• Performance
• Reuse of Kafka outside of BBO

17. © 2023 CGI Inc. Public
Developer Experience
Perception & Skills
• Microservices are not easy
• Event Driven is not easy
Developers (non) comforts:
• Atomic services à Balancing act, not
too atomic
(recall the Amazon Prime case)
• Limited scope of a service
• Object Oriented across services
• Embrace reuse of code in the form of
copy-paste
Or
• Excellent library management
Testing
• Consumer Driven Contract Based
Testing with Events
It is too tempting to let the producer
rule
DevOps
• CI/CD Pipeline
– Templates
– Branching merging
– Etc.
– Getting used to The C truly becomes
continuous
17

18. © 2023 CGI Inc. Public
Insight is a gamechanger
• Developers may be working in an atomic service
• Atomic or not: if another service is not doing its
job the system fails
• Provide insight in
– What topics are available
– Who is producing and who is consuming?
18
Axual Metro Map
• Results
– Finding bugs
– Finding dead topics
– Finding producers without consumers and vice
versa
– Fix and export and make it part of CI/CD

19. © 2023 CGI Inc. Public
Current State
• Matured Team
– Well Established testing framework including PACT
– Z-Shaped Deployment
– Secure
– Insight!
• Performance?
– Kafka
§ Bare minimum set of brokers for RPO=0
§ 30+ MM events per day
à Kafka walk in the park
– Services
§ Vertical Scaling Challenges
§ Internal Optimization
§ Fewer functionalities
§ How micro is micro?
19

20. © 2023 CGI Inc. Public
Conclusions and Lessons Learned
20
• An event driven architecture works
– Supports the Micro Service Pattern
– Self Healing System
– Highly performant and scalable
– Ideal to ‘detect & manage’ RPO=0 requirements
– Kubernetes, Kafka and MongoDb are a golden combination
• An event driven architecture is hard for developers
– Invest in training
– Have seniors that can teach on the job
– Shared code management becomes more challenging
– A system that provides an overview is a gamechanger!
• Managed Platform vs DIY Open Source
– DIY
§ Using Kafka with two or three services can perfectly be done with open source
– Managed
More services, more streams, more topics AND security:
§ Managed platform
§ Axual was a front-runner with their management platform
§ The competitive landscape is on-par
§ Choose your poison
o Operators
o Streaming Language

21. cgi.com
Insights you can act on
Founded in 1976, CGI is among the largest IT and business
consulting services firms in the world.
We are insights-driven and outcomes-based to help accelerate
returns on your investments. Across hundreds of locations
worldwide, we provide comprehensive, scalable and sustainable IT
and business consulting services that are informed globally and
delivered locally.

22. cgi.com
Q&A