This document discusses how NServiceBus was used to build a highly scalable file processing platform. It describes some initial problems with the system design, such as monolithic endpoints that could not scale and complex sagas. It then outlines how the system was refactored into granular components with distinct responsibilities, simplified sagas using a passive observer pattern, and automated deployment. This resulted in a system that could process over 10,000 records per second per machine and provided full visibility, no data loss, and faster onboarding for new clients.

1. Building a Highly Scalable
File Processing Platform
with NServiceBus
Sam Martindale

2. AI
Introduction
2
Sam Martindale
Architecting Innovation
14+ Years Development Experience
Co-Founder & Managing Partner, AI
NServiceBus adopter since v1.x

3. AI
3
The Problem

4. AI
The Problem
4
How do we move this data into our systems?
External Data Files Internal Systems

5. AI
The Problem
5
Automated pickup of many file types
Fixed Width

6. AI
The Problem
6
Record Level Logic and Workflows
New Account
Account Payment
Loan Application
Account Payment
Overdue Credit Account

7. AI
The Problem
7
Many Integration Points

8. AI
The Problem
8
Guaranteed Delivery

9. AI
The Problem
9
Scalability

10. AI
10
Why NServiceBus?

11. AI
Evaluated Technologies !
Sql Server Integration Services (SSIS)
!
Workflow Integration Difficulty
Systems Integration Difficulty
Extensibility
Not just ETL
!
!
!
!
11

12. AI
Evaluated Technologies
12
BizTalk/Sterling Commerce/Axway/Others
!
Licensing Costs
Scalability Costs
Difficult to “extend” functionality easily
Requires development “specialists”
!
!
!
!
!
!

13. AI
Evaluated Technologies
13
NServiceBus
!
Easy & inexpensive to scale
!
Easy to integrate with existing systems
!
Provides Message Durability
!
Great API for experienced .NET developers

14. AI
14
System Design

15. AI
The System
15
Alerts Endpoint
Processing Endpoint
Scheduler Endpoint
Parsing Endpoint
File Management
Endpoint
Api Endpoint
FileRetrieval Endpoint

16. Components
Scheduler
Manages scheduled “jobs” for picking up client files from FTP, SFTP,
FileShare, etc
Alerts Endpoint
Processing Endpoint
Scheduler Endpoint
Parsing Endpoint
File Management
Endpoint
Api Endpoint
FileRetrieval Endpoint
16
Why we needed it:
!
1. Needed the ability to schedule file pickups dynamically

17. Components
FileRetrieval
Performs the actual file pickup and decryption at the request of the
scheduler
Alerts Endpoint
Processing Endpoint
Scheduler Endpoint
Parsing Endpoint
File Management
Endpoint
Api Endpoint
FileRetrieval Endpoint
17
Why we needed it:
!
1. Needed the ability to pickup files
2. Needed the ability to decrypt files

18. Components
Alerts Endpoint
Processing Endpoint
Scheduler Endpoint
File Management
Endpoint
Api Endpoint
FileRetrieval Endpoint
18
FileParsing
Performs the parsing of the file according to the
configured client file format and layout. Divides the
file into logical “record sets” for processing
Why we needed it:
!
Had to be able to dynamically parse any given file
Parsing Endpoint

19. Components
Processor
Processes the record sets sent by the parser component by
performing calculations and persisting records to the database
Alerts Endpoint
Processing Endpoint
Scheduler Endpoint
Parsing Endpoint
File Management
Endpoint
Api Endpoint
FileRetrieval Endpoint
19
Why we needed it:
!
Needed the ability to persist data
Serves as an extension point for current/new workflows

20. Components
FileManagement
Tracks the completion of the file processing in a saga by listening to
various parsing events
Alerts Endpoint
Processing Endpoint
Scheduler Endpoint
Parsing Endpoint
File Management
Endpoint
Api Endpoint
FileRetrieval Endpoint
20
Why we needed it:
!
Had to keep track of the overall status for file processing
Needed the ability to monitor for failures or partial failures

21. Components
Alerts
Receives events from other components, sending notifications and
emails to the end users for system monitoring and alerting
Alerts Endpoint
Processing Endpoint
Scheduler Endpoint
Parsing Endpoint
File Management
Endpoint
Api Endpoint
FileRetrieval Endpoint
21
Why we needed it:
!
Needed the ability to notify users of various events

22. Components
Alerts Endpoint
Processing Endpoint
Scheduler Endpoint
Parsing Endpoint
File Management
Endpoint
Api Endpoint
FileRetrieval Endpoint
22
WebApi
WebApi component with SignalR. Delivers real time events and
updates to the web front end
Why we needed it:
!
Needed to notify users of created alerts in real-time
Needed to allow users to quickly onboard clients
!

23. AI
The System
23
Alerts Endpoint
Processing Endpoint
Scheduler Endpoint
Parsing Endpoint
File Management
Endpoint
Api Endpoint
FileRetrieval Endpoint

24. AI
24
What did we learn along the way?

25. AI
Division of Labor
25
Problem
!
Monolithic Endpoints
A naive implementation of our service placed far too many
handlers within the same endpoint, making it impossible to
prioritize messages and scale portions of the system
!
Api
Scheduler Alerts
Everything

26. AI
Division of Labor
26
Resolution
!
A Service is NOT an endpoint
In order to ensure our success, we had to split the system into
many components, each with its own distinct responsibilities
(and queues!)
!
Granular Scaling
By splitting into several disparate components, we were able to
meet our scalability needs more appropriately. (e.g. scaling
only the parsing endpoint)

27. AI
Saga Responsibilities
27
Problem
!
Scaling the File Processing Saga
Our initial (read: naive) implementation was responsible for
far too much business logic, making it difficult to scale

28. AI
Saga Responsibilities
28
Resolution
!
Keep it Simple!
Sagas should be as clean and tight as possible
!
Scaling Issues
The more complicated the saga, the harder they are to scale
!
“Observer" Implementation
In our case, we found that a “passive” saga responding to events and simply
keeping track of state was far easier to maintain and scale

29. AI
Saga Methodology
29
Problem
!
Saga Contention
Our initial (read: naive) implementation responded to too many
events, creating a large degree of contention and a high number
of rollbacks
Processing
Endpoint
File Management
Endpoint
3
3
90
32
13
2

30. AI
30
Resolution
!
“Check” Pattern
In order to reduce contention, we implemented a “check”
pattern using timeouts within the saga
Processing
Endpoint
File Management
Endpoint
3
90
32
13
2
Saga Methodology

31. AI
Deployment Automation
31
Problem
!
Deployment Challenges Hindering Proper Design
The team (and client) were initially skeptical of breaking the
system into many disparate deployable components, due to the
increased difficulty of deployment
!
!
Svc 1
Svc 2
Svc 4
Svc 3
Svc 4
Svc 5
Svc 1
Svc 4
Svc 5
Svc 1
Svc 2
Svc 4
Svc 1
Svc 2
Svc 3
Svc 1
Svc 3
Svc 5

32. AI
Deployment Automation
Resolution
!
Simple Automation
By leveraging the NServiceBus.Powershell bits along with our own
extended Powershell scripts, we were able to automate the
deployment of all of our NServiceBus projects
32
!
• Configuration Transformations
• Distributed Deployments
• Backup Strategy

33. AI
Firewalls, Servers & Failure, Oh My!
33
Problem
!
Deploying Into New Environments is Hard!
Misconfiguration, network issues, security issues… All these make
moving into a new environment cumbersome and error prone!

34. AI
Firewalls, Servers & Failure, Oh My!
34
Resolution
!
ServicePulse, ServiceControl & Custom Checks
!
• MSDTC Firewall Checks
• RavenDB Firewall Checks
• FTP Accessibility Checks
• Databus File Share Accessibility Checks

35. AI
35
The Result?

36. AI
The Result
36
Business Impact
!
Faster Processing Times
> 10K records/sec parsed and processed, per worker machine
!
Happier Users
Full system visibility and no lost data
!
Happier Developers
Clean, maintainable system
!
Happier Executives
Faster onboarding times for new clients

37. Thank You