Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Building a Highly Scalable 
File Processing Platform 
with NServiceBus 
Sam Martindale
AI 
Introduction 
2 
Sam Martindale 
Architecting Innovation 
14+ Years Development Experience 
Co-Founder & Managing Part...
AI 
3 
The Problem
AI 
The Problem 
4 
How do we move this data into our systems? 
External Data Files Internal Systems
AI 
The Problem 
5 
Automated pickup of many file types 
Fixed Width
AI 
The Problem 
6 
Record Level Logic and Workflows 
New Account 
Account Payment 
Loan Application 
Account Payment 
Ove...
AI 
The Problem 
7 
Many Integration Points
AI 
The Problem 
8 
Guaranteed Delivery
AI 
The Problem 
9 
Scalability
AI 
10 
Why NServiceBus?
AI 
Evaluated Technologies ! 
Sql Server Integration Services (SSIS) 
! 
Workflow Integration Difficulty 
Systems Integrat...
AI 
Evaluated Technologies 
12 
BizTalk/Sterling Commerce/Axway/Others 
! 
Licensing Costs 
Scalability Costs 
Difficult t...
AI 
Evaluated Technologies 
13 
NServiceBus 
! 
Easy & inexpensive to scale 
! 
Easy to integrate with existing systems 
!...
AI 
14 
System Design
AI 
The System 
15 
Alerts Endpoint 
Processing Endpoint 
Scheduler Endpoint 
Parsing Endpoint 
File Management 
Endpoint ...
Components 
Scheduler 
Manages scheduled “jobs” for picking up client files from FTP, SFTP, 
FileShare, etc 
Alerts Endpoi...
Components 
FileRetrieval 
Performs the actual file pickup and decryption at the request of the 
scheduler 
Alerts Endpoin...
Components 
Alerts Endpoint 
Processing Endpoint 
Scheduler Endpoint 
File Management 
Endpoint 
Api Endpoint 
FileRetriev...
Components 
Processor 
Processes the record sets sent by the parser component by 
performing calculations and persisting r...
Components 
FileManagement 
Tracks the completion of the file processing in a saga by listening to 
various parsing events...
Components 
Alerts 
Receives events from other components, sending notifications and 
emails to the end users for system m...
Components 
Alerts Endpoint 
Processing Endpoint 
Scheduler Endpoint 
Parsing Endpoint 
File Management 
Endpoint 
Api End...
AI 
The System 
23 
Alerts Endpoint 
Processing Endpoint 
Scheduler Endpoint 
Parsing Endpoint 
File Management 
Endpoint ...
AI 
24 
What did we learn along the way?
AI 
Division of Labor 
25 
Problem 
! 
Monolithic Endpoints 
A naive implementation of our service placed far too many 
ha...
AI 
Division of Labor 
26 
Resolution 
! 
A Service is NOT an endpoint 
In order to ensure our success, we had to split th...
AI 
Saga Responsibilities 
27 
Problem 
! 
Scaling the File Processing Saga 
Our initial (read: naive) implementation was ...
AI 
Saga Responsibilities 
28 
Resolution 
! 
Keep it Simple! 
Sagas should be as clean and tight as possible 
! 
Scaling ...
AI 
Saga Methodology 
29 
Problem 
! 
Saga Contention 
Our initial (read: naive) implementation responded to too many 
eve...
AI 
30 
Resolution 
! 
“Check” Pattern 
In order to reduce contention, we implemented a “check” 
pattern using timeouts wi...
AI 
Deployment Automation 
31 
Problem 
! 
Deployment Challenges Hindering Proper Design 
The team (and client) were initi...
AI 
Deployment Automation 
Resolution 
! 
Simple Automation 
By leveraging the NServiceBus.Powershell bits along with our ...
AI 
Firewalls, Servers & Failure, Oh My! 
33 
Problem 
! 
Deploying Into New Environments is Hard! 
Misconfiguration, netw...
AI 
Firewalls, Servers & Failure, Oh My! 
34 
Resolution 
! 
ServicePulse, ServiceControl & Custom Checks 
! 
• MSDTC Fire...
AI 
35 
The Result?
AI 
The Result 
36 
Business Impact 
! 
Faster Processing Times 
> 10K records/sec parsed and processed, per worker machin...
Thank You
Nächste SlideShare
Wird geladen in …5
×

Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale

1.444 Aufrufe

Veröffentlicht am

Building a Highly Scalable File Processing Platform with NServiceBus
How do we move this data into our systems?

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale

  1. 1. Building a Highly Scalable File Processing Platform with NServiceBus Sam Martindale
  2. 2. AI Introduction 2 Sam Martindale Architecting Innovation 14+ Years Development Experience Co-Founder & Managing Partner, AI NServiceBus adopter since v1.x
  3. 3. AI 3 The Problem
  4. 4. AI The Problem 4 How do we move this data into our systems? External Data Files Internal Systems
  5. 5. AI The Problem 5 Automated pickup of many file types Fixed Width
  6. 6. AI The Problem 6 Record Level Logic and Workflows New Account Account Payment Loan Application Account Payment Overdue Credit Account
  7. 7. AI The Problem 7 Many Integration Points
  8. 8. AI The Problem 8 Guaranteed Delivery
  9. 9. AI The Problem 9 Scalability
  10. 10. AI 10 Why NServiceBus?
  11. 11. AI Evaluated Technologies ! Sql Server Integration Services (SSIS) ! Workflow Integration Difficulty Systems Integration Difficulty Extensibility Not just ETL ! ! ! ! 11
  12. 12. AI Evaluated Technologies 12 BizTalk/Sterling Commerce/Axway/Others ! Licensing Costs Scalability Costs Difficult to “extend” functionality easily Requires development “specialists” ! ! ! ! ! !
  13. 13. AI Evaluated Technologies 13 NServiceBus ! Easy & inexpensive to scale ! Easy to integrate with existing systems ! Provides Message Durability ! Great API for experienced .NET developers
  14. 14. AI 14 System Design
  15. 15. AI The System 15 Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint
  16. 16. Components Scheduler Manages scheduled “jobs” for picking up client files from FTP, SFTP, FileShare, etc Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 16 Why we needed it: ! 1. Needed the ability to schedule file pickups dynamically
  17. 17. Components FileRetrieval Performs the actual file pickup and decryption at the request of the scheduler Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 17 Why we needed it: ! 1. Needed the ability to pickup files 2. Needed the ability to decrypt files
  18. 18. Components Alerts Endpoint Processing Endpoint Scheduler Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 18 FileParsing Performs the parsing of the file according to the configured client file format and layout. Divides the file into logical “record sets” for processing Why we needed it: ! Had to be able to dynamically parse any given file Parsing Endpoint
  19. 19. Components Processor Processes the record sets sent by the parser component by performing calculations and persisting records to the database Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 19 Why we needed it: ! Needed the ability to persist data Serves as an extension point for current/new workflows
  20. 20. Components FileManagement Tracks the completion of the file processing in a saga by listening to various parsing events Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 20 Why we needed it: ! Had to keep track of the overall status for file processing Needed the ability to monitor for failures or partial failures
  21. 21. Components Alerts Receives events from other components, sending notifications and emails to the end users for system monitoring and alerting Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 21 Why we needed it: ! Needed the ability to notify users of various events
  22. 22. Components Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint 22 WebApi WebApi component with SignalR. Delivers real time events and updates to the web front end Why we needed it: ! Needed to notify users of created alerts in real-time Needed to allow users to quickly onboard clients !
  23. 23. AI The System 23 Alerts Endpoint Processing Endpoint Scheduler Endpoint Parsing Endpoint File Management Endpoint Api Endpoint FileRetrieval Endpoint
  24. 24. AI 24 What did we learn along the way?
  25. 25. AI Division of Labor 25 Problem ! Monolithic Endpoints A naive implementation of our service placed far too many handlers within the same endpoint, making it impossible to prioritize messages and scale portions of the system ! Api Scheduler Alerts Everything
  26. 26. AI Division of Labor 26 Resolution ! A Service is NOT an endpoint In order to ensure our success, we had to split the system into many components, each with its own distinct responsibilities (and queues!) ! Granular Scaling By splitting into several disparate components, we were able to meet our scalability needs more appropriately. (e.g. scaling only the parsing endpoint)
  27. 27. AI Saga Responsibilities 27 Problem ! Scaling the File Processing Saga Our initial (read: naive) implementation was responsible for far too much business logic, making it difficult to scale
  28. 28. AI Saga Responsibilities 28 Resolution ! Keep it Simple! Sagas should be as clean and tight as possible ! Scaling Issues The more complicated the saga, the harder they are to scale ! “Observer" Implementation In our case, we found that a “passive” saga responding to events and simply keeping track of state was far easier to maintain and scale
  29. 29. AI Saga Methodology 29 Problem ! Saga Contention Our initial (read: naive) implementation responded to too many events, creating a large degree of contention and a high number of rollbacks Processing Endpoint File Management Endpoint 3 3 90 32 13 2
  30. 30. AI 30 Resolution ! “Check” Pattern In order to reduce contention, we implemented a “check” pattern using timeouts within the saga Processing Endpoint File Management Endpoint 3 90 32 13 2 Saga Methodology
  31. 31. AI Deployment Automation 31 Problem ! Deployment Challenges Hindering Proper Design The team (and client) were initially skeptical of breaking the system into many disparate deployable components, due to the increased difficulty of deployment ! ! Svc 1 Svc 2 Svc 4 Svc 3 Svc 4 Svc 5 Svc 1 Svc 4 Svc 5 Svc 1 Svc 2 Svc 4 Svc 1 Svc 2 Svc 3 Svc 1 Svc 3 Svc 5
  32. 32. AI Deployment Automation Resolution ! Simple Automation By leveraging the NServiceBus.Powershell bits along with our own extended Powershell scripts, we were able to automate the deployment of all of our NServiceBus projects 32 ! • Configuration Transformations • Distributed Deployments • Backup Strategy
  33. 33. AI Firewalls, Servers & Failure, Oh My! 33 Problem ! Deploying Into New Environments is Hard! Misconfiguration, network issues, security issues… All these make moving into a new environment cumbersome and error prone!
  34. 34. AI Firewalls, Servers & Failure, Oh My! 34 Resolution ! ServicePulse, ServiceControl & Custom Checks ! • MSDTC Firewall Checks • RavenDB Firewall Checks • FTP Accessibility Checks • Databus File Share Accessibility Checks
  35. 35. AI 35 The Result?
  36. 36. AI The Result 36 Business Impact ! Faster Processing Times > 10K records/sec parsed and processed, per worker machine ! Happier Users Full system visibility and no lost data ! Happier Developers Clean, maintainable system ! Happier Executives Faster onboarding times for new clients
  37. 37. Thank You

×