Integrate 2019 - Creating a Processing Pipeline with Azure Functions and AIS
1. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
Wagner Silveira
Microsoft MVP -Azure
Creating a Processing Pipeline with AzureFunction
and AIS
2. About me
• Principal Integration Architect
• Microsoft Azure MVP
• MCSE Cloud Platform
wsilveiranz
wagner.silveira@notetoself.tech
https://notetoself.tech
3. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
Agenda
• Case Study
• Initial Solution
• Updated Solution
• Key Components
• A year later – what
changed?
• Lessons Learned
5. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
The solution requirements
• Onboarding of external EDI messages
• Should allow for different transports
• Message need to be Validated, translated,
enriched, routed
• Control on retries and notification over failed
messages
• Auditing of initial message
• Visibility of where a single message was in the
process
8. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
Reality
Check
• Big Payloads
• EDIFACT schema not
available
• Operational costs
• End to end Monitoring
10. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
How was this
different from
the initial
solution?
• Azure Functions
• EDIFACT support via .NET
Package
• Claim check pattern
• Dedicated instances
• Azure Storage
• Payload storage (claim check
pattern)
• Application Insights
• End to End with custom
events
• Single technology for
monitoring and notification
13. The claim
check
pattern
• Publishing function separates content
from metadata then:
• Store content on a safe repository
• Publish metadata + storage pointer +
Reprocessing count to message engine
• Subscribing processes receive
metadata and retrieve content when
required
Content+
Metadata
Service Bus
TopicPublishing
Function
Storage
Account
Subscription
Function
Subscription
FunctionMetadata
Content
Metadata
Content
14. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
App Insights custom events
• End to end traceability using a Correlation ID
• Allow for exceptions to be captured the same
way
• Leverage Functions support for App Insights
• Implemented as a reusable component
App Insights
Receive &
Validate
Transform Store &
Enrich
Route
1001
Received
1001
Validated
1001
Tranformed 1001 Stored
1001
Enriched
1001
Routed
1002
Received
1002
Failed
Validation
16. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
Key
Management
• Leveraged MSI support where
available
• SQL Server
• Storage
• Key vault
• Kept keys in key vault
where MSI where not
supported directly
• Service Bus
17. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
Exception Management &
Retries
on Functions
• Catch blocks using a notify and
throw pattern
• Leverage Function SB binding
peek-lock pattern
• Adjusted Maximum Delivery Count
on SB
• Messages that failed where
delivered to DLQ
18. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
• Logic Apps polling subscriptions DLQ every 6
hours
• Each subscription DLQ could have it’s own logic
• Reprocessing count
• Resubmission logic
• Notification logic
• Email notification
• Error blob storage
Pipeline
Topic
Validate
DLQ
Persist
DLQ
Reprocessing < 3
Error
Storage
DLQ Management
21. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
What about new
features
• Azure Functions Premium
• Improved support for Key
Vault
• Improved support for MSI
• Virtual Network support +
Service Endpoints
27. INTEGRATE
2019JUNE 3 - 5,
2019
etc.venues, London
In Summary
• Weight your technology
options
• Think about limitations,
operational costs
• Understand how your solution
fit in the big picture
• Understand the components
roadmap
