Toon Vanhoutte s

1. BTUG – BizTalk Tracking
Toon Vanhoutte – Integration Architect

2. Agenda
Configuration
Best Practices
Tools
Under the hood
Demo
Visualization
2

3. Demo
Setup
Specify unique reference for messages #search
Specify simulated response times of backend systems #sla
Specify customer and transporter names #inspiration
3

4. Configuration
4

5. Configuration
|
Port Tracking Configuration
Only track what’s really required #performance
5

6. Configuration
|
Pipeline Tracking Configuration
Bug BTS2010 CU2  CU5 #solved #CU6 #kb2728652
6

7. Configuration
|
Orchestration Tracking Configuration
Avoid Shape start and end
#performance #orchestrationdebug
7

8. Configuration
|
Property Schema Tracking Configuration
Make context properties searchable #troubleshooting
Global + specific property schema #generic #consistent
8

9. Configuration
|
Disable Global BizTalk Tracking
9

10. Under the hood
10

11. Under the hood
|
BizTalk Database Structure
Host Instance processes message
SQL Agent
11

12. Under the hood
|
BizTalk Database Structure
Host Instance processes message
SQL Agent
Message stored in MessageBox
12

13. Under the hood
|
BizTalk Database Structure
Host Instance processes message
SQL Agent
Message stored in MessageBox
TDDS copies events to DTA + BAM
13

14. Under the hood
|
BizTalk Database Structure
Host Instance processes message
SQL Agent
Message stored in MessageBox
TDDS copies events to DTA + BAM
SQL Agent copies bodies to DTA
14

15. Under the hood
|
Store messages in MessageBox database
|
Physical runtime message bodies are stored in:
|
|
|
|
Spool
MessageParts
Parts
Fragments
Fragmentation controlled by large message size
15

16. Under the hood
|
Process messages in MessageBox database
|
Per Host, these tables are created:
|
|
|
|
HostNameQ: Main queue: each host instance polls this queue table
to collect messages that are pending processing
HostNameQ_Suspended: Suspended queue: queue table that
holds supended messages
HostNameQ_Scheduled: Scheduled queue: not used
InstanceStateMessageReferences_Host: State queue: holds list of
messages that have been processed but are required later
Stores only reference to physical message in Spool table
16

17. Under the hood
|
Store messages in MessageBox database
#probts2006
17

18. Under the hood
|
Store tracking data in MessageBox database
|
#probts2006
Host Instance serializes data and stores it in the MessageBox
|
Message events:
|
|
|
TrackingData_0_x [x: 0 - 3]
TrackingData_1_x [x: 0 - 3]
Message bodies:
|
|
|
|
Spool
MessageParts
Parts
Fragments
18

19. Under the hood
|
Copy Events from MessageBox to DTA database
|
Tracked Message Events copied by TDDS Service:
|
|
Tracking Data Decode Service
Runs in each host instance with tracking enabled
Create a dedicated tracking host
19

20. Under the hood
|
Copy Events from MessageBox to DTA database
BizTalk MessageBox
TrackingData_0_0
TrackingData_0_1
TrackingData_0_2
TrackingData_0_3
TrackingData_1_0
TrackingData_1_1
TrackingData_1_2
TrackingData_1_3
TDDS
BAM Primary Import
Activity_Active
Activity_ActiveRelationships
Activity_Completed
Activity_CompletedRelantionships
Activity_Continuations
BizTalk DTA Database
Dta_MessageInOutEvents
Dta_ServiceInstances
...
20

21. Under the hood
|
Copy Events from MessageBox to DTA database
BizTalk MessageBox
TrackingData_x_x
TDDS
BAM Primary Import
TDDS_StreamStatus
TDDS_FailedTrackingData
BizTalk DTA Database
TDDS_StreamStatus
TDDS_FailedTrackingData
TDDS_StreamStatus contains last copied sequence n°
TDDS moves events with > sequence n° from TrackingData_x_x to DTA
TDDS_FailedTrackingData contains tracking failures
21

22. Under the hood
|
Copy Bodies from MessageBox to DTA database
|
Tracked Message Bodies copied by SQL Agent Job:
|
|
TrackedMessages_Copy_BizTalkMsgBoxDb
Scheduled by default each minute (development: 10 seconds)
Monitor the SQL Agent Jobs #SCOM #blog
22

23. Under the hood
|
Copy Bodies from MessageBox to DTA database
BizTalk MessageBox
Spool
MessageParts
Parts
Fragments
SQL
Agent
BizTalk DTA Database
Tracking_Spoolx
Tracking_Partsx
Tracking_Fragmentsx
[x: 1 - 2]:
Always one online / one offline
23

24. Under the hood
|
Remove data from MessageBox
|
This is performed by SQL Agent Jobs:
|
MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb:
|
|
|
Runs continuously (due to former SQL limitation of 1 min job schedule)
This jobs checks weather a message is still referenced by instances
MessageBox_Message_Cleanup_BizTalkMsgBoxDb
|
|
|
Is disabled (keep it like this)
It’s called from with the infinite loop of the ManageRefCount Job
This job deletes physical messages from MessageBox that have no outstanding
references anymore
24

25. Under the hood
|
Remove data from DTA
|
|
Enable + Configure the DTA Purge and Archive Job #avoidlargeDTA
Remove data from BAM
|
SQL Agent Job BAM_DM_Activity moves data to BAMArchive
25

26. Under the hood
|
BizTalkDTADb Database model
|
No foreign keys #performance
|
|
|
Designed for fast insert
Results in slower query
No single chain ID
|
Difficult to reproduce the “route” within BizTalk
26

27. Under the hood
|
BizTalkDTADb Database model
|
No foreign keys #performance
|
|
|
Designed for fast insert
Results in slower query
No single chain ID
|
Difficult to reproduce the route within BizTalk
27

28. Under the hood
|
Create a chain of messages

29. Under the hood
dta_MessageInOutEvents
 Service Instance ID of messages
|
Create a chain
 Message Instance ID
SI = 1
MI = A
SI = 1
MI = B
SI = 2
MI = B
SI = 2
MI = C
SI = 3
MI = C
SI = 3
MI = D
29

30. Tools & SDK
30

31. Tools & SDK
|
BizTalk Administration Console #technical #notsexy
31

32. Tools & SDK
|
BizTalk MessageBox Viewer #troubleshooting
32

33. Tools & SDK
|
BizTalk Terminator #troubleshooting
Import MBV scripts
33

34. Tools & SDK
|
Performance Counters #troubleshooting
|
|
Throttling state  find reason on MSDN
TDDS
34

35. Tools & SDK
|
Retrieve tracked message body #3ways
|
Microsoft.BizTalk.Operations.dll
35

36. Visualization
36

37. Visualization
|
Bad practice
|
Develop a dashboard directly on BizTalkDTADb #performance
|
|
|
No control on user queries
Impacts directly the BizTalk runtime
Bad dashboard performance
37

38. Visualization
|
Bad practice
|
Develop custom tracking mechanism
38

39. Visualization
|
Good practice
|
Use default BizTalk Tracking #CID
39

40. Visualization
|
Good practice
|
Use BizTalk Tracking #CID
40

41. Visualization
|
Business Activity Monitoring? #notideal
|
Development impact
|
|
|
SQL Server Analysis Services
|
|
|
|
|
Tracking Profile Editor is limited
BAM API is rather complex
2008 R2 required, when using SQL 2012
Deprecated BAM portal
Impact on operations
Not ideal OLAP cube structure
Termination of activities not straightforward
41

42. Visualization
|
Use case #1: Message Search
|
BizTalk Administration Console
|
Codit Integration Dashboard
42

43. Visualization
Demo
|
Use case #1: Message Search
|
Search on context properties:
|
|
|
|
|
Application
Flow
Sender
Receiver
Reference
43

44. Demo
|
Visualization
Use case #1: Message Search
44

45. Visualization
|
Use case #2: Operational reports
|
|
|
#managedservices
Message Load
Message Size
Database Size
45

46. Visualization
|
Use case #3: Pro-Active Monitoring
|
#managedservices
Custom SCOM alert: monitoring non-events #vbs
|
E.g.: Just-In-Time ordering
|
|
|
Daily check on expected amount of received orders
Query on BizTAlkDTADb
Automated email alert / support ticket
46

47. Visualization
|
Use case #4: SLA Reporting
|
Processing times of web services
|
|
|
Total processing time
Processing time of every backend system
Processing time in BizTalk
47

48. Demo
|
Visualization
Use case #4: SLA Reporting
GetCustomer
GetCustomerItemList
sp_CRM_GetCustomer.WCF
ORCH
rp_MasterDataService.WCF
GetItem
GetEmployee
sp_SAP_GetItem.WCF
sp_AX_GetEmployee.WCF
48

49. Visualization
Demo
Use case #4: SLA Reporting
BizTalk Processing Time = Total Processing Time – SAP Processing Time
Same MessageID
1
2
3
rp_MasterDataService.WCF
8
4
sp_SAP_GetItem.WCF
7
6
Same MessageID
5
SAP Processing Time
Total Processing Time
|
49

50. Visualization
Demo
|
Use case #4: SLA Reporting
|
SQL Agent Job
|
BizTalkDTADB  Custom database #chaining
50

51. Demo
|
Visualization
Use case #4: SLA Reporting
51

52. Demo
|
Visualization
Use case #4: SLA Reporting
52

53. Visualization
|
Use case #5: Process Overview
|
Visualization of related integration flows
53

54. Demo
|
Visualization
Use case #5: Process Overview
|
Example: EDI Order Processing
54

55. Demo
|
Visualization
Use case #5: Process Overview
55

56. Visualization
|
Use case #6: Dashboards
|
|
|
|
|
|
|
BizTalk 360: Operational focus
AIMS for BizTalk: Behavior / trend analysis
Codit Integration Dashboard: Advanced message handling
Minotaur
Frends Helium
Moesion
…
56

57. Visualization
Demo
|
Use case #6: Dashboards
|
Codit Integration Dashboard
|
|
|
Friendly Naming of technical artifacts
Message Search
Message Resume / Resubmit
|
|
Editing
Multi selection
57

58. Demo
|
Visualization
Use case #6: Dashboards
|
Codit Integration Dashboard: resubmit
58

59. Questions?
You
BTUG
Microsoft
59