1. ETL Stuck WorkFlow Monitor
Presentation Title
Presentation Date Page

2. ETL Stuck WorkFlow Monitor
● Premis:
Presentation Title
Presentation Date Page

3. ETL Stuck WorkFlow Monitor
● Premis:
Every ETL process has distinct PID,
description, & info
Presentation Title
Presentation Date Page

4. ETL Stuck WorkFlow Monitor
● Premis:
Every ETL process has distinct PID,
description, & info
Info includes process start time &
cumulative CPU time
Presentation Title
Presentation Date Page

5. ETL Stuck WorkFlow Monitor
● Premis:
Every ETL process has distinct PID,
description, & info
Info includes process start time &
cumulative CPU time
Snapshots can be taken at regular
intervals and saved
Presentation Title
Presentation Date Page

6. ETL Stuck WorkFlow Monitor
● Premis:
Every ETL process has distinct PID,
description, & info
Info includes process start time &
cumulative CPU time
Snapshots can be taken at regular
intervals and saved
Compare “Current” v. “Previous”
Presentation Title
Presentation Date Page

7. ETL Stuck WorkFlow Monitor
● Premis:
Every ETL process has distinct PID,
description, & info
Info includes process start time &
cumulative CPU time
Snapshots can be taken at regular
intervals and saved
Compare “Current” v. “Previous”
Use simple Unix comm function
Presentation Title
Presentation Date Page

8. ETL Stuck WorkFlow Monitor
● Context Diagram
Presentation Title
Presentation Date Page

9. ETL Stuck WorkFlow Monitor
● Diagram 0
Presentation Title
Presentation Date Page

10. ETL Stuck WorkFlow Monitor
● Flow Chart
Presentation Title
Presentation Date Page

11. ETL Stuck WorkFlow Monitor
Presentation Title
Presentation Date Page

12. ETL Stuck WorkFlow Monitor
● Pseudocode
Presentation Title
Presentation Date Page

13. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Presentation Title
Presentation Date Page

14. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Presentation Title
Presentation Date Page

15. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Presentation Title
Presentation Date Page

16. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
Presentation Title
Presentation Date Page

17. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
Presentation Title
Presentation Date Page

18. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Presentation Title
Presentation Date Page

19. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Presentation Title
Presentation Date Page

20. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Presentation Title
Presentation Date Page

21. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
Presentation Title
Presentation Date Page

22. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Presentation Title
Presentation Date Page

23. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
Presentation Title
Presentation Date Page

24. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
If previous… and current… are BOTH NULL
Presentation Title
Presentation Date Page

25. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
If previous… and current… are BOTH NULL
Then
Presentation Title
Presentation Date Page

26. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
If previous… and current… are BOTH NULL
Then
Send alert “ETL May be down” to WorkFlow support
Presentation Title
Presentation Date Page

27. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
If previous… and current… are BOTH NULL
Then
Send alert “ETL May be down” to WorkFlow support
Exit
Presentation Title
Presentation Date Page

28. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
If previous… and current… are BOTH NULL
Then
Send alert “ETL May be down” to WorkFlow support
Exit
Format alerts messages
Presentation Title
Presentation Date Page

29. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
If previous… and current… are BOTH NULL
Then
Send alert “ETL May be down” to WorkFlow support
Exit
Format alerts messages
Send pages and emails to WorkFlow support
Presentation Title
Presentation Date Page

30. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
If previous… and current… are BOTH NULL
Then
Send alert “ETL May be down” to WorkFlow support
Exit
Format alerts messages
Send pages and emails to WorkFlow support
Format log info
Presentation Title
Presentation Date Page

31. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
If previous… and current… are BOTH NULL
Then
Send alert “ETL May be down” to WorkFlow support
Exit
Format alerts messages
Send pages and emails to WorkFlow support
Format log info
Write info to log
Presentation Title
Presentation Date Page

32. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
If previous… and current… are BOTH NULL
Then
Send alert “ETL May be down” to WorkFlow support
Exit
Format alerts messages
Send pages and emails to WorkFlow support
Format log info
Write info to log
Copy current… into previous…
Presentation Title
Presentation Date Page

33. ETL Stuck WorkFlow Monitor
● Pseudocode
Fetch process info for current ETL processes:
Output PID, start time, CPU time, sub-string(process name, 13)
Sort the results
Place results into current_wf_processes.txt
If previous_wf_processes.txt exists
Then
Compare current… with previous… files
Place results in wf_process_compare.txt
If …compare file is NULL
Then
If previous… and current… are BOTH NULL
Then
Send alert “ETL May be down” to WorkFlow support
Exit
Format alerts messages
Send pages and emails to WorkFlow support
Format log info
Write info to log
Copy current… into previous…
Exit
Presentation Title
Presentation Date Page

34. ETL Stuck WorkFlow Monitor
● Example outputs
Previous -
14739 15:20:01 1:21 wf_Delete_Dim.s_m_Delete_dim
16944 15:20:13 0:23 wf_COM_PRE.s_m_COM_PRE
17296 15:20:23 3:14 wf_ATTR_STG.s_m_ATTR_STG
18070 15:15:26 3:19 wf_SLA_STG.s_m_SLA_STG
19105 15:11:19 2:20 wf_DR_ATTR_DIM.s_m_HDA_ATTR_STG1
23059 15:21:24 0:03 wf_COM_HA.s_m_COM_HA
29936 15:23:52 0:16 wf_AB_SL_FACT.s_m_AB_SL_FACT
3159 15:18:13 2:50 wf_H_FACT.s_m_H_FACT
Current -
10457 15:25:15 0:29 wf_COM_SUM.s_m_COM_SUM
11622 15:25:51 0:01 wf_WIN_SUM.s_m_WIN_SUM
14739 15:20:01 2:08 wf_Delete_Dim.s_m_Delete_dim
17296 15:20:23 3:49 wf_ATTR_STG.s_m_ATTR_STG
18070 15:15:26 3:19 wf_SLA_STG.s_m_SLA_STG
19105 15:11:19 2:50 wf_DR_ATTR_DIM.s_m_HDA_ATTR_STG1
23059 15:21:24 1:03 wf_COM_HA.s_m_COM_HA
29936 15:23:52 0:29 wf_AB_SL_FACT.s_m_AB_SL_FACT
3159 15:18:13 2:50 wf_H_FACT.s_m_H_FACT
7034 15:25:00 0:03 wf_FL_BR_STG.s_m_FL_BR_STG
comm -12 current_wf_processes.txt previous_wf_processes.txt
Result –
18070 15:15:26 3:19 wf_SLA_STG.s_m_SLA_STG
3159 15:18:13 2:50 wf_H_FACT.s_m_H_FACT
Presentation Title
Presentation Date Page

35. ETL Stuck WorkFlow Monitor
Next steps
1
 Coding ~ 5 hours
1
 Testing ~ 9 hours
2
 Documentation ~ 20 hours
1 - Rule of thumb, 4 x more that you would think it would take + one
2 - Rule of thumb, 2 x more that you would think it would take + 50%
Presentation Title
Presentation Date Page