This document discusses monitoring the performance of Azure data pipelines. It recommends using Azure Log Analytics to monitor pipelines near real-time, as traditional testing tools don't support log analytics. It provides steps to configure a Log Analytics workspace and enable diagnostics logging. Sample queries are shown to monitor metrics like pipeline durations, activity times and throughput. Monitoring pipelines helps assess performance, stability and confidence in data processing.

1. PERFORMANCE
IS NOT A MYTH
P E R F O R M A N C E A D V I S O R Y C O U N C I L
SANTORINI GREECE
FEBRUARY 26 - 27 2020
Azure Multiple Pipeline
Performance Monitor
Gopalkrishnan Yadav

2. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Service
Current Team Size: 550+
Customer Base: 50+ active clients
Global presence in 15+ countries
Multiple Engagement and delivery models
Capability across technology domains :Legacy, ERP,
Web, Cloud, Big Data, Mobile
Alliance and Expertise
Portfolio of Testing Services across business domains
Ranked #1 by Ovum, ‘leading position’ by
Nelson Hall, IDC
Testing Leader 2015 by Gartner
• Performance Testing and Engineering
Methodology
• Performance 360 Framework
• PerfNEXT- Perftrack, Perf Analytics, Log
Analyzer, LRAC and SPARK Utilities
Innovation
Leadership
• End to End Performance Testing
(Baseline/Load/Stress/Scalability/Endurance)
• Performance Engineering
• Setting up of Performance testing practice
• Performance Testing CoE setup
• Managed Performance Test Center (MPTC)
• PTaaS
• Performance Consulting
• QBP & Maturity Assessment
• Transformation Consulting
• WAN, Mobile and Cloud Performance Testing
• Performance Lab setup
• HPE LoadRunner provisioning
Service Offerings
Manufacturing RetailTelecomInsurance
Media &
EntertainmentAviationBanking Energy & Util.

3. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
NFT Hub – IP Accelerator
Key Features
Automatic executions with CI/CD facilities for regression projects
Process driven approach standardized across the
enterprise using ‘Perf Track’ as orchestration engine
Interwoven set of tools and accelerators
providing complete life cycle support for
performance projects execution
‘WLM’ tool for realistic work load modeling and test scenario
generation using ‘Log Analyzer’
Reduced scripting time and increased quality using ‘LRAC’
for multiple scripts. Automated script validation.
‘Perf Analytics’ providing deep insights into the past test runs and at the same time
providing predictive views for future runs through machine learning
“Script less Automation”
“Performance Test Management”
“Realistic Load Testing”
“CI/CD Ready”
“Analytics”
“End to End Platform” “Electronic Documents”
“Validation”
Automation Benefits/Savings
Script Design Effort
Management Effort
Report Prep Effort
Enables
“Realistic” Load
Testing
Helps in
Requirement Gathering

4. 4© 2019 Capgemini – Internal use only. All rights reserved.Cloud with AWS | May 2019
OneShare
OneShare is Sogeti’s Cloud Platform solution consisting of Self-Service portal, Templates and Services for cloud
environment provisioning and management.
 Helps speed up the provisioning of Dev and Test environments with Self/Managed service and resource templates
 Easy provision of environments in Azure, AWS, Google Cloud Platform & IBM Cloud and self manage by Dev and
Test teams
 Control Cloud usage costs through usage monitoring and resource scheduling
 Standardize on Environments throughout the enterprise in a unified, robust and repeatable way
 Multi Cloud provisioning, Speed up provisioning of Dev/Test environments, Standardized
Environments; Start, Stop and Schedule VMs & Environments
 Gain insights on environments and subscription costs, Control cloud usage costs, ‘Pay as you
use’ pricing model, , VM Utilization Report
 Microfocus Test tooling and Managed Services
 Dev/Test Template Management Services, Infra Management, Role Based access and Quota
based resource creation
 Azure DevOps Integration for CI/CD configuration and execution
 Solution leads: Santanu De
Overview
Benefits
References
More information
 Fiskars, Posti Group, Boots, Bob Evans, Primark, SignPost, Neste, Outokumpu. Velmet,
 Smiths

5. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Overview
• The Azure performance monitoring cannot be done using traditional approach
of record replay model
• The standard performance testing tools like JMeter, LoadRunner does not
support the log analytics
• The Best recommended model to assess the performance of Pipelines is using
Azure Log analytics feature
• The near real time data is monitored and in advance configuration alert
mechanism can be implemented

6. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
ADF Architecture
ADFv2-Ingest
Date Lake Store
Create QS tables
(Landed/Processed)
Flat Files
Data bases
UDL
BDL
Azure Analysis ServicesPDS-SQLDW
1
2
3
4
5 6
ADF Performance
monitoring
Curate UDL data to BDL
Data bricks
Automation testing
Power BI
ADF
code
commit
to VSTS
Data Bricks– VSTS Git Integration
ADF – Git Integration

7. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Pipeline Execution

8. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Performance Monitoring Strategy

9. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Performance Monitoring Approach

10. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Performance Metrics
01
02
03
05
06
07
Successful Pipeline count
Failed Pipeline count
Output data written Vs output data Read
Activity Duration
Successful Activity count
Failed Activity count
04 Integration runtime CPU utilization Integration Runtime available memory08

11. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Workspace Configuration
In the Azure portal, click All services. In the list of resources,
type Log Analytics.. Select Log Analytics workspaces.
•Click Add, and then select choices for the following items:
Provide a name for the new Log Analytics workspace, such as DefaultLAWorkspace.
•Select a Subscription to link to by selecting from the drop-down list if the default selected is not
appropriate.
•For Resource Group, choose to use an existing resource group already setup or create a new one.
•Select an available Location.
•After providing the required information on the Log Analytics Workspace pane, click OK

12. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Diagnostic Log Enablement Settings
• In the portal, navigate to Azure Monitor and click on Diagnostic
Settings
• Optionally filter the list by resource group or resource type,
then click on the resource for which you would like to set a
diagnostic setting.
• If no settings exist on the resource you have selected, you are
prompted to create a setting. Click "Turn on diagnostics."
• If there are existing settings on the resource, you
will see a list of settings already configured on this
resource. Click "Add diagnostic setting."

13. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
• Give your setting a name and check the box for Send to Log
Analytics, then select a Log Analytics workspace
• Click Save
Diagnostic Log Enablement Settings

14. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Configure Pipeline Monitoring Settings
Microsoft Azure use Kusto Language to develop the query. In the monitoring section there
are many default KPI is available which is very easy to configure.
Following steps needs to be followed to configure the dashboard
• Login to the Microsoft Azure Portal
• Click on Monitor tab
• Click on Explore Metrics
• Select the Resource group
• Select the metrics and choose the right aggregation
• Click on Pin to dashboard

15. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Azure Analytics
• SAMPLE QUERY METRICS
• Output data written Vs Output data read Vs Pipeline Name
• Activity start time Vs Activity End Time Vs Output copy duration
• Activity total duration Vs Succeeded
• Output data written Vs Output data read Vs Pipeline Name:
• AzureDiagnostics
• | where TimeGenerated > ago(7d)
• | project Output_dataWritten_d, activityName_s , Output_dataRead_d , pipelineName_s
• Activity start time Vs Activity End Time Vs Output copy duration
• AzureDiagnostics
• | where ResourceGroup contains "80011"
• | project start_t , end_t, pipelineName_s , activityName_s , Output_copyDuration_d
• | render timechart

16. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
• Activity total duration Vs Succeeded
AzureDiagnostics
| project activityName_s , start_t , end_t , status_s
| extend duration = datetime_diff('second',end_t, start_t)
| extend duration = duration/60
| where status_s != "Succeeded" and activityName_s != "" and durationm > 5
Azure Analytics

17. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Sample Report

18. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
• Succeeded Pipeline VS Duration
• Succeeded Activity VS Duration

19. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Query to Generate Graph

20. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Sample Report

21. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Benefits
• Help customer to assess the performance of pipeline jobs
• Give confidence on the data processing technique
• Stability on the jobs can be achieved
• The Microsoft Azure analytics has many feature to be explored for designing the
feature. The package comes free with full license entitlement.

22. PERFORMANCE
IS NOT A MYTH
P E R F O R M A N C E A D V I S O R Y C O U N C I L
SANTORINI GREECE
FEBRUARY 26 - 27 2020
Thanks