1.
Trusting the Unknown -
Ensuring Safe & Secure Extension Use in CI/CD Workflows

2.
Of extensions contain some vulnerability

3.
Of extensions owned by Microsoft

4.
With a ⭐ rating

5.
Of extensions updated in the last 3 months

6.
Of Node based Azure Pipelines tasks

7.
Of PowerShell Azure Pipeline Tasks…

8.
That’s unfair
But it’s also a bit true…

9.
Jesse Houwing
Chief Trainer & Tinkerer
xpirit.com/jesse
jessehouwing
jessehouwing
jessehouwing
hachyderm.io/jessehouwing
+31 6 41 81 333 8

10.
Extensions in the context of Azure Pipelines
Manifests
Marketplace pritties
Tasks
And potentially tabs and context menu actions…

11.
So what is a task?
Loads of dependencies
At least azure-pipelines-task-lib
Frozen at time of publish
Another Manifest
The actual custom code
More pretties

12.
Or… using PowerShell
A lot fewer dependencies
at least VstsTaskSdk
Another Manifest
The actual custom code
More pretties

13.
An extension can contain more than 1 tasks
Task A
Task A
v1

14.
Where do these tasks come from?
• They’re “built-in” tasks
• Automatically updated by Microsoft in Azure DevOps
• Manually updated through Microsoft Update packs in Azure DevOps Server
• Manually updated by administrator through sideloading (see below)
• They’re from an extension from the public marketplace
• Installed through marketplace.visualstudio.com
• Automatically updated in Azure DevOps and Azure DevOps Server
• They’re from an extension from the private marketplace
• Manually uploaded to Azure DevOps Server
• They’re sideloaded directly
• Manually uploaded to Azure DevOps or Azure DevOps Server

15.
2 problems with task updates
By default, the agent:
1. Pins the major version to the one you selected
2. Updates to the latest minor version available

16.
Where do these tasks come from?

17.
So how do we know we can trust these tasks?
You don’t.

18.
How do you know?
Well…
I downloaded 700GB of vsix files from the Azure DevOps marketplace
Extracted them and scanned them

19.
So how do we know we can trust them?
Trusted publisher?
Verification?
Rating?
Number of issues and pull-requests?
Metadata provided by the author?
Recently updated?
Looks nice?

20.
Is it really that bad?
Maybe not…

21.
All of these potential security issues
Do not mean there is a known exploit chain

22.
But it’s not pretty!

23.
It’s not just the updates you should worry
about

24.
So what’s the risk we’re running
• Some of these extensions turn off TLS security
• Some might overwrite arbitrary files
• Some might be used to send data to the outside world
• Some install things directly from npmjs.
• Others download 100’s of MB, extract it and execute without
verification
• Often on a host with admin permissions
There is a lot of potential

25.
Why is PowerShell so much better?
It might not be but…
• Many of the dependencies are automatically updated through
Windows Update
• PowerShell relies on the .NET framework for most functionality
• There are a lot fewer dependencies to worry about

26.
What is Microsoft doing?
• They’ve introduced a new Node 16 handler for Azure Pipelines
• They’re deprecating Node 6 and Node 10 for Azure Pipelines tasks.
• They’re updating all the built-in tasks for Azure DevOps.
• They’re updating all the built-in tasks for Azure DevOps Server 2022?
• They’ve added the ability to turn off Node 6 on the Pipelines Agent.
• They’re adding warnings to every workflow that runs uses Node 6
(and later 10).

27.
What am I doing / have I done
• I actively update all my extensions regularly
• I automatically update my dependencies using Snyk, Dependabot and
RenovateBot
• I scan my code for vulnerabilities using Snyk Code and GitHub
advanced Security
• I maintain the Azure DevOps Extension Tasks to implement CI / CD for
Azure DevOps extensions
• I recently contributed to RenovateBot to add support for the Azure
Pipeline Marketplace. Now doing the same for dependabot-core.

28.
Guidance for deprecating an extension
• Add a warning to all versions of your tasks
• Timebomb the task to fail after today+X
• Mark all tasks deprecated:true in the task.json
• Add [DEPRECATED] to your task friendly name in the task.json
• Add [DEPRECATED] to your extension name in the vss-extension.json
• Publish this last version
• Unpublish the extension
• Archive the GitHub repository

29.
Guidance for deprecating task version
• Add both the old and the new version of the task to your extension
• Update the old version’s friendlyName and add [DEPRECATED] in
task.json
• Add a warning / error to the implementation to warn you users

30.
What should Extension authors do?
• Employ secure password practices and use 2FA
• Deprecate extensions you’re no longer maintaining
• Update your existing extensions to Node 16 and latest dependencies
• Enable DependaBot to automatically keep your dependencies up-to-
date
• Enable GitHub Advanced Security to detect issues in *your* code
• Implement CI / CD to ship new versions with ease
• Add repo metadata to their extension manifests

31.
What should Azure DevOps admins do
• Disable insecure extensions
• Implement RenovateBot to automatically
suggest updates to installed Azure
Pipelines tasks
• Disable Node 6 support on the Azure
Pipelines Agent
• Do not install every extension in the
marketplace without thought
• Provide Hosted Agents or ephemeral Scale-
set agents

32.
What should Azure DevOps admins do
• Run Endpoint Security on your Azure Pipelines Agents
• For DevOps Server
• Upgrade to Azure DevOps Server 2022 and stay current
• Migrate to Azure DevOps Service
• Manually upgrade “built-in” tasks
• Be careful with internal marketplace

33.
Overwrite a built-in task
npm install -g tfx-cli
tfx build tasks upload --task-zip-path Task.guid-version.zip
--service-url https://yourtfs.com/tfs/DefaultCollection
. ./script/install-task.ps1 -CollectionUrl https://yourtfs.com/tfs/DefaultCollection
-TaskZip Task.guid-version.zip
Delete all versions of a task
npm install -g tfx-cli
tfx build tasks delete --task-id ad884ca2-732e-4b85-b2d3-ed71bcbd2788
--service-url https://yourtfs.com/tfs/DefaultCollection

34.
What should Azure Pipelines authors do?
• Consider whether you actually need an extension.
• Keep your pipelines current. Update your major task versions and
resolve any issues.
• Use Azure Pipelines YAML
• Pin the exact version of your tasks
• Use RenovateBot to automatically keep your pipelines up to date
• Use Hosted Agents / Ephemeral Scale-set agents whenever possible

35.
What should the community do?
• Sponsor the authors of extensions you rely on.
• Submit pull requests to improve extensions.
• Submit pull requests to upgrade extensions to Node 16.
• Leave reviews on the marketplace and update them.

36.
Useful resources

37.
Useful Resources
• Azure DevOps Marketplace Scan
https://github.com/jessehouwing/azure-devops-marketplace-scan
https://jessehouwing.net/security-state-of-the-azure-devops-
marketplace/
• Hardening guidelines for Azure Pipelines
https://learn.microsoft.com/en-
us/azure/devops/organizations/security/security-best-
practices?view=azure-devops#secure-azure-pipelines

38.
Useful resources
• Azure Pipelines Tasks Zips
https://github.com/jessehouwing/azure-pipelines-tasks-zips
• RenovateBot for Azure Pipelines
https://jessehouwing.net/azure-pipelines-enable-renovatebot/
• Azure DevOps Extension tasks
https://marketplace.visualstudio.com/items?itemName=ms-
devlabs.vsts-developer-tools-build-tasks