Security Best Practices in Azure DevOps Pipeline

1. Attendee Introductions
● Name
● Company & role
● What do you want to get out of this session?
● Topics you want to discuss at upcoming events.
1
Please introduce yourself in the chat 

2. [18th of February 2021]
[Wellington] MuleSoft Meetup Group
Security Best Practices for Azure DevOps Pipeline

3. Reminders. . .
● Session will be recorded. Please use the online chat to ask questions.
● Trivia game after the presentation. Winners will get class voucher or certification exam
voucher of their choice!
● Fill out the post-event survey to share your feedback with us, and for a chance to win $50
Amazon.com gift card.
● Group picture at the end of the session 
3

4. 4
Organizers

5. 5
Speaker
Shaun Allen
Integration Developer
Trustpower

6. Security Best Practices for Azure
DevOps Pipeline
Shaun Allen, Integration Developer - Trustpower

7. Introduction
• Based in Tauranga, New Zealand
• Over 3 years of IT experience
• MuleSoft Certified Developer – Mule 4
• Areas of Expertise:
- Pipelines (CI/CD)
- Automation
- Scripting
- Full Stack
Fun fact: Muay Thai enthusiast
7

8. Agenda
• Technical Challenges / Business Objectives
• DevSecOps
• Continuous Integration (CI)
• Continuous Delivery (CD)
• Pipeline Security
• Demo
• Pipeline improvements
• Key takeaways
8

9. Business Objectives
Technical Challenges in Securing the Pipeline
• Externalise property or YAML files – on a separate repository.
• Developers create a pull request for every change.
• Dynamic mule key injection per environment.
• Gives flexibility to update mule key per application.
• Secret Management using Azure Key Vault.
• Secrets to be updated when the mule key changes
• Potential for different mule key per application
• DevOps permissions for Service Connections.
9

10. DevSecOps
• A management approach that combines application development, security, operations and
infrastructure as a code (IaaS) in an automated, continuous delivery cycle.
• Applying security at every stage of the software development process enables continuous
integration, reducing the cost of compliance and delivering software more rapidly.
Benefits of DevSecOps
• Improved quality and security of software.
• Faster software delivery.
• Enhanced communication and collaboration between teams.
• Earlier identification and correction of vulnerabilities in code.
• More opportunities for automated builds and quality assurance testing.
Resource: https://searchitoperations.techtarget.com/definition/DevSecOps
10

11. Setup a new application
This pipeline will handle the scaffolding of setting up a new project.
• Create a new folder under applications
• Create key vaults (generic Trustpower ARM template)
• Setup access policy for key vaults (ARM template parameters file)
• Tokenise key vault names into files
• Create application build and deploy pipeline
• Setup permissions for the new pipeline
How is this pipeline triggered?
11
Pipeline on-boarding

12. Service Hooks with Logic Apps
We use Azure DevOps service hooks for triggering Azure logic apps to do some work.
• We have one centralized logic app that all service hooks are sent to
• We then use a simple switch case statement for extracting the type of event i.e git push, PR
completed etc
• Using the above event types, send request to next logic app
• Inspect project level info i.e does the project name begin with mulesoft-
• Exclude any projects such as the mulesoft-pipelines project
• Trigger pipeline to setup a new application
• Will only add the file if the modified branch starts with feature
12
Automation on-boarding

13. Continuous Integration (CI)
Build application using centralised property file
based on environment variables
• File must be placed in the same path as existing properties file (file="properties/${mule.env}-
properties.yaml")
• Find errors before a deploy (save on time and roll back)(Only works with Munit)
• Using a powershell task for build, maven task was outputting the mule key to the console
13

14. POM Configuration Validation
Why do we validate config?
• Make sure all apps are following our standards
What we validate
• Group Id (must be equal to com.trustpower)
• Artifact Id (must match repo name)
• Packaging (must be equal to mule-application)
• Description (simple not null check)
• Does the app have Munit plugin?
• Required application coverage (must be equal to 75%)
• Fail build (must be equal to true)
• Run coverage (must be equal to true)
• Munit format (must be equal to console)
14
Continuous Integration (CI)

15. MUnit
Used for validating flow output.
Pros:
• Can find errors with an application before a deploy i.e if a property doesn’t exist the app will still build
successfully but only fail when trying to run.
• Make sure updates to the application still produce the required result.
• Integrate testing into the CI/CD pipeline.
Cons:
• Takes extra time for the pipeline to run.
• Extra overhead to setup (although easier with recorder).
15
Continuous Integration (CI)

16. Deploy application based on branch name
Pipeline triggers allow for a full CI/CD pipeline, we just need a way of knowing what environment
we want to deploy into
• Trigger build for all target branches
• Note: Not all branches listed in the triggers section can deploy without specific permissions i.e
group membership or using the auto deploy variable.
16
Continuous Deployment (CD)

17. Deploy application based on branch name
Needed a way to deploy to environments in a CI/CD fashion
Pipeline conditions
• feature/* deploys to development
• develop deploys to qa
• release* deploys to preprod
• master deploys to production
17
Continuous Delivery (CD)

18. Deploy application using Anypoint CLI
How do we deploy to cloudhub?
• Using the Anypoint CLI
Commands used by anypoint CLI:
• runtime-mgr cloudhub-application list
• runtime-mgr cloudhub-application start
• runtime-mgr cloudhub-application deploy/modify
• api-mgr api edit
List of commands: https://docs.mulesoft.com/runtime-manager/anypoint-platform-cli
18
Continuous Delivery (CD)

19. Auto deploy and group membership
Pipeline has two ways of deploying into environments.
1. Application settings file has an auto deploy flag
• For dev and qa
2. User who triggered the pipeline is a member of the mulesoft-pipelines “Release Administrators”
group
• Restrict who can deploy to cloudhub (must be manually triggered)
• If triggered from a PR into master, the user comes through as a Microsoft service
account
• Users in this group should be aware that anytime they trigger a pipeline run it will deploy
19
Continuous Delivery (CD) / Pipeline Security

20. Pipeline Security
Property Files Management
We wanted a way to externalise properties for cloudhub environments from the project repository.
• Easier for code reviewers to keep an eye on production configurations (no secrets in plain text etc)
• Centralised management (all application property files stored in one place)
We wanted a way to specify environment defaults for cloudhub deploys
• Use an application settings file (JSON format) to control deploy properties
20

21. Property Files Management - One Policy to rule them
all
One single repository means we can have one policy setup on the master branch to control all pull
requests (code reviews)
• Use one Azure DevOps group who can approve the PR’s.
• Use One file exclude pattern so developers can make quick changes to dev and qa for application
specific properties.
• Still need to create a new branch for updates, but the PR can be completed with no approvals
needed
21
Pipeline Security

22. Secure Credentials and settings.xml files
Credentials file is what we use to authenticate to the anypoint platform before using the anypoint
CLI commands.
• We use one single profile (default)
• This is so we don’t need to continuously set the env for each powershell task
We use a connected application for authentication (we were using a native account)
• Easy to restrict or revoke access
• Control access to data using scopes
22
Pipeline Security

23. Secure Credentials and settings.xml files
Our settings.xml file is how we authorize our pipeline to download dependencies for our
applications.
• Azure DevOps artifact feed (common dependency for all apps)
• Trustpower exchange assets
• Anypoint exchange v2
• Enterprise artifacts (nexus)
Settings file cannot use connected app directly, we must generate an access token using the
connected app details. Dependencies using the access token are:
• Trustpower exchange assets
• Anypoint exchange v2
Azure DevOps is using a PAT token.
Enterprise artifacts (nexus) is using a native login (username and password). We don’t have
access to change this setup.
23
Pipeline Security

24. Secret Management with Azure Key Vault
Automation will setup the scaffolding of a new project.
• Creating the key vault
• Setup the access policy for the service principal (service connection identity) on the key vault
• Key vault name is automatically added to the application settings file with tokenisation
Secrets must be manually setup
• Updates are also manual i.e. if a password or security token changes we would need to manually
update the key vault secret
Applications use secure properties, so if any changes are made to key vault secrets this means
we would have to do a re-deploy to see the update.
• Could use Azure service principals and have the applications pull secrets directly from key vault
• Each app would need it’s own identity for each environment (or at least sandbox & production)
• Key vault cost associated with each secret transaction
24
Pipeline Security

25. Secret Management with Azure Key Vault
Using a pre-defined Azure DevOps pipeline kay vault task (where possible) to pull through
secrets.
• Easy to use
• Specify the service connection to use and what secrets we want to be pulled through
• Secrets can then be accessed like pipeline variables $(MuleKey)
• Also use the Az.KeyVault powershell module to access secrets if we need to do logical statements
• i.e if we need to overwrite the global mule key for our environment with an application specific
one
25
Pipeline Security

26. DevOps permissions for Service Connections
Service connection identity in Azure must be granted permissions on the key vault we are
wanting to access.
The newly created pipeline needs permissions to be able to use the service connections
• The ‘setup new application’ pipeline will allow the new app pipeline to access the service connections
as a user
• For the above to work, the service connections must exist before the pipeline runs
• No CI/CD otherwise, pipeline will hang waiting on permissions
• Just created a new api we can integrate with this pipeline to ensure service connections exist
(Work In Progress)
26
Pipeline Security

27. Secure config encryption tool
When the pipeline pulls secrets from our application key vault, we need a way to encrypt them.
This is where the mulesoft secure config encryption tool comes in
• https://docs.mulesoft.com/mule-runtime/4.3/secure-configuration-properties#secure_props_tool
• We need to supply the following arguments
• File or string
• Algorithm
• Mode
• Mule key
• Secret to be encrypted
Once we have encrypted pipeline secrets setup, we can tokenise the {env}-properties.yaml file
and add this to the src/main/resources/properties folder
27
Pipeline Security

28. Secure Configuration properties
Extract mule key from environment or application key vault
• Mule key length has to abide by the algorithm settings i.e AES algorithm has an exact key length of 16
• This is where you may need different mule keys for applications if migrating to a different algorithm
• Environment mule key vs application mule key
Extract algorithm and mode from the application global-configs.xml file
• Or use the default set in the application.settings file
• If using default of values, there is no properties being setup in the global-configs.xml, hence
the need for defaults in app settings file
28
Pipeline Security

29. SLA Tiers
We use an automated policy to enforce the need for SLA tiers across all environments.
As part of our application pipeline, when the deploy stage happens, we deploy Trustpower
default SLA tiers that are setup in the default.settings file.
We can modify or add to these SLA tiers using the application.settings file for each application in
each environment.
When a user is requesting access to an application this is where they will be able to select their
required SLA tier.
• Auto approved for dev and qa
• Manual approval needed for production
29
Pipeline Security

30. 30

31. Pipeline improvements
• Use a step template for generating the mule key
• Remove inline scripting
• Some PowerShell tasks need to be smaller and more understandable/readable
• Look through devops improvements (can we use stage dependencies for stage conditions
yet?)
• Integration Postman agent (newman)
• Automate setting up API Manager
• Only trigger one build at any time
• Use of approval gates (Environment)
31

32. Key takeaways
What have we learnt (hopefully):
• How to externally manage property files
• Use policy to exclude files/folders from PR review for faster development
• How to use key vault for secrets and use within pipelines (service connection for access)
• How to use secure config tool with above for setting up encrypted secrets
• How to tokenise a property file to inject secrets from above
• Build an application using the external properties file
• Munit is good for integrating testing into the CI/CD pipeline, but is also handy for exposing errors
before a deploy if the properties file doesn’t match what the application is expecting
• Deploy application using pipeline conditions and flags
• Deploy SLA tiers
32

33. Call for Speakers!
● Incentive for speakers:
 Complimentary training class voucher or certification exam voucher of their choice.
● Please contact organizers for more information!
33

34. Thank you!