2. 1. Who is HMS Eliza
2. Why Modernize
3. SDLC Toolset Selection
4. The Transition
1. SCM and Branching
2. Dependency Management Strategy
3. Scalable CI System
5. Challenges
6. Demo, Next Steps, and Roadmap your
own project
7. Questions
2
3. 3
Quick Facts
• Mission: Keep healthcare costs down
• Customers
• Payers
• Providers
• PBMs
• Main business lines
• Coordination of Benefits (CoB)
• Payment Integrity (PI)
• Pop Health
• Care Management
• Healthcare Communications
• Publicly traded (HMSY)
• $600M Revenue
• 2500 people
4. 4
Located in Danvers, MA
Acquired by HMS April 2017
200 People & $60M Revenue
Healthcare Communications at
Scale
Print & All Digital Channels
All Workloads in AWS
13. SCM: From
TFS
13
TFS Leave Behind Strategy
Leave history behind
Read-Only Mode
Lock parts of TFS projects as they are moved to Git
Process in place for backporting while transitioning to Git
One branch in TFS could become multiple Git repositories.
14. SCM: From
TFS
14
Migrate from VS 2015 to 2017
Migrate to the Nuget Package Reference Format
Including version ranges for middle of the stack
Lock files for top of the stack
Migrate to
Break up solutions into ‘N’ Git repositories
Change Namespaces
Create shared AssemblyInfo file for versioning multiple
CSPROJs.
Start at the bottom of the stack, and work your way up
18. Jenkins:
Architecture
18
Key Requirements
Scalable
Pipelines to define our CI/CD processes
Design
Automation first approach
Technologies we would like to use:
Jenkins Master - Stateless
Jenkins development environment can be
stood up in as little as 30 minutes.
25. 25
Jenkins
Architecture:
Takeaways
Uses the Amazon Elastic Container
Service Jenkins plugin
Jenkins Stateless
All builds push artifacts to Artifactory
Build numbers are not used
Jenkins master is configured entirely with
init.groovy.d scripts, no manual
configuration.
Development environment dynamically
changes build job configurations to not
impact production.
IE – do not build branches automatically
29. Demo
Developer use case:
Create a branch
Make a modification
Open a PR
Merge the PR
See the develop build push to Artifactory
See the branches automatically get built
Quality Gates today:
Dependency version enforcement
MSBuild Warnings
Unit Tests and Code Coverage
Branching Strategy
29
30. Next Steps
Metric collection and radiating
Creating a Capability Maturity Model
(CMM) and start to quantify and track
KPIs
SonarQube Static Analysis
Static Application Security Testing
(SAST)
Visual Studio 2019 and Windows Server
2019 containers
Integration Testing – Automated Test
Environment Provisioning
30
31. Roadmap
What is the technical debt in your code base? What anti-
patterns are active? Design for them.
Binaries in SCM
Versioning
Etc.
Company’s objectives?
KPI’s to start measuring?
Build speed
Etc.
Determine your toolset – SCM, CI and Artifact management
SCM infrastructure online
Determine how you will manage 100’s of repositories
Naming convention, permissions, CI integration, Branch
Protections
CI Infrastructure online
Manage Stateful-ness and Stateless-ness
Isolated per build containers – no leakage
Artifact management
Dependency management design and tight CI integration
Configuration management design and tight Artifact management
integration – properties, searching etc.
Organizational alignment
Buy in to new SDLC processes
KPI measurement for success
Transition team in place for full transition to new environment
31
Debt
Partial use of Dependency Management
Build Workflows were stitched together with custom PowerShell scripts
Branches existed, but were not well thought out and developers were not using them consistently.
Not all branches were built, only when entering a main line branch
No versioning across most of code base
Code Ownership
Everyone owned everything = No one owned anything
Desired
Measurable KPIS
Unit Tests Static Analysis Integration Tests Performance Tests Code Coverage
Standardized Build Processes
-Standard pipelines for Widgets – Nuget, Products, MSI, ClickOnce, eventual Java Maven, Python PIP packages etc
Evaluated – Cloud and Self Hosted
Created use cases - IE: Encryption - at Rest and Transit (0-2 points)
Performed a POC with Azure DevOps and Github Enterprise
- Azure DevOps
Cost less per agent – due to agent images managed by MSFT
Found our counterparts in Irving, TX still use private agents for private software installs and build speed – negates the agent savings
Found our counterparts in Irving, TX also were getting updates to agent images they didn’t want and was causing build breaks
Jenkins
All agents are managed by us – but we can reuse a lot of what MSFT has done and Jenkins has done for build agents – so we are not designing from the ground up.
Can keep the CI environment entirely private
Can promote to internal artifact repositories.
Most amount of integrations – Dependency Management
Python
C#
Java
Take a single TFS Team Project and split it into multiple Git repositories
High level design is needed for standardizing CI System integration
How to version software
What file is updated by developers for Semantic Versioning?
Important Points
Build system determines which repositories can be used from Artifact binary manager system based on branch
Control of external packages was desired
How would this design fit into a large company
This is only one dependency type, Nuget.
Build artifacts for products generate deployable pieces like – MSI, ClickOnce, Zip go to different repositories
Enable use of development dependencies when on development branches
Non-Secrets
Linux/Windows Cluster ARN
Dev Stack
Master Cluster ARN
ECR repository path – for each image
Artifactory URL
Windows/Linux ASG Names/ARN
SECRETS
Put in Secret Manager, read via role
Master container
Can run as a specific role for access
Used by the WithAws Jenkins pipeline steps
Linux Cluster for linux containers
Windows for Windows
CloudWatch Alarms for scale up and down
During Scale down – we have a problem
What is the usual use case for containers? Fleet. Build jobs are not fleets, cant easily stop them
Need to start draining an EC2 instance and then kill after all builds are done.
Short Jenkinsfile
Pro/Cons:
Pro
Standard build template
All projects of type X build the same
DRY principle
Onboarding
Cons
Developer freedom – Justify why you need it to be different, better project management
First Bullet - Spins up a container in one of many ECS clusters based on Build queue and node label.
With transitioning from an unstructured branching strategy to a structured branching strategy, folks needed to learn why they use specific branch names
Versioning was new, so something folks did not have to think about before.
With 150+ git repositories now
Naming Conventions
Repository dividing lines – what pieces of code belong in one repository versus a separate repository
Standardizing the look and feel of common repositories. For example Nuget ones
Enforced Code Reviews
Git Repository Settings – Overcome with custom CI integration – REST API for Github
Manual Testing
A lot of manual Integration/System testing. Next on our maturity list
Jenkins Windows images
Size
Installing VS 2017 on the CLI
https://github.com/microsoft/azure-pipelines-image-generation
What issues do we think we could see:
Open PR from wrong
What issues do we think we could see:
Open PR from wrong
What issues do we think we could see:
Open PR from wrong