6 Design Principles for Azure Pipelines

As companies grow, pressure inevitably builds on different parts of their systems. With more customers, certain code paths start to see increased traffic, and developers begin to overlap in unexpected areas, leading to code conflicts.

Empower has recently faced these challenges in its delivery system. Our backend is organized as a monorepo, meaning nearly all backend components are housed within a single repository. Among these components is a service uses by our support team. This service shares a data model with our main API, which means we have to version and deploy both pf them together. As teams began needing to deploy these services separately, we encountered maintenance challenges due to coupling of our build and deployment implementations across these services.

We decided to restructure our deployment system to address the general maintenance overhead using a new model. This model was discussed in a previously post - which you can check out if you’re interested. In summary - the model was to build the mono-repo continuously, and have the build be decoupled from deployments. So all backend services would be built together, but they would be deployed independently.

Throughout this process of refactoring and rebuilding, we identified and established several key principles to follow to keep our pipelines comprehendible and maintainable. I’d like to share these principles and associated insights with you in this post.

Principles for Developing Deployment Systems in Azure

The following are 6 core principles you can follow to achieve success in your azure pipeline implementations. You come across fair bit of yml in this post - so apologies for that! In that yml, three are variables and parameters referenced. Parameters can come from the pipeline global scope, or from template scopes. Variables will typically come from the global scope.

1. Decouple Build from Deploy

This is more of a reiteration of the general model described in the previous post, but its worth enshrining as a principle: Do not pollute deployment pipelines with build pipelines. Doing so keeps maintenance complexity down and prevents build overhead during deployments, making them much faster.

In times when you need to redeploy quickly, having to wait for a build is a significant time sink. Also - when delivering software, we should always aim to follow a ‘build-and-publish-once’ philosophy. Once your software is built and published - that is the final version prepared for a release. You can test it multiple times, you can deploy it multiple times, and you can promote it through environments - but you should only ever build and publish it once. As soon as you recompile (and download external dependencies…) and publish - you should consider it a new version.

2. Design at the Job Level

Pipelines, whether single or multi-stage, are nothing more than a collection of jobs that execute in some sequence. Jobs represent all of the actions your pipeline will take while on a given agent (i.e. worker computer). Having the ability to define distinct jobs allows for things like parallel execution and encapsulation of tasks.

Jobs should be meaningful encapsulations of components of your build or deployment pipelines.

Consider the following job:

jobs:
  - template: ../jobs/CalculateVersionWithReleaseBranchOption.yml
    parameters:
      jobName: SetVersion
      versionSetStepName: SetVersionStep
      versionVariableName: AppVersion
      releaseBranchName: ${{ variables.releaseBranchName }}
      bumpMinor: True
      variableGroupName: ${{ variables.mainlineVersionTrackingName }}
      variableGroupKeyId: ${{ variables.mainlineVersionTrackingId }}
      devopsUrl: ${{ variables.devopsUrl }}

It should be clear from the name of this job template that we have encapsulated all of the concerns of calculating a release version in this job. This keeps this job’s purpose plain and understandable for maintainers reviewing the design of the pipeline, and ensures that we can re-use this template in any pipeline that requires version calculation.

Lets consider this principle in the context of the full pipeline

jobs:
  - template: ../jobs/CalculateVersionWithReleaseBranchOption.yml
    parameters:
      jobName: SetVersion
      versionSetStepName: SetVersionStep
      versionVariableName: AppVersion
      releaseBranchName: ${{ variables.releaseBranchName }}
      bumpMinor: True
      variableGroupName: ${{ variables.mainlineVersionTrackingName }}
      variableGroupKeyId: ${{ variables.mainlineVersionTrackingId }}
      devopsUrl: ${{ variables.devopsUrl }}

  - template: ../jobs/CompileAndPublish.yml
    parameters:
      jobName: CompileAndPublish
      repository: self
      dependsOn: SetVersion
      condition: succeeded()
      agentPool: ${{ variables.linuxVmPool }}
      version: $[ dependencies.SetVersion.outputs['SetVersionStep.AppVersion'] ]
      artifactName: pub
      pathToSolutionDirectory: $(Build.SourcesDirectory)/api
      pathToEFDbContextProjectDirectory: $(Build.SourcesDirectory)api.data
      runtimeProjectMap:
        runtime-a:
          - api.Tests
          - api
          - api.webjobA

  - template: ../jobs/TestTheSolution.yml
    parameters:
      solutionArtifactName: pub
      retryCount: 3
      dependsOn:
        - SetVersion
        - CompileAndPublish
      agentPool: ${{ variables.linuxVmPool }}
      testTargetName: api.Tests
      condition: succeeded()
      version: $[ dependencies.SetVersion.outputs['SetVersionStep.AppVersion'] ]
      testMatrix:
        UnitTests: TestCategory!=IntegrationTest
        IntegrationTests: TestCategory=IntegrationTest

  - template: ../jobs/PushPackages.yml
    parameters:
      jobName: PushPackages
      condition: succeeded()
      version: $[ dependencies.SetVersion.outputs['SetVersionStep.AppVersion'] ]
      artifactsToDownload:
        - a
        - b
      agentPool: ${{ variables.linuxVmPool }}
      dependsOn:
        - SetVersion
        - CompileAndPublish
        - UnitTests
        - IntegrationTests

A quick skim of the template names should have it become apparent what is going in this pipeline:

1. Calculate a new version
2. Compile and Publish the solution
3. Test the solution
4. Push deployment artifact packages to a place where they can be later retrieved

It also should become clear that when we assemble dependencies between jobs, we’re assembling a directed-acyclic-graph, or DAG. All successful pipelines are technically DAGs, but the point here is that _it is obvious that we’re composing a DAG when we start our model implementation designs at the job level. _

One thing that makes pipelines so confusing to follow when this principle is not followed is that it becomes a difficult challenge to model the DAG clearly in your head.

3. Think of Templates as Functions

For any given job, there are likely logical groupings of behaviors. These related behaviors should be encapsulate within template files - not just for reusability, but also for clarity of purpose.

Frankly - one of the most challenging aspects of designing and maintaining azure pipeline yml is simply understanding what any given block of yml actually does. Variables can come from anywhere, yml can be extensive and hard to follow.

Leveraging templates as functions solves this problem by providing, essentially, a label to a logical grouping of behavior. Take for example the following template used to invoke dotnet test

parameters:
  - name: settingsFilePath
    type: string

  - name: filterString
    type: string

  - name: displayName
    type: string

  - name: testTargetCsprojOrDll
    type: string

  - name: retryCount
    type: string
    default: 0

steps:
  - task: DotNetCoreCLI@2
    displayName: "Status Check: ${{ parameters.displayName }}"
    retryCountOnTaskFailure: ${{ parameters.retryCount }}
    inputs:
      command: test
      projects: ""
      publishTestResults: false
      arguments: |
		      ${{ parameters.testTargetCsprojOrDll }}
		      --filter ${{ parameters.filterString }}
		      -d ${{ parameters.displayName }}Logs.$(Build.BuildId).txt
		      --configuration Release
		      --logger trx;verbosity=detailed
		      --results-directory ${{ parameters.displayName }}Results.$(Build.BuildId)
		      --collect "XPlat Code Coverage"
		      --settings ${{ parameters.settingsFilePath }}

  - publish: ${{ parameters.displayName }}Logs.$(Build.BuildId).txt
    artifact: ${{ parameters.displayName }}Logs

  - task: PublishTestResults@2
    displayName: "Post: Publish Test Results"
    condition: succeededOrFailed()
    inputs:
      testResultsFormat: VSTest
      testResultsFiles: "*.trx"
      searchFolder: ${{ parameters.displayName }}Results.$(Build.BuildId)
      mergeTestResults: true
      buildConfiguration: Release
      publishRunAttachments: true
      failTaskOnFailedTests: true

  - publish: ${{ parameters.displayName }}Results.$(Build.BuildId)
    artifact: ${{ parameters.displayName }}Results.$(Build.BuildId)
    displayName: "Post: Publish TestResults"

This template groups together behaviors for executing tests as well as publishing the results. It does so in a way that generalizes the function outputs which are artifacts. These behaviors will always move together, and if I absolutely must disable publishing for some reason, I can introduce a parameter to facilitate that.

Template ‘functions’ can then be composed into more complex templates, or directly in jobs. Consider the following job, which is a composition of templates:

parameters:
  - name: jobName
    type: string

  - name: jobDisplayName
    type: string

  - name: reposWhereToCreateReleaseBranch
    type: object
    default: []

  - name: versionSetStepName
    type: string

  - name: versionVariableName
    type: string

  - name: condition
    type: string

  - name: releaseBranchName
    type: string

  - name: createReleaseBranch
    type: boolean
    default: False

  - name: bumpMajor
    type: boolean
    default: False

  - name: bumpMinor
    type: boolean
    default: False

  - name: bumpPatch
    type: boolean
    default: False

  - name: saveToAdo
    type: boolean
    default: True

  - name: variableGroupName
    type: string

  - name: variableGroupKeyId
    type: string

  - name: devopsUrl
    type: string

  - name: dependsOn
    type: object
    default: []

jobs:
  - job: ${{ parameters.jobName }}
    displayName: ${{ parameters.jobDisplayName }}
    dependsOn: ${{ parameters.dependsOn }}
    continueOnError: False
    condition: ${{ parameters.condition }}
    workspace:
      clean: all

    variables:
      VariableGroupName: ${{ parameters.variableGroupName }}
      VariableGroupId: ${{ parameters.variableGroupKeyId }}

    steps:
      - template: ../steps/LogAgentState.yml

      - template: ../../utils/IncrementVersion.yml
        parameters:
          saveToAdo: ${{ parameters.saveToAdo }}
          variableGroupName: $(VariableGroupName)
          variableGroupKeyId: $(VariableGroupId)
          devopsUrl: ${{ parameters.devopsUrl }}
          setVersionStepName: ${{ parameters.versionSetStepName }}
          versionVariableName: ${{ parameters.versionVariableName }}
          bumpMajor: ${{ parameters.bumpMajor }}
          bumpMinor: ${{ parameters.bumpMinor }}
          bumpPatch: ${{ parameters.bumpPatch }}

      - ${{ if eq(parameters.createReleaseBranch, True) }}:
          - ${{ each repo in parameters.reposWhereToCreateReleaseBranch }}:
              - checkout: ${{ repo }}
                displayName: 'Prep: Download ${{ repo }} Source'
                clean: True
                fetchTags: False
                lfs: False
                submodules: False
                persistCredentials: True
                fetchDepth: 0
                continueOnError: False
                path: ./s/${{ repo }}

              - template: ../steps/CreateNewReleaseBranch.yml
                parameters:
                  repositoryRootDirectory: $(Pipeline.Workspace)/s/${{ repo }}
                  releaseVersion: $(${{ parameters.versionSetStepName }}.${{ parameters.versionVariableName }})
                  releaseBranchName: ${{ parameters.releaseBranchName }}

          - task: PowerShell@2
            displayName: 'Update build number'
            inputs:
              targetType: 'inline'
              script: |
                $newVersion = " 🚧 Release $(${{ parameters.versionSetStepName }}.${{ parameters.versionVariableName }}) under construction 🚧"
                [string] $buildName = "$($newVersion)"
                Write-Host "Setting the name of the build to '$buildName'."
                Write-Host "##vso[build.updatebuildnumber]$buildName"

      - ${{ else }}:
          - task: PowerShell@2
            displayName: 'Update build number'
            inputs:
              targetType: 'inline'
              script: |
                $newVersion = " 🚧 Build $(${{ parameters.versionSetStepName }}.${{ parameters.versionVariableName }}) under construction 🚧"
                [string] $buildName = "$($newVersion)"
                Write-Host "Setting the name of the build to '$buildName'."
                Write-Host "##vso[build.updatebuildnumber]$buildName"

4. Parameterize values that are not intrinsic to the template

When designing the boundaries of a given template, you will frequently need to make judgement calls on what is a concern or responsibility of a template, and what is not. A decent rule of thumb to follow for this is: behaviors are intrinsic, details are not. This will no always be true, so lets see an example:

parameters:
  - name: version
    type: string

  - name: dependsOn
    type: object
    default: []

  - name: jobName
    type: string

  - name: condition
    type: string

  - name: artifactsToDownload
    type: object
    default: []

  - name: agentPool
    type: string

jobs:
  - job: ${{ parameters.jobName }}
    displayName: 'Push Packages'
    continueOnError: False
    dependsOn: ${{ parameters.dependsOn }}
    condition: ${{ parameters.condition }}
    pool: ${{ parameters.agentPool }}
    variables:
      Version: ${{ parameters.version }}

    workspace:
      clean: all

    steps:
      - checkout: none
      - ${{ each drop in parameters.artifactsToDownload }}:
          - task: DownloadPipelineArtifact@2
            displayName: 'Gather Packages'
            inputs:
              buildType: 'current'
              targetPath: $(Pipeline.Workspace)/drop
              patterns: '**/*.zip'
              artifact: ${{ drop }}

      - publish: $(Pipeline.Workspace)/drop
        artifact: Release-$(Version)
        displayName: 'Push'

      - script: rm -rf $(Pipeline.Workspace)/drop
        displayName: 'Remove downloaded artifacts'

There are two values I’d like to point out with this example. The first is an example of a non-intrinsic value. The second demonstrates an intrinsic value.

Intrinsic

targetPath: $(Pipeline.Workspace)/drop

Non-Intrinsic

- ${{ each drop in parameters.artifactsToDownload }}:

Drawing the line between intrinsic and non-intrinsic values prevents us from introducing magic into our pipelines - which only serves to confuse maintainers about their behavior.

5. Parameterize values that are likely to change

Certain variables - when loaded into a pipeline - can be accessed from anywhere. Consider the following variable group, which defines a variable MyVariable, which is then accessed:

name: MyPipeline
variables:
  - group: MyLibraryVariableGroupInADO
jobs:
 - job:
		steps:
			- tempalte: ./MyTemplate.yml
				parameters:
					someVariable: $(MyVariable)

If we were to access this variable from within a template, as apposed to passing it to a template as a parameter, we would immediately lose sight of its presence and introduce a potential maintenance bug (whereby we forget to update it).

6. Use a Global Variable Template

The final principle I’d like to share in this post is not without some contradiction to advice against introducing magic - however its been found to be a major benefit in the the Empower pipelines. A global variable template is simply a central place to provide access to a common set of variables that may be used by one or more pipelines.

Consider the following global-variables.yml

variables:
  #  agents
  - name: linuxVmPool
    value: linuxAgent

  # service account PAT
  - group: CommonPipelineAccessTokens

  - name: githubRestApiAccessToken
    value: $(GitHubPAT)

This can be imported into any pipeline using:

variables:
  - template: ../../global-library-variables.yml

This does obscure the existence of the variables with bit of misdirection, however that is offset by the sheer convenience of reusability and it well mitigated by following principles 4 & 5.

---

About Paul Gradie

_Paul is a software engineering leader / data scientist / biologist / team player / father with published contributions in the fields of reproductive biology, artificial intelligence, and software engineering. _

*Paul enjoys pair programming, learning, and building things well and is often told that he is personable, approachable, and enjoyable to work with 😀. He also maintains *Sailfish_ (a performance testing library for C#), and endeavors on / contributes to various side projects to hone his skills._

Connect with Paul