Modeling World-Class Delivery Systems

Introduction

Whether you’re a software engineering purist, a DevOps enthusiast, a tinkerer, or even a manager - we’re all modelers. In other words, we are always trying to find abstract conceptual framings that help us implement a structure when we’re building systems of any kind. Those systems can be a business, a product, a tool, a game, or even a process for running a team.

The typical modeling scenario software engineers or developers face is trying to implement a product that supports a business. This is something I’ll bet many of you are quite familiar with. Perhaps you have implemented a store and you need to serve behaviors to a webpage that will allow customers to browse a catalog, or pay for some products. Or maybe you’ve done something more like what Empower Finance does, and you’ve facilitated the movement of money from one bank to another.

Either way, you’ll talk conceptually about your problems, which need to be translated into concrete implementations as code.

When it comes to implementing a software delivery systems - this is also true.

The Language of the Delivery System

Modeling begins with speaking conceptually about a problem - in this case, software delivery. Doing so will let us develop and evolve a language about the domain of software delivery just like any other domain.

In this post, we’ll develop a rudimentary language about delivery to see what I mean.

To kick things off, let’s come up with a most basic story:

“I’m going to write some code and then run that code in the cloud”

To do this, we’ll need to first compile our code to an executable, and then copy those executables to something that will execute them. This gives us our first two concepts: Build (compile) and deploy (copy).

Developing our Conceptual Model: The Build Process

Let’s talk about this concept of **Building **and expand on our story.

Let’s say you’ve finished version 1 of your application and you’re thinking about sending it off to the cloud. You’ve considered manually moving your published software, but prefer to stand up an automated solution because you will be iterating new versions quite rapidly and want to continuously send out updates without too much hassle.

Versioning and Publishing

We’ve already encountered our first concepts: Versioning and Publishing. Let’s make a note of those since we’ll likely need to bring those into our system at some point.

If we were to do this manually, we would have to take several steps in sequence - for example we would have to compile the application, and then publish it. Next we would need to run tests against it.

At this point, we’ve introduced the concept of a series of **steps **that we’ll need to take - in other words, a **process **of some kind. We can already start to assemble these conceptual blocks in order:

• Compile the application
• Publish the application
• Assign a version to the published application
• Test the application

Once testing is complete, we are left with a set of compiled binaries that need to be moved onto the actual machine that will execute them - so we will need to assign each to one or more targets.

Packages

Naturally, we may need to send a number of files off to the same place - so it makes sense to group these files together in a neat little package, like a package you might receive from an online retail store that has a few items in it. These packages represent deployable units which will always need to be sent to the target together. These packages will typically include the main executable, as well as any dependencies that the executable will need to load in order to function.

The Release

We may need to build one or more **packages, **which will have been versioned together. Conceptually, we can think of this as a single object that can be released into the wild - so let’s call that object a Release. A release is therefore a logical grouping of deployable units called packages, which are versioned and deployed together.

Build Process Summary

We haven’t come that far, but we’ve already loaded up our language with enough concepts to start telling a story that will help inform the **Build **phase of our delivery system. Here’s what that might read like:

Automating a delivery system begins with defining a **process** which is composed of several **steps**. The first step is to download the source code to an **agent **that will act on our behalf. The agent’s **job** will be to perform the following **steps**:

1. Compile the app
2. Publish the app (which may produce one or more directories of multiple executables)
3. Version all of the executables
4. Execute the published test executables
5. Zip each deployable unit into a package in preparation for deployment.
6. Collect all of the packages that have now been published, versioned, and tested together into a Release

This is what we’ve done at Empower. We’ve used this conceptual model of a build process to create our **MainLine - which is responsible for continuously building merge commits on the default branch into versioned Releases. **The mainline is something like a conveyor belt, which is constantly pushing out new versions of the product - built, tested, and packaged - ready for deployment at any time.

Decoupling the Build Process from Everything Else

This feels like a good stopping point for this agent’s job. Why? Because a few things might happen at this point that will prevent us from deploying.

For example - perhaps one or more tests have failed. Or perhaps we’ve decided that we simply don’t want to send this version of the product out to the targets until there are more features added. Perhaps the application failed to compile at all (there are a variety of reasons this one can happen, but let’s save that discussion for a future post). The point is we have good reason (informed by experience) to stop here.

Since we haven’t actually deployed anything - let’s call this our MainLine build process.

So conceptually, we have now developed the language of a build process, which is an early phase of the delivery system. And by doing so, we have conceptually decoupled the build process from what comes next - the deployment process.

There is enormous benefit in decoupling the build process from everything else. If you think of the delivery system as the nexus for change in your business, then being able to manage, modify, extend, and make continuous your build not only unlocks better delivery speed - it:

• enables you to control the change sets that are delivered which helps minimize risk when shipping changes
• let’s you pre-build releases so that you can ship them on demand at any time
• enables you to spawn any number of consumer processes thus keeping your system more understandable and far easier to extend
• improves your ability to recover from disaster
• more easily create and deploy to new environments as your business grows
• and more!

Adding to our Conceptual Model: The Deployment Process

Now that we have a release ready to deploy, the next phase of our delivery system is to actually deploy them. This also requires a number of steps. So again, we have a process of steps that we’ll use to deploy our packages - let’s call that a deployment process.

Recall that the build process produces packages grouped into a Release. That release was then staged for consumption by other processes. So - our deployment process will want to consume those packages. Recall also that we have kept these two phases of our delivery system decoupled, so we’ll need to remember to introduce concepts that indicate that we’re **feeding **our deployment process.

Feeds

The concept of a feed may be familiar to some - and it’s a nice way to think of where the build process will generally place packages. Build processes push packages (or a release) into a feed, which then feeds into other processes.

Targets

A deployment process will consume a release from a feed, and it will probably need to **promote **that release through a number of stages. For example, the first **stage **will be to copy one or more packages from the feed to various targets - a webapp or some place else.

Environments

Most companies - like Empower - will want to take a staged approach to delivering their software by progressively promoting their software through multiple environments. A very common pattern is to first copy files into a Development Environment - which is a set of targets that are interacted with by the developers wishing to test the application in a ‘live’ environment that is not their own computer.

**Environments ** are a collection of targets that are set up similar or even identical to those in the final live environment - normally referred to as the Production Environment.

So to summarize - deployments often follow a particular lifecycle of environments:

Development - Test - Production

Sometimes, we might need to send a change directly to production - usually referred to as a Hotfix. In this case, we’d need a different lifecycle:

Production-Only

⚠️ This is a very special - and dangerous - lifecycle that should only be used in emergencies ⚠️

Or perhaps for some reason, we want to stop our deployment at the Test environment. For this, a lifecycle might look like:

Development-Test

The concept of **lifecycles **is powerful because you can use them to inform an implementation that allows teams to deploy to different environments without interrupting one another.

For example - we recently had the need for a feature team to *only *deploy to our test environment (using a Test-Only lifecycle) so that our regular deployments could continue while the feature team made changes - which would have otherwise prevented the deployments through to the Production environment. Our QA team focused their testing on Dev during this period and both teams were able to proceed without impeding the other.

ℹ️ Just to note - all the normal testing was still undertaken during this procedure! Automated testing also still occurred during the build phase.

Having concepts like these in our model can be a great productivity boost for our teams!

Stages vs. Environments

To put a finer point on it - a stage is not necessarily an environment - a stage can be a higher level abstract component of any process. For example - You may promote a release to a Development environment in the first stage of your process. Then in the next stage, you may perform a collection of actions that do not directly involve deployments. We’ll see a clear example of this in the next post when we discuss implementation details and discuss design principles and how to best structure Azure pipelines specifically.

Post Deployment Actions

Lastly - we may have actions we need to take before or after a deployment occurs within a given stage to a collection of targets grouped within the environment. Conceptually, we can simply refer to these as **Pre-Deployment Actions **and Post-Deployment Actions.

Summarizing the Deployment Process

So to summarize the conceptual steps of a deployment process:

1. **Consume a Release **from a feed (which is produced by the build process)
2. Execute any **Pre-Deployment Actions **for the environment
3. Deploy the **Release **using one of potentially several different lifecycles to one or more targets to the initial stage (which will be linked to a specific environment)
4. Execute any Post-Deployment Actions
5. **Promote **the **Release **to the next environment or execute the next stage** **until the lifecycle is complete

Empower’s Main Deployment Pipeline

This is what our core deployment process looks like at Empower. each rectangle is a stage - which may or may not be an environment. For example, the first **Download Release **stage is not an environment - but it is a stage in the process. If we were to expand it, we would see it was comprised of multiple jobs, each with multiple steps. The final two stages are also not environments - they are independently controlled segments of the pipeline that made sense to place in their own stage.

Summary

With this, we have defined a basic model for deployments, and have developed a language that we can use with one another. This language can be extended, reshaped and shared in order to both keep everyone on the same page about the delivery system as well as disseminate knowledge about it more rapidly within the organization.

At Empower, we have found that modeling the delivery system in this way has unlocked an enormous amount of potential. We’ve:

• reduced our delivery time from hours down to minutes by decoupling the build process from the deployment process.
• given our QA team tools that are simple to understand by removing half of the overall complexity of the system (i.e. the build process)
• enabled rapid redeploys when necessary
• significantly simplified and expedited the rollback process
• introduced additional rapid paths to production in case of a critical error
• unblocked the work of multiple teams and substantially improved their iteration speed

And the list goes on!

Overall, a better model for our delivery system has enabled Empower to not only develop faster, but also to ensure that stable reliable software is delivered to our customers more quickly and effectively.

This is huge for both our developers as well as our client base - because we’re all about empowering people to do their best.

---

About Paul Gradie

_Paul is a software engineering leader / data scientist / biologist / team player / father with published contributions in the fields of reproductive biology, artificial intelligence, and software engineering. _

*Paul enjoys pair programming, learning, and building things well and is often told that he is personable, approachable, and enjoyable to work with 😀. He also maintains *Sailfish_ (a performance testing library for C#), and endeavors on / contributes to various side projects to hone his skills._

Connect with Paul