With this article I’m going to share my experience building continuous delivery ecosystems that is based on a couple of projects I’ve been participating. They all are based on our wins and failures so I don’t just put some theory on paper.
Pipelines
A typical continuous delivery ecosystem contains a number of deployment pipelines. Every pipeline has a repository in a version control system. In this case we use a trunk based development model. The trunk based development model enforces early integration which helps to avoid branch merging overhead. Here is a whole website dedicated to the trunk based development. This is how a common pipeline workflow looks like:
- Triggered on commits to its repository;
- Build and/or validate the source code from the repository;
- Run unit tests;
- Create a versioned package;
- Deploy the package to continuous integration environment;
- Run tests dedicated to continuous integration environment;
- Push the package for release.
Pushing a package for release is a manual step. Developers can decide to release the package or to do more commits. Don’t forget to push often to avoid big releases. Big releases are hard to test and they have a much higher risk of having problems on production. It’s also important to avoid changes for more than one task/story/bug in one pipeline. Pushing often can help with this as well.
After a package has been pushed for release a release candidate can be created. Then the release candidate will be deployed on a regression environment, tested and promoted to a production environment. In fact the release candidate is just a record of the latest versions of packages in all pipelines and also an order in which these packages have to be deployed. When deploying the release candidate a package deployment will be skipped when the same version is already deployed to a target environment. Below is a diagram explaining the process:
Testing
Now let’s talk a bit about testing. In case of continuous delivery it’s important to have a good test automation in place. Because when you deliver often there not much room for manual testing. We basically have to automate as much as possible. In the setup explained above we have three points where tests can be executed:
- unit tests executed before deployment to continuous integration;
- tests focused on one pipeline that run over continuous integration environment;
- cross pipeline tests that run over regression environment;
To reduce time of a full regression test run we execute tests only once and focus on different aspects of the quality. By the way, it’s always better to have a unit test instead of a test running on continuous integration or regression environment. Because unit tests run faster and its feedback loop is shorter. Unfortunately not everything can be covered with unit tests. So the next stop is continuous integration environment. When your test is focused on one pipeline this would be a natural choice. When a test covers multiple pipelines it can be executed on a regression environment. This approach has been developed on top of the test pyramid concept developed by Mike Cohn. Here is an article from Martin Fowler describing the test pyramid concept.
Dependencies
As you can see on the diagram above there are four types of pipelines: applications, workflows, services and repositories. The idea is to structure a platform in such a way that there are applications on top, they can talk to pipelines below and never to other applications. Workflows are just below applications and they can talk to pipelines below and never to other workflows. Services are just below workflows and they can talk to repositories and never ot other services. When we structure pipelines this way it’s very easy to deploy without breaking dependencies. Repositories are always deployed first, then services, then workflows and finally deployed applications. When api of a pipeline changes we make sure we support one version back and this should be enough to not to brake the system.
Feature toggles
When we deliver continuously it means that on production could be features that not ready for end users. Usually that’s not what business wants. In order to be able to do continuous delivery we have to use feature toggles and this is how it could be implemented. I introduce three stages of ready:
- Alpha - a feature is not ready at all but we want it on production environment for early integration;
- Beta - a feature is implemented and tested but we want to show it only to a limited set of users;
- Commercial - a feature is implemented and visible for all users.
Features in all these stages can be deployed to production environment although alpha and beta functions have a hardcoded switch to be shown only for users who are enabled to see those type of functionality. There is another trick to keep track of what features in what stage. On my last project we found out that we can turn our board around and have columns like Alpha | Beta | Commercial. When developer take a story into progress he puts it into Alpha and it reminds him to implement a switch. When he has done with implementation and has everything deployed to production he puts a flag on the story so quality engineers could carefully test it. When quality engineers are done with the story they ask developers to change the hardcoded switch to Beta and to move the story to Beta column. Then marketing people can take a look and prepare necessary changes in documentation and do other stuff that has to be done before going life. When all preparation is done developers can simply remove the hardcoded switch and move the story to commercial column. In my case this also means the story is done. This approach doesn’t work very well for small fixes although there is no silver bullet as we know.
Happy coding,
Michael Borisov.