I recently published a post criticizing the state of developer tooling in AWS. Today I’ll be more constructive and describe the features that encompass a proper pipeline service.
Please note: this article is about pipeline services, not a specific pipeline. In other words, we’ll look at a tool used to create pipelines, not the pipeline itself.
A pipeline service should be versatile, allowing developers to create pipelines which deploy their applications and infrastructure in a way that works for them. A pipeline service should support multiple sources, be flexible in the number of build steps, allow for parallel builds, and so on.
In this article we’ll explore a number of core features to support many different types of workflows. There are some references to AWS, but these principles are generic and should apply to any cloud or SaaS solution.
Any Git repository, any authentication method
This might seem like an open door, but for some reason generic Git repositories are not supported in AWS CodePipeline. For a pipeline service to be successful, it should not require developers to host their application on a specific platform like GitHub. Instead, the pipeline service should support the raw Git implementation and allow developers to choose whether to use HTTPS, SSH with username and password, SSH with SSH keys, or OAuth deploy keys.
By supporting these options, the pipeline service offers maximum flexibility while allowing developers to utilize basic security measures like read-only deploy keys.
Any branch, any commit
Many pipelines services require a pipeline to be hardcoded to a specific branch; one pipeline which detects changes on the develop branch, then builds and deploys environment A, and a completely separate pipeline that detects changes on the main branch, then builds and deploys environment B. This might be fine 80% of the time, but it’s a real hassle in the other 20%.
A common example that does not work in this setup is a hotfix branch, typically used to branch from main to fix a high priority bug. In some cases, this fix is experimental - you do not know if it will resolve the issue. If your pipeline is ‘stuck’ on the main branch, you would need to merge this untested change to the main branch before deploying, which goes against Git principles.
Of course, an argument of ‘no untested code should be deployed in production’ can be made, but not all of us are Google, and some of us need to do deployments in the middle of the night without anybody available to do a code review.
Another use case for dynamic branch selection is deploying feature branches into an acceptance environment. There are use cases where the feature branch has been reviewed, but needs to be tested in a very specific context. Instead of merging the change before executing the test, developers should have the freedom and responsibility to deploy a feature branch to an acceptance environment, testing the feature, and merging the change after it has been verified.
Expanding on the ‘any branch’ feature, a pipeline should allow developers to deploy a specific commit, not just the HEAD of a branch. Maybe your developers have been working on a feature, but decide that they want to test an earlier version to compare the differences. The pipeline should not force users to use HEAD or require them to create a separate branch from a commit to be able to build older commits.
To support these use cases, a pipeline should be flexible in its branch selection, and a pipeline service should either allow a pipeline to be easily updated or, better yet, decouple the source, build and deploy steps altogether. More on that in the next chapter.
Decoupled source, build and deploy steps
To resolve the ‘any branch, any commit’ issue, a pipeline should offer an interface that displays all the branches, commits and tags in the repository, and allows developers to choose which commit to run the pipeline on. This means the source and build steps are decoupled. Instead of running the pipeline when a new commit is detected on a preconfigured branch (strictly coupling the source and the build step), the source repository’s metadata should synchronize independently, and choosing a commit should result in an artifact that is used as the initial step for the pipeline.
You might wonder how decoupling the source and build steps impacts automated testing and deployments. Clearly, we can’t run the full pipeline on every branch and every commit. We’ll cover this problem in the ‘Automated builds and releases’ section later in this article.
Once a source commit has been chosen, it should be built. Optionally, this step can be skipped, instead moving the source artifact to the deploy step immediately. More on the build step follows in the next chapter.
At this point, either the source step has provided an artifact that’s ready to deploy, or the source artifact has been built by the build step, and that has resulted in a deployable artifact.
Promoting builds
To guarantee that the code running in production is exactly the same as the code that was tested in an acceptance environment, it should be possible to ‘promote’ an artifact from acceptance to production. To achieve this, developers (or an automated system) should be able to choose where to deploy a build artifact - to a developer, test, sandbox, acceptance, production or any other environment.
This results in decoupled build and deploy steps, and the artifact is used to glue any build result to any deploy environment. Through this mechanism, any commit can result in an artifact which can first be deployed to acceptance, and then be deployed to production. This solution moves the responsibility for deploying the right commits to the different environments from the pipeline’s configuration to the engineers. With great power comes great responsibility - in other words: this allows engineers to **** **** up big time, for example by deploying an untested commit to production - but with the fine grained access control discussed in a later section, the pipeline maintainers will be able to manage who wields this power.
“Wait!” I hear you shout from the audience, “doesn’t that mean that all environments will be production sized?”. But no, this can be controlled through parameters. In other words, the deployment process ‘knows’ which environment it’s deploying to and will read the correct parameters for that environment, for example configuring t3.medium instances for acceptance and m5.xlarge instances for production.
Build anything
The build step described in the previous step is used to convert any source artifact to a deployable artifact. That might mean running unit tests, compiling files, compressing or packaging files, cleaning caches, generating CloudFormation templates, or anything in between. To support this, the build phase should be able to run about any kind of script or code, including Ruby, Python, Bash, .NET, NodeJS, Zip, Terraform, CDK, and so on. AWS CodeBuild is an awesome single-run container platform that is perfectly suited for this requirement.
Rollbacks
An essential feature for any pipeline is the support of rollbacks. Simply put: the pipeline should be able to revert back to a previous version of the environment in case any issues with the latest release are encountered. This can either be done automatically, for example when a test reports a failure, or manually, in case engineers or customers report an issue. This strongly relates to the next chapter: build and release history.
Build and release history
Because the source, build and deploy steps have been decoupled, releases are not guaranteed to match any Git branch’s chronology anymore. For example, the acceptance environment might have releases from a feature branch, the dev branch, and older feature branch and the main branch, in any order. Likewise, the last commit on the main branch (commit N) might be deployed to production, but commit N - 1, N - 2 and N - 3 might have not have been, making the previous release on production commit N - 4. Because of this, it is important to have an overview of every build and release in the environment, including build logs and audit trails.
When a release history for an environment is available, rollbacks can be implemented by including a ‘redeploy this build’ button for every build artifact.
Fine-grained access control
The pipelines created are inherently flexible and powerful, which means that any user with sufficient access can deploy any (potentially broken) feature to any environment. To mitigate this, the pipeline service should provide fine-grained access control, allowing specific users or groups to edit pipelines, build sources, and deploy to specific environments. When properly implemented, this allows some engineers to deploy only to their own environments, some to the acceptance environments, and only a small subset to be responsible for production releases.
Automated builds and releases
There are a number of cases in which automated builds or releases can be beneficial or necessary. Common situations are automatically building and testing any commit on the develop branch, or automatically deploying the HEAD of the main branch to acceptance.
Another common scenario is scheduled scaling, for example shutting down developer environments when they are not in use or automatically scaling down acceptance environments outside of office hours.
To provide solutions for both use cases, the pipeline service should allow specific branches or tags to be configured for automated releases; when a new commit on this branch or of this tag has been detected, the pipeline will automatically build that commit, generate a build artifact, and deploy this to the selected environment.
To achieve the scaling objective, a scheduled trigger should be able to select a source commit and to supply environmental variables to the build and deploy processes, for example SCALE_DOWN = true, so the deployment knows it should use smaller instances or set autoscaling groups to a maximum of 0 instances.
Immutable pipelines - or not
To support various uses cases, the pipeline service should both allow the deployment process to be mutated through the code in the source repository, and be able to block that. To clarify, many pipeline services allow the build and deployment process to be defined in a pipeline.yml, deploy.yml, or buildspec.yml file. This is very powerful, because it allows the developers to change deployment features based on what that specific version of the application requires. However, managing and maintaining the deployment pipeline can also be another team’s responsibility, and in those cases the developers should go through the right channels to request changes to the build process.
An all-round pipeline service should be able to deploy both mutable and immutable pipelines.
Serverless
All of the features above can be achieved some way or another with existing products. However, these services can be very hard to deploy, maintain and secure, or have restrictive per-user-month licensing costs.
The perfect pipeline service supports all the features above as a SaaS, including maintenance of the underlying infrastructure and (of course) not exposing any hardware or operating system. High availability and scalability is an integral part of the service, and the cost model is pay-as-you go without upfront costs.
Conclusion
I know the article above reads like a ‘if I were a king, I’d ask for…’ piece. However, it is less of a pipe dream (ha-ha) than it might seem at first glance. Jenkins, Bamboo and Gitlab already offer very similar experiences, and CodePipeline and CodeBuild have the serverless part on lock. Azure DevOps and GitHub Actions also provide parts of this feature set, so what I’m hoping for is one of the big players to step up and bring it all together. And of course, I wouldn’t mind if that would be AWS.
To bring AWS CodePipeline closer to this reality, I would suggest starting with three features that I believe are relatively easy to implement (#awswishlist):
- Support native Git integrations
- Allow the result of one pipeline to be used as the source and trigger for another pipeline through a manual
upstreamstep - Rollbacks
Features that are harder to implement, but would make CodePipeline so much better:
- Full branch, commit and synchronization into CodePipeline
- A web UI to interact with this metadata, allowing users to choose which commit or tag to build
- Parallel builds (but not deployments)
- Scheduled builds with environmental variables for preconfigured branches
- Fine-grained access control in IAM or Cognito
Get in touch
If you have questions or remarks, or would just like to get in touch, you can find me on Twitter and LinkedIn.
