If your goals are high-speed software development and getting working builds to production often, you need to automate at least part of the testing and deployment process. Ideally, this means implementing CI / CD pipelines for your projects, along with test-suites to detect errors before customers see the software, and scripts that implement the pipeline’s steps.
Continuous Integration (CI) is a method for the consistent automation of software builds, packaging and tests. CI helps reassure a team that changes they check into source code version control won’t break the build or introduce bugs into the software. The endpoint of CI is usually a completed check-in to the main branch of a software repository.
Continuous Delivery (CD) automates the delivery of tested software to infrastructure environments. That usually doesn’t mean throwing it straight into production to see if customers complain. Typically, organizations begin by moving the build to a development environment. After the developers themselves post and publish the new build, it usually goes to a test environment where it is used by a broader group of users (sometimes just dedicated in-house testers, sometimes a larger cadre of users signed up for beta tests or “dog food “) And closely monitored. If all goes well, the testers finally log out and bring the new version into a production environment.
At each stage of the CD, there are options to quickly revert to an older build and generate bug report tickets for developers to work on in the new build. The goal is not to get many builds into production, but to continuously improve and expand the software without introducing regressions. Another term for these practices is “devops”.
Why host CI / CD in the cloud?
Hosting a CI / CD platform in your own data center is a viable option, especially for companies that require their applications and data to be hosted inside the firewall. This has the disadvantage that you need a dedicated team to maintain the infrastructure and there is some investment in servers.
If you are allowed to host in the cloud, this is usually the better option. The cost of hosting in the cloud is low, and these operational costs are offset by the services provided: onboarding, infrastructure maintenance, security maintenance, support and CI / CD software maintenance. Hosting your CI / CD software in the cloud often makes it easier and faster for pipelines to interact with your source code repositories when they are also in the cloud. If your developers and testers are geographically dispersed, hosting your repositories in the cloud often gives developers a better experience than hosting on remote servers behind firewalls.
It is also possible to deploy CI / CD on a hybrid of local and cloud servers. Several of the latest CI / CD offerings run in containers on Kubernetes clusters, which are enjoyed to run both locally and in the cloud alike. In a hybrid deployment scenario, you can place each component where it makes most sense, given the physical location of the developers themselves and the network locations of the other servers in the development infrastructure.
CI / CD needs to be integrated into your repositories
As you may have noticed by reading “The endpoint of CI is usually a completed check-in to the main branch of a software repository,” repos for CI and CD are essential. In addition to being the endpoint of the check-in and testing process, software repositories are the preferred place to store your CI and CD scripts and configuration files. Yes, many of the CI / CD platforms can store scripts and other files internally, but it’s usually better to have them in version control outside of the tool.
Almost all CI / CD tools can interact with Git. Some also integrate directly with GitHub, GitHub Enterprise, GitLab and / or Bitbucket. Some also support Subversion and / or Mercurial.
Your CI / CD tools must support your programming languages and tools
Each programming language or language group (JVM languages, LLVM compiled languages, .NET languages, etc.) usually has its own build and test tools. To be useful to you, a CI / CD tool must support all of the languages that are part of a given project. Otherwise, you may need to write one or more plug-ins for the tool.
Docker images are becoming increasingly important for distributed, modular, and microservice software deployments. It helps a lot if your CI / CD tool knows how to work with Docker images, including building an image from your source code, binaries and prerequisites, and deploying an image in a specific environment. Again, you may need to write plug-ins or scripts to implement the Docker functionality you need. Likewise, you want your CI / CD tool to support Kubernetes and any other container orchestration systems that you use in your environments.
Do your developers understand CI / CD and the tools you are considering?
The principles of CI and CD may seem obvious, but the details are not. The various CI / CD tools have different levels of support and documentation. There are several books on Jenkins, which is not surprising given that it is the oldest book. For other products, your due diligence in choosing a tool may require you to review documentation, support forums, and paid support options.
General background information on CI can be found in the Addison-Wesley book Continuous Integration by Duvall et al. For general background information on CDs, consider Continuous Delivery by Humble and Farley. Both books won Jolt Awards when they were published.
You can choose different CI / CD tools for different projects
While this guide is about choosing a CI / CD platform, don’t assume that one platform will be optimal for all of your software development projects. Most stores use multiple programming languages and environments, and not every CI / CD platform will support all of them well.
Feel free to choose the CI / CD platforms that are best for each of your projects rather than finding a single compromise platform. The general principles of CI and CD are carried over from one platform to another, even if the scripts you write for them are not always portable. The additional onboarding time for each new platform can cost your development team some time, but it is probably less expensive than having to fully customize a CI / CD tool.
Plan for future CI / CD migration
With this in mind, please do not assume that any particular CI / CD platform will forever meet the needs of your projects. Always secure your bets by, for example, storing scripts in repositories instead of in the CI / CD tool.
If necessary, prefer serverless CI / CD
In general, cloud container deployments are less expensive than cloud server instance deployments, and serverless cloud deployments are less expensive than container deployments. Unfortunately, only a few CI / CD platforms can run serverless at this point in time.
Serverless means that the container running the process of interest is instantiated as needed, generally in response to an event. With CI / CD, the triggering event is generally a code check-in into a specific repository branch; The repository webhook then starts the serverless process. When the process is complete, the resources will be released.
One of the few CI / CD platforms that can run serverless is Serverless CI / CD, part of Serverless Framework Pro, an extended version of the open source Serverless Framework. Serverless CI / CD is optimized for the delivery of serverless apps and currently only runs on AWS. You need to determine if it supports your application well enough to use it.
Where are your current cloud assets located?
In order to optimize a cloud-based CI / CD configuration (or any cloud application), it helps if all your assets are “close” together. “Near” in this context refers partly to the geographical location and partly to the network location.
For example, if your database is in Australia and your application is in North America, there will be a large delay every time the application needs to read or write data. On a smaller scale, the latency between them is minimized when your application and database are in the same Availability Zone (AZ). If they are in different zones of the same region, the latency will be higher, but not as high as in different regions.
If your database is on Google Cloud Platform in Virginia and your application is on Microsoft Azure in Virginia, the latency is correspondingly higher than if both are on the same cloud provider in the same AZ. All of this also applies to your repository (which is essentially a database), your CI / CD software, your actual application, and your developers and testers. It helps when everyone is “close”, although the effects of the delay are not as obvious in this situation as they would be in a real-time interactive game, for example.
If the developers can push code commits into the master repository reliably and without noticeable waiting times, they will usually be satisfied. When users and testers are “close” enough to the application to get response times of less than a second, they will be satisfied too. Reliable connections to the mount points are key for CI / CD software. A small delay can be tolerable as long as it doesn’t time out or lose packets.
Make a proof of concept before you commit
CI / CD will be an important part of your infrastructure once you have it fully implemented. Keep this in mind as you go.
It is important to have a rigorous proof of concept before you begin implementing the CI / CD pipelines. Shake the CI portion before starting the CD phase. Make sure you run your test suites and rollback functions before connecting CI / CD pipelines to production instances, and keep people informed until you are sure the automation is rock solid.
Copyright © 2021 IDG Communications, Inc.