Build Tools in a Microservices Architecture

Using microservice architectures to build and deploy their systems can be helpful for companies looking towards continuous integration and continuous delivery. Care should be taken to ask the right questions and spend the appropriate time planning as companies consider their automated build/deploy process.

For those unfamiliar, a microservices architecture is a development technique that structures an application as a collection of services. In turn, this can make the application easier to understand, develop, maintain, and test through focused functionality.

I recently helped to refine and stabilize the build and deployment processes for a client in the throes of converting their existing systems into a microservices architecture. They use a tool called Concourse CI. My previous build tool experience has primarily come from using Jenkins, perhaps one of the more popular build tools around.

Using the above-mentioned client as an example, let’s discuss the ways they use microservice architectures and what other companies can learn from our experience.

Build Tools − A Traditional View in Brief

I was intrigued by the project — from the challenge of learning a new tool, to the interesting way the client builds/deploys their systems, all the way to the fact that they already had several microservices that were being deployed. Here we’ll discuss some fundamental concepts that Concourse uses that allows your build to scale up along with your architecture.

In many projects I’ve been involved with, the building and deploying of the application was largely an afterthought. If time and budget allowed, perhaps an automated build could be cobbled together using a tool such as Jenkins.

Jenkins creates a low barrier of entry to approach in this way, offering a variety of plugins to support certain tasks and an easy way to script items that require customization. This tool, and perhaps the methodology in general, makes putting something into place that analyzes code, runs automated tests, assembles build artifacts, and deploys your application relatively easy.

When Your Architecture Outgrows Your Build Tool

Indeed, this cobbled-together methodology makes it easy to create a nice build process for one or even a small number of applications.

However, when you try to scale this model for a larger number of applications you might discover this approach may become increasingly cumbersome. Some items to consider include:

• What if you use this approach to deploy fifty different applications? Or one hundred?
• What if your organization decides to switch from Subversion to Git for source control? 
• Or change the server of the database that your integration tests typically use? 
• Or decide that the technology that it relied on is now legacy and all new applications should use a significantly different stack? 

In these and similar such cases, the cobbled-together methodology loses its luster as you have to go through each job and make appropriate changes. Mistakes are bound to happen when you go job-to-job making the tweaks necessary for your builds to keep up with the evolution of your infrastructure. Perhaps these mistakes are minor, but in certain cases, they could ultimately result in lost sales, a bad reputation, or upset customers if your application doesn’t deploy or deploy correctly. Indeed, we expect our build scripts to “just work,” and when they don’t the impact is largely unpredictable.

A Different Way of Thinking About Your Build

This concern has heightened my appreciation for Concourse CI and its approach to architecting builds. The beauty of Concourse CI lies in separating desired functionality into resources and jobs to create build pipelines. 

A resource is typically a “thing” that is acted upon, triggers an action, or both. Some examples of a resource include a source control repository, Docker repository, an Amazon S3 bucket, Slack messaging, and many others. At its heart, a resource is software that provides the functionality required to interact with it and typically contains separate functionality to return its version to distinguish it from other resources of the same type. 

Next, a job contains the actions required to complete a particular build step, acting on and from resources to complete the task at hand. 

A build pipeline brings these two concepts together, stringing along one or more jobs with one or more resources to complete work.

Taking a step back for a second, there are some interesting ramifications to this type of architecture. 

First, the separation of concerns encourages encapsulation and reuse of the components directly involved with the build. Instead of having a bunch of build scripts that embed environment-specific configuration and commands within them; you can simply tailor resources to your application and use a consistent interface to interact with it. This becomes quite appealing when looking to scale the number of services up to build/deploy a microservices architecture with many deployments. Perhaps a downside to this benefit is added complexity. 

Make no mistake about it, Concourse is quite a bit more complex to pick up than Jenkins. Part of this complexity, though, relates to thinking through what your pipelines should look like for the services that you deploy. 

To best utilize, consistency should be the name of the game. These aren’t just one-off scripts that are built in an ad hoc manner, though Concourse could certainly support this. Alternatively, though, Jenkins could also possibly emulate a similar reusable pipeline if the correct plugin was utilized.

Continuous Integration/Deployment in a Microservice Architecture

Perhaps the most important takeaway here is to carefully consider your automated build/deploy process, especially if you intend to create a microservices architecture deployed in a manner consistent with CI/CD. Ask the important questions up-front and make sure your build and deployment plan would best be supported by a microservice architecture or another methodology. With time spent discovering and planning upfront, you can be better set up for success when implementing microservices. 

Treating the automated build process as an afterthought, while it may pay immediate dividends in the short term, is a recipe for tech debt and refactoring work in the long run. Fortunately, there are build tool alternatives that support creating robust pipelines that enable seamless integration with new services as your architecture expands.