More Releases, Fewer Problems: The Case for Continuous Delivery in Government Software Development

By Alex Pedenko, Chief Technology Officer

As the speed of technology innovation continues to accelerate, the public sector is making progress in keeping pace.

Traditionally the public sector has moved slower than the private sector for a variety of reasons — one being that government has historically used waterfall methodology in software development.

The waterfall approach to technology deployment was more common 20–30 years ago than you see today. It originated in the physical engineering world, where the need to use raw materials meant that iterations of planning and design were substantially cheaper than actual building — hence “measure twice, cut once.” In software development, this approach caused teams to sit and build stuff for a long, long time, and then test for a long, long time. Then they would shut everything down, cross their fingers, deploy the new technology, and hope it wouldn’t break (spoiler: it would).

Conversely, agile methodology acknowledges that the process of building software is not distinct from planning, and has become the best practice in developing software across industries. It’s the approach CityBase uses in building public sector technology. In agile software development, we build, test, and launch technology often. Read more on the difference between waterfall and agile methodologies here.

Launching Technology More Frequently, with Fewer Risks

When you launch technology only once a quarter or twice a year (or even more infrequently) using a waterfall method, the rate of change doesn’t necessarily decrease, you’re just restricting the windows for when those changes are released. What you’re doing is increasing the amount of things you’re rolling out at once. This makes any one release much more difficult to troubleshoot. It makes your launch much more likely to break something, and makes any issue you may introduce much more substantial and significant.

In contrast, the notion of continuous delivery and continuous deployment is that you put a process in place that makes your deployments as small and as frequent as possible. While the two are different, they boil down to the aim of moving quickly while maintaining stability and creating the best possible solutions.

Continuous deployment means that as you finish the cycle of designing, building, and testing a piece of government technology software (or, any software), you are launching immediately. In its extreme, the way this works is that a developer says they are done with a piece of code. At that point, you run automated tests, and if the code passes, the code is automatically deployed into production.

Continuous delivery, on the other hand, means you complete the cycle just short of deploying into production. You say you’re done with the code, run automated testing, and then it stops and waits for someone to say, “ok, you can deploy.” The idea is still to deploy as frequently as possible.

Why Your Code Won’t Break My Code

With both continuous delivery and continuous deployment, any single change is unlikely to cause a major error in the system. You also significantly diminish your integration burden.

Typically, we have half a dozen developer teams working in parallel, often working on different things within the same system. If they all have to deploy at the same time, but they’re working separately, at some point they need to come together and make sure they aren’t stepping on each others’ toes. The longer teams work in isolation, the more likely they are to have done so, and they will have to spend extra time negotiating among themselves on how to bring these things together.

If the changes you deploy to the code are small and frequent, it’s much more likely that you have teams working on the most up to date code, with any changes already incorporated into what they’re working on.

So, what do you need to facilitate continuous delivery or continuous deployment? The answer is one part tech, one part business.

Approaches to Rapid Buy-In

On the business side, you need a way to have fast sign-off. You need a way for your client to move as quickly as your deployment process. Ultimately it doesn’t matter so much when something is ready, what matters is when that thing actually gets into production. If you ship something and then it sits for three months waiting for sign-off, then you still run into many of the same problems outlined above.

In continuous delivery, you’ll remember that your code does not deploy into production right away. Usually in such cases, there will be another layer of vetting, whether that’s on the client side, by testing with constituents, or sometimes we use an approach called “canary releases” to launch just a small piece of the new code. This approach can help empower your client to sign off on an important piece of new technology by diminishing the risk associated with doing so.

With canary releases, you may have 1% of web traffic hitting your new code, with 99% of your traffic continuing to hit your existing code. This enables you to test in a live environment with much lower risk than if you switched everything over to the new code. With this approach, you will usually ramp up the percentage that hits your new code until you are ready to switch over to 100%.

For instance, if you have a new payment application, you may have 1% of traffic hitting your new application in the first week, with the remaining traffic still paying through the established application. Once you’re confident your code is working well at some designated period of time, say, a week, you will increase the traffic to 5%. The following week, you may increase that to 25% of web traffic, then 50%, and so on.

Testing, Testing

On the technology side, you need a very robust pipeline for testing, deploying, and managing deployments.

There is only so much testing you can do, especially as systems get more complicated. The more complicated a system gets, the more moving pieces it has, and the less likely you are to cover every single use case. You still build as many automated tests as you can. But the best way to test is in production (as heretical as that sounds).

So after you’ve run all of your tests traditionally, at some point, you switch from that mode of testing to using a sliver of production traffic.

This is where Kubernetes come in. Kubernetes is the operating system of the cloud. It is an infinitely scalable infrastructure-as-code solution that operates on a dynamically allocated cluster of underlying hardware. Real-time performance monitoring adjusts the number of nodes in the cluster as needed, adapting to the load as it changes throughout the day. (For additional reading, we recommend this book on Kubernetes.)

One thing that Kubernetes lets you do is to shadow traffic. As requests come into your system, kubernetes will make a copy of that request and send it somewhere else. So you can take on production traffic, without impacting production. In doing so, you’re testing all of the different use cases that are actually hitting your system, beyond what you might have thought of during traditional testing.

This approach requires having the infrastructure and processes in place to monitor what’s happening in real-time, and respond to it just as quickly.

Choosing Continuous Delivery or Continuous Deployment

Taken together, these things:

• Increase the speed that you can move a change through your process
• Decrease the likelihood of a major outage or bug
• Improve the security of your system, since you’re more likely to be up to date on all your system packages
• Overall increase the agility of your offering  

The benefits to continuous delivery and continuous deployment are clear. But how do you choose which one to use in which instances?

Fundamentally, there are two types of changes you can make in introducing new code: those that are client- or public-facing and those that are internal. Client-facing changes oftentimes have unavoidable approval processes, and because of that you would need to implement continuous delivery. The best thing you can do is shorten the approval process as much as possible.

Another major factor to consider is the application you are building. You need to tailor your deployment strategy to the specific system that you are operating in. Some systems have a higher degree of comfort with error than others.

For instance, in an application that lets you search for information online, if a person encounters an error one out of one thousand times, it’s not the end of the world. However, if that person is using your application to pay a parking ticket that’s about to double at midnight, one error out of one thousand is unacceptable.

Different systems have different burdens of proof for their stability. Continuous deployment does trade speed for stability to a certain extent. That’s why you apply it to some systems but not others.

An engineer should want their system in a global, testable environment as soon as possible. Whether or not that environment is production or staging — meaning in production or just before production — can and should be a business decision, and not an engineering decision.

When Govtech Moves Fast, It’s Moving in the Right Direction

At the end of the day, both continuous delivery and continuous deployment are undeniably faster and substantially more stable than the traditional quarterly or semi-annual release model of technology development. The benefit to government is that it’s more efficient (see: less expensive) to operate because the amount of human testing involved is much lower, the amount of breakage is much lower, and the ceremony around releasing a new system goes away. It becomes just another day at the office.

By normalizing continuous innovation, local government can better deliver and manage services, meet and exceed the private-sector expectations of constituents, and have technology in place that’s as reliable as, say, taxes, but is infinitely more pleasant.