How many deployment stages do we need for a web application pipeline?

Many web applications use staging and automated deployments as part of the software deployment process. Deployment stages are distinct levels in the software release cycle where individual services are deployed, such that the underlying infrastructure, user-access (mostly internal vs external), and at times data-access configurations can be separately managed. These stages are often used to validate the quality of the software being released, before end-users see an impact.

In my 10+ years of experience as a software developer, I have seen several instances where the number of deployment stages varies from one (just the production environment) to several. In this post, I try to share my views on how many deployment stages we really need for most use cases.

Traditional stages

A typical flow of changes through the traditional stages

1. Alpha — There are conventionally isolated stages where the service is not interconnected with the service mesh (other dependent services), rather sometimes offers broken or mocked service-to-service interactions.
2. Beta/Integ (integration) — This is usually the lowest stage where the service mesh is interconnected. For every interacting service, there would be a pointer in the stage to the corresponding dependent service stage and is often used to perform automated testing of service interface.
3. Gamma/Pre-prod — The Gamma/Pre-Prod stage is traditionally where the quality assurance (QA) teams perform manual testing to ensure the quality of the release before the end-customers see the changes.
4. Prod One-box (Canary)— Prod One-box is where changes are tested on a single node of the production environment before being rolled out to all nodes, which means only a subset of the users would see the impact. In the event of something going wrong, the scale of the impact can be minimised and the change can be rolled back before all customers see the impact.
5. Prod stage(s) — The Prod stage usually refers to a set of environments that serve live traffic. In certain deployment strategies, there would be multiple environments per region such that the changes deployment of the changes to different markets could be controlled.

Wait, those are too many stages to manage. Do we really need those many? In my experience, not all services would require these many stages, but some surely do. Here are the aspects that could control the number of stages required and ensure the desired quality is maintained.

Dev-testing scenarios

In my opinion, most of the software services should be able to support local development environments (such that deployment is accessible to a particular developer) and wiring those to interact with certain stages of dependent services. However, there are specific scenarios where a separate stage for dev testing may be necessary.

Feature control capability

Feature controls are mechanisms that allow turning on/off certain features through an external configuration. Some companies like Amazon have advanced feature control and A/B testing frameworks, which let the teams control the on/off state and, at times, percentage rollouts for customers based on individual customer IDs, marketplaces and certain custom parameters.

While feature control could let the changes be enabled for only a few “tester” accounts such that QA teams can test the changes in production, not all changes could be isolated through feature toggles. For example, database schema changes, certain configurations etc. cannot be controlled through feature toggles. Also, certain organisations and certain use-cases require manual quality assurance in a production-like environment before the changes are propagated to production. Such scenarios might require an explicit Gamma/Pre-prod environment.

Integration tests

Feature controls ensure that the changes do not cause any impact as long as they are deactivated. However, for changes that cannot be feature-controlled, it is essential to ensure that the customer experience is not disrupted. This is where automated integration tests come to the rescue. These are the tests that are written on the service interfaces to make sure the interfaces remain unchanged or is as expected.

Ideally the integration tests are to be written and shipped along with the feature changes, if applicable. And like unit tests that usually run at the class-level or package level, integration tests also must cover happy-cases and negative cases. Also it is recommended to add any issues observed in production to be added to the integration test-suite such that it would cover the regression tests while the changes are being shipped.

Geographic regulations

Sometimes, regional and international regulations might mandate data-residency requirements (for example, within the same region) and certain other such rules that would need separate environments. At times, such requirements could be catered through techniques like data sharding, but that could be more complicated and sometimes not even feasible for some scenarios.

The optimum number of stages

In my opinion, most services would need only two or three stages. These are:

1. Integ — For executing automated integration tests and serving dependent services local deployments.
2. Pre-prod (optional) — For any manual validations required. This includes scenarios where high impact changes that would need to be manually tested before deployment, changes that could not be covered through integration tests (for example, certain visual changes). Also, this stage would also help to triage and validate the fix for certain production issues.
3. Prod — Serving live traffic. However, certain geographic regulations may require more than one prod stages. Also certain teams might prefer multi-region deployments through different stages for gating controls. The One-box use case also could be controlled through the feature control framework if it lets sequential dial-up of the changes, and would be more controllable compared to OneBox.

The change flow through a pipeline with the recommended stages

Conclusion

The number of deployment stages that is needed would vary largely based on certain aspects like testing scenarios, feature controllability, quality of integration tests, need for manual testing, geographical regulations etc. In most scenarios, I feel that the three stages we discussed are good enough.

I hope you find this article interesting and helpful.Let me know your thoughts on this as comments below. Feedback like those would make me happy and help me get better.