The Deployment Pipeline

Roughly speaking, a deployment pipeline is assembled from four larger parts or areas of practice: building,testing,deploying andreleasing (Humble and Farley, 2010). Automated scripts for building and testing with proper con- figuration management along with automated deployment to various envi- ronments are the hallmarks of a functioning deployment pipeline. Figure 2.2 illustrates some of the practices and activities linked to the various stages of the deployment pipeline, commonly associated with continuous delivery and continuous deployment (Humble et al., 2006; Humble and Farley, 2010;

Adams and McIntosh, 2016; Parnin et al., 2017).

Activities in the build stage of a deployment pipeline are well aligned with those found in continuous integration. Developers use build systems and check in code to version control systems, aware of the possibility to use version control branches for efficiently managing changes from multiple developers working on the same code (Adams and McIntosh, 2016). With the prospect of delivering the changes rapidly to further test, staging and production environments in mind, emphasis is put on efficient builds (Hum- ble et al., 2006). The arrangement and building of software modules should be such that all necessary source code is compiled in a single run, preferably without many compiled intermediate binaries that other projects might rely on. Having fewer build and compilation stages of other projects triggered helps to minimize the time for the build stage of a single project. This mode of thinking that puts speed at the forefront is an overarching principle in the design and implementation of a deployment pipeline (Humble and Farley, 2010). Automated and manual processes that are part of the pipeline can be timed to understand which of the processes have the greatest effect on cycle time and thus have room for improvement.

A version control system is one of the primary junctions in a deployment pipeline also for the reason that a version control system can serve other purposes beyond storing software code and related artifacts. For ensuring proper configuration management, version control systems can be used to store infrastructure blueprints for servers and complete copies of virtual- ized servers in the form of virtual machine images (Humble and Farley, 2010). Stored infrastructure configuration written in a specific infrastruc- ture programming language and virtual machines help to establish similar environments for development, testing and production, which facilitates the rapid flow of software changes from one environment to another (Humble and Farley, 2010; Adams and McIntosh, 2016).

Testing as part of the deployment pipeline consists of performing auto- mated and manual tests split into test suites that test the validity of the built software version on different tiers with emphasis on automated tests (Humble et al., 2006). In the lower-level tiers, testing includes performing unit-level and other tests that focus on isolated code components and units.

Unit tests are followed by higher-order functional and integration tests that can take advantage of production-like testing environments. Passing unit and integration tests indicate that the software version under test might very well be a potential release candidate and ready for the highest cate- gory of acceptance tests.

Deployment and the capability to deliver packaged software versions to other environments becomes important at this stage at the latest, as ac-

ceptance tests should at the very least be tested in an environment that resembles the production environment (Humble and Farley, 2010). Accep- tance tests and the acceptance criteria for the tests can be either functional or non-functional. Functional acceptance tests, which can be automated, are driven by user stories to determine whether the implementation fulfills the needs set by the user regardless of the technicalities in the background.

Working with the customer and users in short cycles, continuous deploy- ment favors rapid validation of what is being implemented at any given time (Olsson et al., 2012). Acceptance tests as such have their place in ensuring that the perspective of the user is not forgotten. Acceptance test criteria can also be non-functional in the sense that properties such as performance and security can be tested in a production-like environment to get a sense of whether the system and the current release candidate respond according to the set performance thresholds without serious security vulnerabilities (Humble et al., 2006).

Preferably, acceptance testing should be done in an environment that resembles the production environment, which calls for proper configura- tion management and provisioning of test infrastructure. If the creation of testing environments for acceptance testing is not properly automated, developer teams might not get the immediate feedback necessary for con- tinuous delivery and deployment. Setting up testing environments without automation takes considerable time not only from the developers but from other people as well and as a result acceptance testing tasks do not move in the development workflow towards deployment to production, piling up when testing resources eventually become available (Chen, 2017). Replicat- ing production environments for testing is not an easy task when configu- ration between environments can still vary due to various reasons, making it difficult to ascertain the release readiness of the built version (Parnin et al., 2017). A high rate of incoming changes and inconsistency in envi- ronment configuration means testing is partly done in production, which is an undesired situation and increases the possibility of observed failures.

Fine-tuned deployment practices and automated deployment are pre- requisites for efficiently testing a particular software version and ultimately shipping the tested version to a production environment with minimal over- head. Without proper deployment practices, the process of deploying a new version for testing in another environment can take days instead of the mere hour or so it takes with automated deployment (Humble et al., 2006). Continuous integration servers, such as Jenkins, enhanced with ca- pabilities for workflow orchestration and deployment can provide assistance in automating the end of the deployment pipeline suitable for continuous

delivery and continuous deployment (Armenise, 2015). Workflows in Jenk- ins can be composed of any number of arbitrary phases that build software packages, execute automated functional and non-functional tests in various environments, process the resources in any manner and deploy the packages to various environments. Phases that require human intervention can be configured to halt, waiting for additional permission to continue with the subsequent phases upon approval bypromoting the build to the next phase.

Workflow tools that are able to illustrate and visualize parts of the deploy- ment pipeline (Armenise, 2015), help raise developer awareness of parts of the pipeline that still require attention in terms of continuous delivery and continuous deployment (Chen, 2017). Automating deployment gives de- velopers the chance to deploy changes to production at will that can also strain and put pressure on developers as they are responsible for their own changes, needing to keep alert and monitor production systems for errors when the changed code is invoked for the first time by users (Parnin et al., 2017).

Deployment and specifically deployment to production environments does not necessarily mean that a new software version is immediately re- leased to all users. Releasing software changes right away was incorporated in the original line of thought of continuous deployment (Fitz, 2009). Fewer changes at once was seen to make it easier to pinpoint and correct any possible failures in production, owing to the fact that there would be less ground to cover and where to look for failures. Chiefly, the idea still rings true and remains valid but in the continuous deployment parlance of late there has been a tendency to clearly distinguish between deployment and release. Deployment refers to the capability of moving necessary files re- lated to a particular software version from one environment to another with appropriate environment configurations while releasing means making the version available to the end users (Adams and McIntosh, 2016). Continuous delivery and continuous deployment deal with both concepts – deployment and release – but the terms are not entirely interchangeable.

Continuous delivery can be seen as a precursor to continuous deploy- ment. What separates the two is their relationship to deployment to pro- duction. There is a shared understanding that in continuous delivery, the deployment pipeline has been set up so that development teams have the capability to push and deploy recent changes to production but deployment and release have not been fully automated unlike in continuous deployment (Ståhl et al., 2017; Fitzgerald and Stol, 2017). Continuous deployment goes a step further in ensuring that changes made in development are effectively deployed to production and released to end users with a rapid schedule.

Conceptually, this difference in the operation of the last part of the de- ployment pipeline is the major difference between continuous delivery and continuous deployment although there has been some ambiguity in the use of the terms and their meaning has been mixed over the years (Fitzgerald and Stol, 2014, 2017).

An automated deployment pipeline can make the task of releasing new software versions to the end users easier, especially so if the pipeline extends all the way to the production environment as it should in continuous de- ployment. Releases and releasing changes, however, can take many forms.

Strictly speaking, if a deployed software change or feature that is part of the release cannot be used by the end users or is not otherwise available, the change has not been released yet. Forms and modes of release exist where this is exactly the case as new code and other changes can be silently de- ployed to production but associated features remain disabled for end users with the help of feature flags or then the whole feature is built incremen- tally step-by-step with dark launches (Parnin et al., 2017). Other options include releasing changes only to some users but not to all of them with canary releasing (Humble and Farley, 2010; Adams and McIntosh, 2016;

Parnin et al., 2017).

Beyond disabling individual features and releasing versions only to a small subset of users lies the most experimental release and deployment methodologies where end users are exposed todifferent versions based on a certain logic in dividing users into various groups. Experimental methods like A/B testing are part of this continuum in which implicit user feed- back and telemetry from the production environment guides development efforts (Adams and McIntosh, 2016). In continuous software engineering, continuous innovation and continuous experimentation are linked to work- ing methods that combine frequent releases of continuous deployment with user centered experimentation (Fitzgerald and Stol, 2017). Such methods form the basis of experiment systems that are at the top of the conceptual user centered agile software development staircase (Olsson et al., 2012) – methods that require the capability to deploy and release often to work in a sensible manner. The next section covers some of the release models that can be used in conjunction with continuous deployment.