Metrics for Characterizing Deployment and Release

A release cycle consists of all the activities that take place between two releases and release cycle time is essentially the period of time between the releases measured in minutes, hours, days, weeks, months or years. A release cycle time of a month means a new software version of a particular system is released every month. In reality, to better understand deployment and release capability, the release cycle needs to be broken into smaller pieces.

Excluding the actual development of a feature, deployment capability of the deployment pipeline can for instance be characterized by how much time is spent on individual parts of the deployment pipeline (Bass, 2016).

Deployment time can thus be seen to consist of nine factors that each have an effect on how quickly a change can be released (Bass, 2016). These factors include the time it takes to build a version, run integration tests, deploy changes to a staging environment, run acceptance and other tests in staging environments, and the time it takes to reach a decision on whether the version in the staging environment is ready for release or not. When canary releasing is used, additional time is spent on deploying release canaries to selected users and figuring out how the canaries work out. The final factors are then deploying the version to the production environment, effectively releasing it to all users, and keeping an eye out for unexpected failures in the production system.

Time in the deployment pipeline is partly spent on activities preparing the release, which do not have a direct impact on the end users as they are internal to the company or organization. Only when a change is released, can users interact with the version with the changes. Internal deployment capability could in addition be measured by measuring the time it takes to have the next version ready at the staging environment, just waiting for the go-ahead from people responsible for the acceptance test phase.

This leads to several metrics useful in characterizing deployment capability and release frequency: cycle time to potentially deployable software (de-

Build

Test

Deploy

•M1Cycle time to potentially deployable software

•M2Is deployment pipeline to potentially deployable software automatic?

Release

•M3Fastest possible cycle time to production

•M4Actual cycle time to production deployment

•M5Is deployment pipeline to production automated?

Figure 4.1: Five metrics for characterizing deployment capability and release cycles in stages of the deployment pipeline (Publication I).

noted M1 in Figure 4.1) and actual cycle time to production deployment or release (M4) (Publication I; Publication II). Understanding whether the deployment pipeline is automated up to the staging environment (M2) or all the way up to the production environment gives additional yes or no metrics (M5), signifying the degree of automation (Publication I). Inertia of the deployment pipeline can be reflected on by thinking about the time it takes for a small change like a change to a single line of code to propa- gate through the deployment pipeline to the production environment (M3) (Publication I). The five metrics that can help to understand the internal capability for preparing releases and actual release practices are mapped to the respective deploy and release stages of the deployment pipeline and further illustrated in Figure 4.1.

In practice, there seems to be a great difference in the cycle time to potentially deployable software and the actual cycle time to production deployment (i.e. release cycle time). This means that in many cases the deployment capability or deployment readiness is much higher than the rate at which software is actually released. Company cases illustrated in Figure 4.2 show in months the relation of release cycle times depicted in gray to po- tentially deployable software cycle times depicted in black (Publication II).

Release cycle times are often in the order of several months although the

Figure 4.2: The cycle time to release (gray) is in many cases considerably longer than the cycle time to potentially deployable software (black) (Publication II).

cycle time to potentially deployable software suggest that there would be capability to release at least every month or every few weeks. In some cases, the difference is notable as for instance in cases C5 and C10 where the release cycle time is a year or over but the cycle time to potentially de- ployable sofware is a month or less. It appears that in practice continuous delivery is more likely or at least more common than continuous deployment to production.

There are a number of factors that can at least partly explain why there is such a large discrepancy between release cycle times and poten- tially deployable software cycle times. Factors such as the domain in which a company operates and the platform for which the software is built seem to make a difference. Companies that had the longest release cycles devel- oped software for embedded systems, telecommunications devices, or mobile devices in the mobile games domain (Publication II). When access to the devices is constrained in some manner such as is the case in the telecommu- nications industry or in a factory housed with embedded systems, there is a lesser tendency towards continuous deployment (Publication I). Continuous deployment and frequent release cycles are also less likely in the embedded medical systems domain where devices are required to adhere to quality standards for ensuring the safety of devices and users. Longer release cy- cles in mobile games may be due to games being treated more as a form of art where testing is more subjective and not only focused on functional

correctness but on the overall user experience (Publication II). Reasons for ramping up or scaling down release frequency are further explored in the following section.