TFS 2017 Build System - Maintain last "N' builds

In my last blog post, I described retention policies in the TFS 2017 build system. I described how different it is from the the retention policies we get in XAML build system. 

One of the limitations I found in the new style retention policy is that I couldn't retain a specific number of builds for each status. We needed to do it for some builds that are triggered very frequently (once every couple of minutes) and check if there is some work to be done. If it found work, it would do it, otherwise it will reschedule a build for itself after a couple of minutes. Another scenario, where you might have a lot of builds is when you it is triggered by a commit of a very busy repository.

So, in order for us to retain only "N" builds for each status, we created a Powershell Module to clean up builds. In the module, we create a command-let that takes as parameter the name of the build, the number of builds to keep, the result filter and tag filter. Our command-let looks as following

***************************************************

.SYNOPSIS

 Cleans up all builds for the given build definition keeping the latest N number of builds where n is passed a parameter

 If a status is provided, it would only keep N builds with the given status

.DESCRIPTION

 PATCH https://{instance}/DefaultCollection/{project}/_apis/build/builds/{buildId}?api-version={version}

 Uses api-version 2.0 to update the build result

***************************************************

function Cleanup-Builds([string] $tfsCollection,

                    [string] $tfsProject,

                    [string] $buildDefinitionName,

                    [int] $numberOfBuildsToKeep = 10,

                    [string] $result="",

                    [string] $tagsFilter = "")

{

    if (${env:system.debug} -eq $true) {

        $VerbosePreference="Continue"

    }

    if ($status -eq ""){

        Write-Verbose "Deleting all but the latest $numberOfBuildsToKeep builds for definition $buildDefinitionName."

    }

    else{

        Write-Verbose "Deleting all but the latest $numberOfBuildsToKeep builds for definition $buildDefinitionName with status $status."

    }

    $buildDefinition = Find-BuildDefinition($buildDefinitionName)

    if ($buildDefinition -eq $null) {

        Write-Error "No build definition found $buildDefinitionName"

        return

    }

    $buildDefinitionId = $buildDefinition.id

    $query = [uri]::EscapeUriString("$tfsCollection$tfsProject/_apis/build/builds?api-version=2.0&definitions=$buildDefinitionId&queryOrder=2&resultFilter=$result&tagFilters=$tagsFilter&`$top=5000")

    $builds = Invoke-RestMethod -Method GET -UseDefaultCredentials -ContentType "application/json" -Uri $query

    $retainedBuild = 0

    $deletedBuildCount = 0

    for ($i = $builds.Count - 1; $i -gt -1; $i--) {

        $build = $builds.value[$i]

        $buildId = $build.id

        $buildNumber = $build.buildNumber

        

        try {

            $query = [uri]::EscapeUriString("$tfsCollection$tfsProject/_apis/build/builds/$buildId/tags?api-version=2.0")

            $tagFound = $false

            # Not delete the latest numberOfBuildsToKeep builds

            if ( ($retainedBuild -lt $numberOfBuildsToKeep)) {

                $retainedBuild = $retainedBuild + 1

            }

            else {

                Write-Verbose "Deleting build $buildNumber"

                $query = [uri]::EscapeUriString("$tfsCollection$tfsProject/_apis/build/builds/$buildId`?api-version=2.0")

                Invoke-RestMethod -Method DELETE -UseDefaultCredentials -ContentType "application/json" -Uri $query

                $deletedBuildCount = $deletedBuildCount + 1

            }

        }

        catch {

            Write-Error "StatusCode:" + $_.Exception.Response.StatusCode.value__ +

                        "`r`nStatusDescription:" + $_.Exception.Response.StatusDescription

        }

    }

        

    Write-Output "Deleted $deletedBuildCount builds for build definition $buildDefinitionName"

}

We create a PowerShell Module file for the above command let. To set up the Powershell module, we modified the PSModulePath environment variable as first step of our build to include the module path. Then to set-it all up we added a PowerShell task group calling the Cleanup-Builds command in an inline script as shown below

Our build definition looks like below

Retention Policies for TFS 2017 Build System

TFS build system has had a major overhaul since TFS 2015. For people working on team builds since TFS 2010, there is some major learning curve. One of the things the people often find confusing is the retention policy in the new build system. In earlier versions of TFS, you could specify how many builds you want to retain for each status as shown in the screenshot below

Retention Policies for Xaml Builds

The retention policy is quite obvious and you have a deterministic number of builds retained at each status. It's not quite the case in the new build system. A sample retention policy in the new system looks like following

Retention Policies for TFS Builds

So what does it mean? 

Well, to say it simply, it means exactly what it says on the tin!! In the example above, the build would keep all builds from the last 4 days and keep a minimum of 20 builds. That is if there are less than 20 builds present for the given build definition, it would keep older builds until there are a minimum of 20 builds. Lets ignore the options with the lock sign, we will come back to it later. Note there is are no maximum count. It means that you can't control how many builds you keep for your build definition. This is a major shift from earlier retention policy where the number of builds kept for a build definition was deterministic. 

When are builds actually cleaned up?

If you are using an on-premise version of TFS (I am using TFS 2017 Update 2), the builds are actually not cleaned until 3:00 AM every day. For VSTS, it happens several times in a day but the time is not deterministic. The actually explains why there is only a "Minimum to Keep" option in the retention policy.

If you have a build definitions that is triggered very frequently, you will need to find a solution of actually deleting the build definitions. I will explain it in the next post.

What about Keep For 356 days, 100 good builds?

This is the option you see below your policy in the screenshot shown above. This is in fact a TFS Project Collection wide policy and enforces the maximum retention policy. So, in the example above, you can't set "Days to Keep" to more than 365 and "Minimum to Keep" to more than 100. In fact, if you have appropriate permissions, you can change it for the entire Team Project Collection.

TFS Project Collection Retention Policy

Multiple Policies

If you want, you can add retention multiple policies for your build definition. It is very useful, if you have build definition that builds different code branches (release branches for instance). You can use the retention policies to keep different number of builds from each branch. 

Multiple Policies

If you have multiple retention policies for the same branch, the retention would be the most lenient of all the retention, so whatever retains the most builds.

In the next blog post, I will show how we are keeping a lid on the number of builds for builds which are build very frequently, every couple of minutes in our case.