Queue TFS/VSTS builds VIA PowerShell

My team has used TFS in our private datacenter for over six years now as our primary source control, build and test management system.  We utilize TFS build definitions, as described in my last blog post, to perform all our application building, packaging and testing.  When we started down the path toward Continuous Delivery three years ago we did so via small iterative experiments in an attempt to deliver value quickly and with little risk.  Part of that approach meant trying to reuse as much work/infrastructure as possible instead of rebuilding unnecessarily.  Since we were already using TFS build definitions to perform our building, packaging and testing we saught to automate those builds via a release orchestration tool (Thoughtworks GO!).

Orchestration Overview

The process consists of three high level steps:

1. Get the deploy.json file from TFS source control
2. Queue the builds list in the deploy.json file and store their resulting package as artifacts in the pipeline
3. Deploy the packages to the target servers listed in the deploy.json file

The first two steps are encompassed in the QueueBuildsFromConfig.ps1 PowerShell script which is the focus of this blog post.

We utilized the C# TFS API in the first version of this script because that was the only available API at the time.  We used this script for three years with great success but the C# objects inside PowerShell were hard to read and overly verbose. 

Time to Improve

Last year we upgraded to TFS 2015 which provides the new vNext build/release system which is EXCELLENT.  Although the older C# TFS API did still work with 2015, it could only interact with the old XAML-based build definitions.  Microsoft also provided a new set of APIs that could interact with the new build/release system as well but we really wanted to take this opportunity to simplify the queue builds script.

If you’re interested in learning more about the C# API, Buck Hodges had a good blog post about all the TFS/VSTS APIs for TFS 2015 last year.

In addition to the new C# API, TFS 2015 also introduced a new REST web service for integration with TFS.  The web service can reference both the new and old build defitions which allows us to use the newer build system.  REST calls are much easier to work with in PowerShell in my opinion.

Step 1 – Get the Deployment Configuration

We must connect to TFS with the appropriate credentials so we do have to create one C# object which is the PSCredential class and provide it with the TFS username and password.  Then we make our REST call using the Invoke-RestMethod commandlet passing the credentials and service method URI. 

[void][Reflection.Assembly]::LoadWithPartialName('System.Management.Automation')

$p = ConvertTo-SecureString $password -AsPlainText -Force
$creds = New-Object -TypeName "System.Management.Automation.PSCredential" -ArgumentList $username, $p

$path = "$/DeployConfigurations/$($configName).json"
$deployConfigJson = Invoke-RestMethod -Method Get -Credential $creds -Uri "$TFSUri/_apis/tfvc/items/$($path)?api-version=2.0"

In this case the call uses the GET REST method so the path to the config file is part of the service URL:

GET https://{instance}/{teamcollection}/_apis/tfvc/items/{path}?api-version={version}

The Microsoft documentation site provides further details:

https://www.visualstudio.com/en-us/docs/integrate/api/tfvc/items#get-a-file

Configuration File Structure

The deployment configuration file basically lists each of the TFS builds (one for each deployable component) that produce a deployment package.  Additionally, it can list the target servers that the component should be deployed to by environment for use at the later deployment stages of the pipeline:

{
  "builds": [
    {
      "name": "MyWebsite (Pkg)",
      "teamProject": "MyTeamProject",
      "environments": [
        {
          "name": "DEV",
          "servers": [ "DEVServer1" ]
        },
        {
          "name": "QA",
          "servers": [ "QAServer1", "QAServer2" ]
        },
        {
          "name": "MOCK",
          "servers": [ "MOCKServer1" ]
        },
        {
          "name": "Release",
          "servers": [ "Server1", "Server2" ]
        }
      ]
    },
    {
      "name": "MyApp.Db.Sql.Deploy (Pkg)",
      "teamProject": "MyTeamProject",
      "environments": [
        {
          "name": "DEV"
        },
        {
          "name": "QA"
        },
        {
          "name": "MOCK"
        },
        {
          "name": "Release"
        }
      ]
    }
  ]
}

Step 2 – Queue the Builds

For each build we will retrieve the TFS build definition:

	$project = $buildConfig.teamProject
	$buildName = $buildConfig.name

	$buildDef = Invoke-RestMethod -Method Get -Credential $creds -Uri "$TFSUri/$project/_apis/build/definitions?api-version=2.0&name=$buildName"

This operation again is a GET method where we provide the definition name:

GET https://{instance}/{teamcollection}/_apis/build/definitions?api-version=2.0&name={buildName}

https://www.visualstudio.com/en-us/docs/integrate/api/build/definitions#get-a-build-definition

Next we will queue the build using a POST call.  We place the definition id in body of the request. 

	$json = "definition: { id:$buildDefId }"
	$json = "{ $json }" 
	$build = Invoke-RestMethod -Method Post -Credential $creds -ContentType application/json -Uri "$TFSUri/$project/_apis/build/builds?api-version=2.0" -Body $json

POST https://{instance}/{teamcollection}/_apis/build/builds?api-version={version}

https://www.visualstudio.com/en-us/docs/integrate/api/build/builds#queue-a-build

Note – you can add other parameters to the queue request by adding other parameters in the body.  The link above provides the full list of optional parameters.

Once the build is queued we wait for it to complete by polling TFS every 10 seconds to get the status of the build until its complete.

	$status = ""
	Write-Host "Building $buildName ($buildId) in $project ..."
	while ($status -ne "completed")
	{
		Start-Sleep -Seconds 10
			
		$build = Invoke-RestMethod -Method Get -Credential $creds -ContentType application/json -Uri "$($TFSUri)/$($project)/_apis/build/builds/$($buildId)?api-version=2.0" 
		
		if ($status -ne $build.status)
		{
			$status = $build.status
			Write-Host $status
		}
		Write-Host '.'
	}

Here we use another GET method to retrieve the queued build to see its status:

GET https://{instance}/{teamcollection}/_apis/build/builds/{buildId}?api-version={version}

https://www.visualstudio.com/en-us/docs/integrate/api/build/builds#get-a-build

Save the Artifacts

The last step for each build is to retrieve and store the package.

		$artifacts = Invoke-RestMethod -Method Get -Credential $creds -ContentType application/json -Uri "$TFSUri/$project/_apis/build/builds/$buildId/artifacts?api-version=2.0"
		foreach($artifact in $artifacts.value)
		{
			$filePath = $artifact.resource.downloadUrl.Replace("file:", "").Replace("/", "\").Replace("%20", " ");			
			$path  = $filePath + "\*Package\*"
			if (Test-Path $path)
			{        
			    $packagePath = "MSDeployPackages\$project-$buildName\";	
			    if ((Test-Path $packagePath) -eq $false) {
			    	New-Item $packagePath -ItemType directory
			    }
			    Copy-Item -Path $path -Destination $packagePath -Recurse -Force 
		    }
		}

We retrieve the list of artifacts using the following GET operation and save each artifact to our artifacts folder for the pipeline.

GET https://{instance}/{teamcollection}/_apis/build/builds/{buildId}/artifacts?api-version={version}

https://www.visualstudio.com/en-us/docs/integrate/api/build/builds#get-build-artifacts

Summary

TFS is a super flexible tool which offers many features.  In this post, we reviewed how you can utilitize the TFS build system from an outside orchestration tool using their REST service endpoint via PowerShell.  We use Thoughtworks GO! to orchestrate the release at work but this would also work with Jenkins and any other tools that can run PowerShell scripts.

If you don’t already have an orchestration tool, I would suggest using the new release automation system in TFS 2015/VSTS.  I’m using this for several personal project which is working really well and is even easier with tighter integration than the above process.

Please let me know if this post was valuable by commenting below, posting to Twitter or other social places.

Happy Deployments!