Accessing Azure DevOps APIs with large volumes of data

by Jesse Houwing

Most REST APIs exposed by Azure DevOps are limited in the amount of data they will return. You can choose to override the default number by passing a $top=### query string parameter. Azure DevOps will try to honor this request, but I've seen it refuse and return a lower number anyway.

Note: When using PowerShell, be sure to escape the $ in the query string, otherwise PowerShell will try to inject the value of whatever variable called top holds. You can escape the $ with a preceding backtick:
$urlBase="...?api-version=6.0&`$top=5"
                              ^^^^^

Using $top= has been tricky at times. It may or may not give you the requested number of items, it seems to depend on how busy the backend is. But it does promise you this: All the REST API's will return a x-ms-continuation-token header next to the payload and you can use this to fetch the next batch of items:

x-ms-continuation-token is returned by Azure DevOps in case these are more results available for the request than just those returned.

You can fetch the next batch of projects by requesting the exact same REST query and adding the &continuationToken=${x-ms-continuation-token} query parameter to the call. Repeat this until the server stops sending the x-ms-continuation-token header, which signifies you've gotten all the values you were after.

You can grab the headers by passing in a second variable:

# Change top=5 below to 100 or so depending on your use case.
$urlBase="https://dev.azure.com/jessehouwing/_apis/projects?api-version=6.0&`$top=5"

$url = $urlBase
$results = @();

do 
{
    write-host "Calling API"
    $response = Invoke-RestMethod -Uri $url -Headers @{Authorization = "Basic $token"} -Method Get -ContentType application/json -ResponseHeadersVariable headers
    $results += $response.value

    if ($headers["x-ms-continuationtoken"])
    {
        $continuation = $headers["x-ms-continuationtoken"]
        write-host "Token: $continuation"
        $url = $urlBase + "&continuationToken=" + $continuation
    }
} while ($headers["x-ms-continuationtoken"])

$results

When you run this, it will show:

Calling API
Token: 5
Calling API
Token: 10
Calling API

id             : 06cb494c-b535-4f16-9323-bd0f63c38163
name           : CMMI
url            : https://dev.azure.com/jessehouwing/_apis/projects/06cb494c-b535-4f16-9323-bd0f63c38163
state          : wellFormed
revision       : 414360096
visibility     : private
lastUpdateTime : 08/07/2019 20:19:09

id             : 88a7ca79-b5c8-41d2-99e7-a5578a1df424
name           : BattleJSip
description    : Battleship case in JS/TS
url            : https://dev.azure.com/jessehouwing/_apis/projects/88a7ca79-b5c8-41d2-99e7-a5578a1df424
state          : wellFormed
revision       : 414360059
visibility     : private
lastUpdateTime : 15/04/2018 13:43:44

... for 12 projects

As you can see, 3 calls were made, and 2 continuation tokens were returned.

In case of the /projects endpoint the continuation token takes on a predictable value of the number of projects to skip. Other API's may return a GUID or a base64 encoded string or the ID of the last returned item. Make no assumptions about the contents of the token, always copy it from the header and put it into the next request verbatim.

Here's an example of what's passed over the wire:

GET https://dev.azure.com/p/_apis/projects?api-version=6.0&$top=5

x-ms-continuation-token:5

{"count":5,"value":[...]}

And then the next call will fetch projects 6 to 10 and sends a continuation-token back with the value 10 for the call after that:

GET https://dev.azure.com/p/_apis/projects?api-version=6.0&$top=5&continuationToken=5

x-ms-continuation-token:10

{"count":5,"value":[...]}

Until the x-ms-continuation-token header is no longer returned

GET https://dev.azure.com/p/_apis/projects?api-version=6.0&$top=5&continuationToken=10

{"count":2,"value":[...]}

Originally answered on StackOverflow.

Leave a comment.