Fail your builds when tests are skipped in Azure DevOps Pipelines

When the Visual Studio Test Task in Azure DevOps Pipelines fails to find any tests it logs a warning and happily succeeds. It has been a regular request on the MVP community to do something about that and to ensure that builds fail when no tests have executed.

Since test results are published to the build results, it turned out to be quite easy to create a Server Task which handles this for you. This post will both introduce the task and explain how I built it.

Ensure Tests Task

You can find the Ensure Tests task in the Visual Studio Marketplace, once you have them installed you can add them to your build pipeline. To do so you first have to add an "Agentless job":

Then configure that to depend on all of the other phases in your build pipeline that are used to run tests:

Add the "Ensure Tests" tasks to your Agentless job and you're all set!

How does it work?

The "Ensure tests have executed" task relies on the information that's automatically captured by Azure DevOps Pipelines when tests are executed. Test results and coverage data is automatically uploaded. This data is continuously available, even while the build is running.

The task queries the test result API and uses the same endpoint that's used by the Build Result page. I used the Chrome Developer Tools to look up the endpoint used by the build summary page to display the Total Tests number:

The result json contains quite a few useful metrics, the ones I used was totalTests, but as you can see it would be easy to also check for the number of Test Runs, increases or decreases in the number of tests etc.

{
  "aggregatedResultsAnalysis": {
    "previousContext": {
      "contextType": 0,
      "build": null,
      "release": null
    },
    "resultsDifference": {
      "increaseInTotalTests": 16,
      "increaseInFailures": 0,
      "increaseInPassedTests": 16,
      "increaseInOtherTests": 0,
      "increaseInDuration": "00:00:02.5170000"
    },
    "totalTests": 16,                             // Number of tests
    "duration": "00:00:02.5170000",
    "resultsByOutcome": {
      "Passed": {
        "outcome": "passed",
        "count": 16,
        "duration": "00:00:02.1570000"
      }
    },
    "runSummaryByState": {
      "Completed": {
        "state": "completed",
        "runsCount": 1,                           // Number of test runs
        "resultsByOutcome": {
          "Passed": {
            "outcome": "passed",
            "count": 16,
            "duration": "00:00:02.1570000"
          }
        }
      }
    },
    "runSummaryByOutcome": {
      "Passed": {
        "runsCount": 1
      }
    }
  },
  "testFailures": {
    "previousContext": null,
    "newFailures": {
      "count": 0,
      "testResults": []
    },
    "existingFailures": {
      "count": 0,
      "testResults": []
    },
    "fixedTests": {
      "count": 0,
      "testResults": []
    }
  },
  "testResultsContext": {
    "contextType": "build",
    "build": {
      "id": 120,
      "definitionId": 0,
      "uri": "vstfs:///Build/Build/120"
    },
    "release": null
  },
  "teamProject": {
    "id": "068e8d2d-878b-405c-9d71-e653b8412284",
    "name": "PSD-001",
    "state": "unchanged",
    "visibility": "unchanged"
  }
}

I then used my Server Expression Tester to create the correct condition to pass or fail the build using the above information.

You can find more details on the Server Expression Tester in a previous post and download the latest version from GitHub.

And then proceeded wrapping that into a Task:

{
  "id": "25d3d29e-5ea1-4453-9ce6-02e1b34ab30c",
  "name": "Ensure tests have executed.",
  "friendlyName": "Ensure tests have executed.",
  "description": "Ensure tests have executed.",
  "author": "Jesse Houwing",
  "helpMarkDown": "",
  "category": "Test",
  "version": {
    "Major": 0,
    "Minor": 0,
    "Patch": 3
  },
  "visibility": [
    "Build",
    "Release"
  ],
  "runsOn": [
    "Server"
  ],
  "preview": true,
  "instanceNameFormat": "Ensure tests have executed",
  "inputs": [],
  "execution": {
    "HttpRequest": {
      "Execute": {
        "EndpointId": "",
        "EndpointUrl": "$(system.teamFoundationCollectionUri)$(System.TeamProject)/_apis/test/ResultSummaryByBuild?buildId=$(Build.BuildId)",
        "Method": "GET",
        "Body": "",
        "Headers":"{\n\"Content-Type\":\"application/json\"\n, \"Authorization\":\"Bearer $(system.accesstoken)\"\n}",
        "WaitForCompletion": "false",
        "Expression": "le(1, jsonpath('$.aggregatedResultsAnalysis.totalTests')[0])"
      }
    }
  }
}

I grabbed the EndPointUrl from the QueryWorkItems task on the vsts-tasks GitHub tasks.

Tip: Many of Microsoft's tasks do fancy things. Some of these tricks aren't well documented, but most of it is happening out in the open. If you want to replicate a certain behavior it pays to browse through one of these repositories:

- https://github.com/Microsoft/vsts-tasks/
- https://github.com/Microsoft/vsts-task-lib
- https://github.com/Microsoft/vsts-agent

Not only will you find examples and test approaches, but also preliminary documentation.

I packaged the task up in an extension and published it using the CI/CD Tools for Azure DevOps Extensions like I do with all of my extensions.

It looks like people are finding this a useful extension, there have been 21 installs, only 5 star reviews and I've already merged the first pull request. What other test metrics would you like to Ensure as part of your build pipeline? Leave a comment below or file an issue on GitHub!

Photo by: Jon Ardern used under Creative Commons