I've personally shipped a version of my Variable Toolbox extension that impacted 10.000 developers at a single corporation that relied on my pipeline tasks.

And yesterday Microsoft had a world-wide outage due to the npmAuthenticate@0 task suffering from an unexpected breaking change.

Most people don't know what to do when something like this happens. But luckily there are a few things you can do to fix things by yourself in case you find yourself in this situation.

Pin to an older version

If you're using YAML pipelines, then the simplest way to get unblocked is to pin your workflow to a specific version of the task. This is one of the big advantages of YAML based pipelines. By default, Azure Pipelines will reference a task by its major version @0 in the case of the npmAuthenticate task, but you can specify @0.208.1 or any other previous version instead:

# Broken
- step: npmAuthenticate@0
  inputs:
    workingDirectory: $(Build.SourcesDirectory)
    
# Fixed
- step: npmAuthenticate@0.208.1
  inputs:
    workingDirectory: $(Build.SourcesDirectory)

The easiest way to find the version to enter here is to look at the last pipeline that succeeded, the Initialize Job step will list the exact versions used:

2022-12-23T09:05:03.7682146Z ##[section]Starting: Initialize job
2022-12-23T09:05:03.7684106Z Current agent version: '2.213.2'
2022-12-23T09:05:03.7719717Z Current image version: '20221215.2'
2022-12-23T09:05:03.8121128Z Download all required tasks.
...
2022-12-23T09:05:03.8228407Z Downloading task: npmAuthenticate (0.208.1)
2022-12-23T09:05:06.6642168Z ##[section]Finishing: Initialize job

If the task is no longer installed in the Azure DevOps Organization, you can also try to fetch of from the Azure DevOps Marketplace, from the example above. Use tfx extension show to find all the metadata available for that specific extension, then from the list of versions, grab the source location of the asset with type Microsoft.VisualStudio.Services.VSIXPackage. Download and extract that to find the task(s) to upload:

tfx extension show --publisher SQLPlayer --extension-id DataFactoryTools

{
    "publisher": {
        "publisherId": "6e2d04c5-e827-4358-a426-97b85f170d67",
        "publisherName": "SQLPlayer",
        "displayName": "AzurePlayer",
        "flags": 2,
        "domain": "https://sqlplayer.net",
        "isDomainVerified": true
    },
    "extensionId": "d729bc25-03da-47fb-8b04-7cdc6f15df0e",
    "extensionName": "DataFactoryTools",
    "displayName": "Deploy Azure Data Factory (#adftools)",
    "flags": 260,
    "lastUpdated": "2023-05-16T22:39:13.940Z",
    "publishedDate": "2020-05-11T22:03:16.563Z",
    "releaseDate": "2020-05-27T01:18:46.197Z",
    "shortDescription": "Tools for deploying entire ADF code (JSON files) to ADF instance",
    "versions": [
        {
            "version": "1.29.1369",
            "flags": 1,
            "lastUpdated": "2023-05-16T22:39:13.940Z",
            "files": [

                ...

                {
                    "assetType": "Microsoft.VisualStudio.Services.VSIXPackage",
                    "source": "https://sqlplayer.gallerycdn.vsassets.io/extensions/sqlplayer/datafactorytools/1.29.1369/1684276472780/Microsoft.VisualStudio.Services.VSIXPackage"
                }
            ]
        }
  
  ...
  }

Overwrite the broken task

But if you're using classic build and release pipelines (UI based), then you can't pin to a minor version. In that case you'll need to upload a version of the task that works. You can use tfx-cli to do that:

TFS Cross Platform Command Line Interface v0.12.0
Copyright Microsoft Corporation

Syntax:
tfx build tasks upload --arg1 arg1val1 arg1val2[...] --arg2 arg2val1 arg2val2[...]

Command: upload
Upload a Build Task.

Arguments:
  --task-path      Local path to a Build Task.
  --task-zip-path  Local path to an already zipped task
  --overwrite      Overwrite existing Build Task.

But first you'll need to get a copy of the task that works. Your Azure DevOps instance already has a copy of that task, you just need to know where to find it:

$pat = "{{AZURE DEVOPS PAT TOKEN}}"
$org = "{{AZURE DEVOPS ORG NAME}}"
$url = "https://dev.azure.com/$org"
$header = @{authorization = "Basic $([Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(".:$pat")))"}


$taskName = "npmAuthenticate"
$taskid = "ad884ca2-732e-4b85-b2d3-ed71bcbd2788"
$taskversion = "0.208.1"
$taskZip = "$taskName.$taskid-$taskversion.zip"

Invoke-WebRequest -Uri "$url/_apis/distributedtask/tasks/$taskid/$taskversion" -OutFile "$taskZip" -Headers $header

Or you can grab a copy of an older version from my azure-pipelines-tasks-zips repository:

Now to "fix" the broken version we'll have to overwrite it. To do so we're going to patch the older working version with the broken version's version number.

First extract the zip file, then update the task.json file and set the version number to the one you want to overwrite (in our case 0.214.1):

Put the task.json back into the zip file.

We're now ready to upload the task. We can use tfx-cli to do that.

We first need to delete the existing task using tfx build task delete --task-id (warning this will delete all older versions):

And then we can upload the patched task tfx build tasks upload --overwrite --task-zips-path file-we-just-patched.zip:

Instead of deleting the existing task and re-uploading it, we can also patch the taskzip with a version number that's newer than the broken task, in our case 0.214.2 and then upload that. In that case we won't have to delete all older versions.

As you can see here, that's what the Azure Pipelines team does too, but they roll out the task through their own provisioning process, they either hotfix the task, or simply roll back their changes:

Temporarily replace the broken task

Instead of overwriting, we can also install a temporary version of the task with a different id and name, reconfigure our pipelines to use the temporary replacement and switch back when the Azure Pipelines team has fixed theirs.

The process is very similar to replacing the existing task, but in this case, we need to make a few more changes, we need to replace the id (choose any guid), name and update the friendlyName properties in both task.json and  task.loc.json (if it exists):

Or you can download a pre-patched (-sxs) version from the azure-pipelines-tasks-zips repository.

Then we can upload the patched file:

But now you'll need to go through all your pipelines and replace the old task with the temporary replacement. A bit of scripting against the REST API would make that a quick change.