Say goodbye to your Personal Access Tokens

Michael, author of some of my favorite Azure DevOps Extensions, reached out on the Azure DevOps Club slack channel for help. He wanted to automatically rotate Personal Access Tokens to integrate with Azure DevOps, which lead to a quest to completely eradicate Personal Access Tokens altogether. After sharing my work to make the Azure DevOps Extension Tasks work without Personal Access Tokens, Michael went to work. I've asked him to share his results in this guest blog, since I feel they're useful for anyone doing automation against Azure DevOps. As you can see he agreed!

Jesse Houwing

PATs, PATs and more PATs - our 5 scenarios we used PATs for

Building Bravo Notes here at Agile Extensions we are extensively using the Azure DevOps REST APIs in many ways.

And I'm not talking about the API calls the Bravo Notes extension makes at runtime e.g. to load work items, publish wiki pages etc.

We use the Azure DevOps REST APIs in these 5 scenarios as well:

1. publish multiple versions of the Bravo Notes extension for development, staging and production via the marketplace APIs (in Azure Pipelines)
2. retrieve marketplace event data for installs/uninstalls and more on a schedule (in Azure Functions app)
3. run integration tests for many Bravo Notes components that need Azure DevOps APIs (in Azure Pipelines)
4. run e2e tests of the main Bravo Notes app outside of Azure DevOps (in Azure Pipelines)
5. run integration and e2e tests locally

In the past we used personal access tokens (PATs) for all of those scenarios.

For years our process was:

• watch for notifications that a PAT was about to expire
• be annoyed that this manual regular chore was on your plate (AGAIN!)
• regenerate PAT in Azure DevOps
• update pipeline variables and secrets with the new PAT
• feel bad and wonder why there wasn't a more elegant solution
• be done and push responsibility for improving this to future Michael

PAT REST APIs to the rescue?

A while I ago a new set of APIs got introduced to manage PATs. That sounded intriguing and a few weeks back I reached out to the community to ask for guidance on how to automatically renew and manage PATs and do away with the manual process entirely.

🤔 Me: Is there a good resource online on how to transition from manually rotating PATs to rotating PATs automatically (a pipeline using PAT REST API, KeyVault)?

Thankfully Jesse pointed me to a better (the right) way:

💡 Don't use PATs but use an Azure Service principle with workload Federation.

When you are reading this on his blog you probably already know that Jesse is one of those people who uses the internet for what it was build for:

He shares what he has learned (often the hard way) to make the lives of all of us easier. In this case in form of this blog post on how to publish marketplace extensions without using PATs.

I was convinced already that using PATs was less than ideal.

But at this point I was still hesitant about whether this approach could help us do away with PATs entirely. After all we had 5 different scenarios.

There was only one way to find out so we started by tackling the scenario that Jesse wrote about in his blog post.

Scenario 1: Publish marketplace extensions without a PAT

The blog post didn't disappoint as it contained all the pieces to help us publish marketplace extensions without a PAT - whohoo 🥳!

Below are the steps we needed to take. Please refer to Jesse's blog post for all the datails. Our mileage varied a bit as the ARM service connection now uses an "App registration" in Azure instead of a "Service Principal".

The process is the still the same in general, though.

1. Create the ARM service connection (the UX changed a bit since Jesse's blog post)
   • Choose "Azure Resource Manager" Connection
   • Identity Type: "App registration (automatic)"
   • Credential: "Workload identity federation"
   • the rest is about the same
2. Add the App registration/service principal as a user in Azure DevOps
3. Extract the Azure DevOps Identity Id from the Profile API via a pipeline task
4. Use that Id to add the App registration as a member to the marketplace publisher
5. Update our pipeline to use the ARM service connection

Note that we could use the newest version of the marketplace extension tasks (v5) that allows you to publish using the ARM service connection directly.

So there is no need to fetch an access token and overwrite the marketplace service connection credentials anymore. After changing our pipelines we were able to delete our old "Visual Studio Marketplace" service connection.

- task: PublishAzureDevOpsExtension@5
  displayName: 'Publish Extension'
  inputs:
    connectTo: 'AzureRM'
    connectedServiceNameAzureRM: 'marketplace-service-connection'
    fileType: vsix
    vsixFile: '$(Pipeline.Workspace)/vsix/production.vsix'
    updateTasksVersion: false

And that was it. The pipeline worked as before and we could revoke the PAT for publishing extensions to the marketplace.

Of course we didn't want to stop here and tackle the next scenario and remove one more PAT.

Scenario 2: Call Azure DevOps APIs from a Azure Function app

We maintain an Azure Function app to query data about install and uninstall events and pull statistics from the marketplace. Authentication works using - what else could it be - a PAT.

In Azure Functions of course we don't have the magic of service connections that we can utilize in Azure Pipelines.

After initially failing to find the code to use for our NodeJS based function app, searching for the magic GUID 499b84ac-1321-427f-aa17-267ca6975798 that represents Azure DevOps led me right to what we needed to acquire an access token inside the Function app.

After finding the right way to acquire the access token, we got this solved in 2 simple steps:

1. Add the Azure Function app as a user in Azure DevOps
2. Use the code below to acquire an access token and authorize Azure DevOps REST API calls

const { ManagedIdentityCredential } = require("@azure/identity");

async function getMarketplaceAccessToken() {
    const credential = new ManagedIdentityCredential({
        // `AZURE_CLIENT_ID` points to the App registration created via the ARM service connection
        clientId: process.env.AZURE_CLIENT_ID,
    });
    const tokenResponse = await credential.getToken(
        "499b84ac-1321-427f-aa17-267ca6975798/.default",
        {
            tenantId: process.env.AZURE_TENANT_ID,
        }
    );

    return tokenResponse.token;
}

There was no stopping now - but still 3 PATs to go.

Scenario 3: Run intergration tests in Azure Pipelines

Next up was our integration testing pipeline tasks that run integration tests using NodeJS and jest. Until now a PAT was used from a secret pipeline variable. Another one that we didn't want to renew manually every few months.

As we already acquired an access token in a pipeline via an ARM service connection in scenario 1, this dind't seem to difficult to pull off.

Following the least priviledge principle we didn't want to use the same service principal from scenario 1. This would have given it access to the marketplace AND Azure DevOps resources that were needed for our integration tests.

Luckily you can create as many ARM service connections as you want and so we did. To use the newly created App registration/service principal, the pipeline task from Jesse's blog post to retrieve an access token that we dind't need for scenario 1 now came in handy again:

- task: AzureCLI@2
  displayName: 'Accquire token for integration testing'
  inputs:
    azureSubscription: 'azure-devops-integration-testing-connection'
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    useGlobalConfig: true
    inlineScript: |
      $accessToken = az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query "accessToken" --output tsv
      write-host "##vso[task.setsecret]$accessToken"
      write-host "##vso[task.setvariable variable=SECRET_INGETRATION_TESTING_TOKEN;issecret=true]$accessToken"

Now we could use the the token for subsequent integration test tasks. Yay only 2 PATs to go!

Scenario 4: Run e2e tests in Azure Pipelines (against external Azure DevOps organization)

While this scenario seemed to be exactly the same es scenario 3 at first, we had an issue here that stopped us in our tracks at first.

The e2e tests called APIs in a separate Azure DevOps org that only existed for testing.

[OrgA Pipeline] ---- REST API call ----> [OrgB Work item API]

The solution was too simple to be true.

We were just able to add the App registration/service principal as a user in OrgB and give it access to the resources needed for the e2e tests.

Another PAT down!

Scenario 5: Running integration and e2e tests locally

This might be my least favorite solution so far, but it was simple and did the trick for now.

We use jest to run our tests and in our npm scripts we just stuck in a Azure Developer CLI call to log in prior to starting the test script.

In our code to set up API authentication we now run some special code to acquire the access token, again with the azure identity SDK.

So

1. Modify testing script to log in - azd auth login && npx jest --config jest.integration.config.js"
2. Retrieve the access token from the Azure Developer Cli:

let token = process.env.SECRET_INGETRATION_TESTING_TOKEN;
if (!token && !process.env.CI) {
    const credential = new AzureDeveloperCliCredential({
        tenantId: process.env.AZURE_TENANT_ID,
    });
    const tokenResult = await credential.getToken(
        '499b84ac-1321-427f-aa17-267ca6975798/.default',
        {
            tenantId: process.env.AZURE_TENANT_ID,
        },
    );
    token = tokenResult.token;
}

(Would love to hear about a better solution to securly get an access token for running scripts locally in a development environment.)

All 5 scenarios done! The champaign 🍾 was almost open when I revoked all PATs that were not needed anymore when I discovered that I had forgotten one last PAT.

Bonus scenario: the last token - the renovate pipline

We use renovate to update dependencies and automatically create pull requests in yet another pipeline. (You can thank Jesse for his work online here as well).

Immideately I had a bad feeling and I thought that renovate would only accept PATs as it only accepts RENOVATE_TOKEN as a means to authenticate. The docs make it look that way for sure.

Looking at the code (yay open source) I could see that renovate automatically detects whether the token is a PAT or a OAuth token and chooses the Authorization header (Basic or Bearer) accordingly.

So we could just use the same AzureCLI pipeline task as in scenarios 3 to acquire and 4 and be done!

P.S.: Talking to Jesse about that he mentioned that Azure DevOps now accepts OAuth tokens when using the Basic authentication scheme as well which is great in my opinion.

Wrapping up

Just reading all the terms involved:

• ARM service connection
• Open Id Connect
• Workload Identity federation
• service principal
• App registration

made the whole process seem daunting to me.

But in the end we got rid of all our PATs within a couple of hours.

Personal access tokens are great to quickly test an API call, authenticate a throw-away script where you don't have other means of authentication ready.

However for the long term you really don't want to have to create PATs and renew them until the end of time.

Thanks again to Jesse and Joost Voskuil for their help.

In case you are facing your own challenge with Azure DevOps or GitHub, don't hesitate to join the Azure DevOps Club.