Fixing complex bugs with AI - A deepdive into the PowerShellScriptAnalyzer

Some build failures are annoying. Others are simply impossible. Mine was the second kind.

On a plain ubuntu-latest runner, a recursive Invoke-ScriptAnalyzer run would stop dead with an error saying that Get-Command was not recognised as the name of a cmdlet. Not some obscure module. Not a typo in a script. Get-Command. The single most fundamental discovery cmdlet in PowerShell, missing from PowerShell.

The same workflow had been perfectly happy on PSScriptAnalyzer 1.24. The only thing that changed was the analyser version.

What follows is the story of PR #2206. It started as a bug fix, turned into a performance project, and nearly became a pile of AI-generated slop. Along the way, AI tooling helped me, led me in the wrong direction, and showed that getting the code ready is not the same as getting it released.

It starts because it is open source

The broken workflow was mine. It was actions-semver-checker, one of the small projects I maintain, and the only change in the failing pull request was the pinned analyser version. So I filed issue #2205 with links to both runs.

This is easy to take for granted.

In most of my working life, that would have been the end of my involvement. You raise a ticket. You attach logs. You wait. Somebody you will never meet triages it against priorities you cannot see, and your only remaining move is to pin to the last version that worked and hope the next release quietly fixes it. You are a spectator to a problem that is costing you time.

Here I could carry on. The repository is public, so I cloned it, diffed 1.24 against 1.25, and had a source-level diagnosis in the issue within twenty minutes: AvoidAlias asks the engine about every non-alias command it sees; that question travels through CommandInfoCache, which used a pool of child runspaces and ran Get-Command inside them via PowerShell.Create(); PR #2125 had recently taught that path to parse Module\Command and pass -Module along, and the build reference for System.Management.Automation had moved from 7.4.7 to 7.4.14.

So the error had a fairly ordinary cause. We were not losing Get-Command from PowerShell. We were losing it from our own child runspace, under concurrency, on a machine with a particular set of modules on disk.

In that first write-up I was careful about what I could not prove. My repository does not use module-qualified invocations, so I could not show that #2125 alone caused it. I could only show that it was the most likely place to look. It is important to keep measurements and suspicions separate.

No support ticket or NDA. I could read the code, form a theory, write a test and open a pull request. That ability to take ownership is one of the main reasons open source matters.

The first fix was the obvious one, and it was not enough

My first instinct was containment. Catch the transient failure, return null, and let the rules follow their existing "I could not resolve that command" path. A failed lookup should not take down an entire analysis run.

One detail mattered: the cache stored Lazy<CommandInfo> values, and Lazy<T> caches exceptions as well as results. One unlucky lookup would poison that entry for the rest of the process. So the fix evicted only the faulted entry rather than clearing everything.

Then I went a step further and serialised all command lookups onto a single dedicated runspace behind a lock. If the engine is not thread-safe here, stop being concurrent here.

That felt like the answer. It compiled, the tests went green, and the original failure disappeared.

It was not the answer.

The review that changed the question

Twenty-six days later, one of the maintainers replied. And in a few short paragraphs he handed me a pile of context I had no way of acquiring on my own.

He told me this was not the first time CommandInfoCache had produced concurrency issues — there had been several, over a period, from different reporters. He told me the root cause as the maintainers understood it was PowerShell engine internals not being thread-safe, which turned my careful hedge ("I cannot prove #2125 alone triggers this") into a settled question. He pointed me at an attempt to fix it on the PowerShell side. And he mentioned, almost in passing, that he had previously seen lock-based optimisations in this area degrade performance badly by removing concurrency.

I want to be fair about what I could have found on my own. Working through the 1.24-to-1.25 diff revealed links to the PowerShell runtime repository, where the underlying concurrency problems are documented.

What he added was the knowledge that had not been written down. This was not a one-off. The maintainers already agreed on the likely cause. Most importantly, a previous lock-based fix had made this code much slower. It was tried and abandoned, so there was no merged change for me to find.

He did not replace a day's research with five minutes of magic. He gave that research context that only a long-serving maintainer had. He told me which questions were already settled, which ideas had caused trouble before, and where the real risk was.

Then he asked for one specific thing.

Analyse PowerShell's own build.psm1 in a freshly started shell, then immediately again in that same shell. Do it for my branch and for the implementation in 1.25. Post the numbers.

Those two measurements matter because PSScriptAnalyzer gets used in two very different ways, and the rest of this story hangs on the distinction.

In a pipeline, it runs once. A GitHub Actions job starts a runner, installs the module, runs Invoke-ScriptAnalyzer -Recurse, and exits. Nothing is cached or warmed up. If it takes thirteen seconds instead of three, every contributor pays those ten seconds on every push and pull request. If it fails, the check is red and somebody has to find out why.

In an editor, it keeps running. The PowerShell extension in VS Code keeps the analyser alive in a background process and runs it again as you type. The first analysis after opening a workspace is a one-off start-up cost. After that, the caches are hot and .NET has optimised the busy code paths. Those later runs need to be fast and consistent, without warnings appearing and disappearing between keystrokes.

So the benchmark measures both situations:

• The first run in a fresh process stands in for the CI job. It is the number most users experience.
• The immediate repeat run in the same process stands in for the editor. It shows what happens once everything is populated and optimised.

The two numbers can move in opposite directions. Aggressive caching makes the second look good at the expense of the first. Serialising everything behind a lock — my fix — protected correctness but made the first run much slower. Non-determinism is also easy to miss in timings: in an editor, it becomes a warning that appears and disappears while you work.

Keep that trap in mind, because I walked straight into it later on and very nearly shipped the result.

(In the pull request I call these the cold and warm runs.)

He had seen this type of fix go wrong before and asked for the cheapest experiment that would settle the question.

At that point the pull request stopped being "can we make it stop crashing" and became "can we make it stop crashing at a cost anyone would accept". The information that changed the work was not in the repository. It came from a person.

The review was the most valuable input I received in the entire project. A maintainer is not an obstacle between you and a merge.

Building a benchmark

So I built a benchmark. Not a stopwatch in a script — an isolated GitHub Actions matrix across Ubuntu and Windows, analysing two real projects (the PowerShell repository itself, and actions-semver-checker, the very repository that had started all this), timing both the fresh-process run and the immediate repeat, and taking a SHA256 fingerprint of the normalised findings so I could prove the analyser's output had not changed while its timings did.

Every comparison from here on is my branch against the implementation in 1.25, built from source in the same job, on the same runner, analysing the same files.

Then the benchmark told me my solution was bad.

Analysing the PowerShell repository in a fresh process, 1.25 took 7.45 to 8.12 seconds. My branch took 25.90 to 27.54. On the semver project, 9.49–12.76 seconds became 33.55–36.80. Repeat runs in the same process were fine, even slightly better. But the fresh-process runs — the ones that represent almost every real use — were roughly three times slower.

The maintainer's response was pragmatic: perhaps make it an opt-in switch, so people who need determinism can pay for it and people who need throughput can keep their concurrency.

I did not want to ship that compromise. A correctness fix that you have to opt into is a correctness fix most people will never get.

There was one encouraging result in that matrix. The 1.25 samples kept failing and needing reruns. Mine never did. The benchmark had also become a determinism test.

How it nearly became slop

The middle of the project is not the part people usually publish.

Over a few hours on 14 September, the branch gained retry loops, telemetry counters, two build variants, per-rule exception handling and a summary job for the growing matrix. I then had to fix the matrix because a later change overwrote the results and made the retry variants indistinguishable.

Every change was reasonable on its own. Copilot produced several of them quickly from a clear brief. I ran eleven separate agent tasks on my fork; eight were merged and three were closed as superseded. I would not have explored half as many directions without it.

But I had built an increasingly complex system for recovering from the problem, followed by another system for measuring the recovery. The diff and test suite kept growing, while none of it addressed why we were doing something dangerous in the first place.

This is what I mean by a vibe-coded mess. The individual changes were not bad code and could have passed review on their own. The problem was the direction: an agent will keep adding layers unless you stop and ask a different question.

The numbers did not improve enough to justify the complexity.

So I deleted it. Retries, telemetry, build variants, rule-level catches — all of it came out in one commit. The repository-wide regression test, which the automated reviewer had flagged as depending on whatever happened to be in the repo, was replaced by a small, self-contained test that hammers the rules in parallel. The branch got smaller for the first time.

Asking a different question

With the scaffolding gone, I went back to instrumenting, but this time to answer one question: why are we calling into PowerShell so much?

The answer was stark.

• A failed Get-Command lookup cost around 59 ms. A hit cost around 0.8 ms. Seventy-four times more expensive, because a miss sends PowerShell off to autodiscover modules across the entire PSModulePath.
• On a real 111-file repository, 170 of 223 lookups — 76% — resolved to nothing. Those misses accounted for roughly 77% of total runtime.
• Priming the cache did not help. Get-Command * took 78 seconds to enumerate around 116,000 commands, and later misses were no faster because PowerShell does not cache negative lookups.
• The cache was already shared across Invoke-ScriptAnalyzer calls within a process. A second analysis in the same session performed zero lookups. The editor case was already in good shape. The pipeline case — one analysis, from nothing — was the whole problem.

So the expensive operation was not "look up a command". It was "look up a command that does not exist". And once you frame it that way, the question becomes: which of these 170 non-existent commands should we never have asked about?

That turned out to be a much better question, and the answers were not performance bugs. They were correctness bugs that happened to be slow.

Four lookups we should never have made

Probing Get- in front of a hyphenated name. AvoidAlias has a sensible fallback: if it sees process, it tries Get-Process. But for an unrecognised Test-ActionInput it produced Get-Test-ActionInput, which matched none of the 116,289 commands on my machine and doubled the cost of every miss.

Ignoring functions the script itself defines. If my script defines its own Get-ChildItem, the analyser was resolving that name against the cmdlet installed on my machine. UseCorrectCasing could then advise me to rename a function based on a command my script deliberately shadows. Collecting FunctionDefinitionAst names once per analysis fixed the advice and removed many 59 ms misses at the same time.

Treating member names as commands. UseShouldProcessCorrectly built a call graph with a vertex for every invocation member, so $x.Substring(1) triggered Get-Command Substring. Worse, $dir.mkdir() could be credited with SupportsShouldProcess, because mkdir happens to exist as a PowerShell function on Windows. A method call was being scored against an unrelated cmdlet. Tagging vertices reached through genuine command invocation fixed the semantics and the cost.

Forgetting that modules dot-source themselves. A function in one library file routinely calls a function defined in a sibling file, with no import visible in either. I grouped files by directed dot-source relationship and gave each file the functions reachable through its own dot-source closure — deliberately not merging entire connected components, so a test script that dot-sources one library does not inherit functions from its siblings.

That last one took a 111-file module from 13.5 seconds to 2.9 in a fresh process. Repeat runs unchanged. Findings identical.

Then I removed the instrumentation counters. They had done their job.

The complete list of fixes

Here is the full set, sorted by the kind of problem it is.

Threading and lifetime bugs

Concurrent use of a non-thread-safe engine surface. CommandInfoCache resolved commands through a pool of child runspaces, each running Get-Command via PowerShell.Create(). PowerShell's command-discovery internals are not safe to drive this way from multiple threads. The symptoms included CommandNotFoundException for Get-Command itself, null reference exceptions, pipeline creation failures, even process aborts and hangs in PowerShell 5.1. The pool is replaced with one dedicated runspace behind a re-entrant lock.

Lazy<T> exception poisoning. The cache stored Lazy<CommandInfo> values, and Lazy<T> caches the exception just as faithfully as it caches a result. One transient failure permanently poisoned that command for the rest of the process, so a momentary race became a persistent wrong answer. Now only the faulted entry is evicted — and crucially, genuine negative results are still cached, so "this command really does not exist" does not become a repeated 59 ms lookup.

Unsynchronised disposal. The disposal paths did not take the runspace lock, so teardown could race an in-flight lookup. Both paths now lock.

Metadata access escaping the lock. Even with lookups serialised, rules were reaching back into the engine afterwards — resolving parameters on a CommandInfo re-enters PowerShell on the caller's thread, outside the protection. Metadata reads are now serialised alongside the lookups through a Func<CommandInfo, T> callback, and static export metadata is used in preference to dynamic parameter resolution where it is available.

Get-Command resolved by unqualified name. The internal call used the bare name, which is itself subject to shadowing and discovery failure. It is now fully qualified as Microsoft.PowerShell.Core\Get-Command.

Non-deterministic output as a consequence. All of the above combined into the symptom that bothers me most: the same input produced 154, 155 or 157 findings depending on which lookups happened to lose a race. In a pipeline that means a build that fails on a rerun of an unchanged commit. In an editor it means warnings that appear and vanish while you type. Static analysis that is not reproducible is not much use in either.

The analyser could also crash. This is what users saw.

Your check goes red with an error saying Get-Command does not exist. Nothing in that message points at the analyser's internals. You re-read your own diff and wonder whether the runner image changed, a module failed to install, or PowerShell itself is broken. Eventually you rerun the job. A fresh run succeeded between 55 and 59 times out of 100, so it may turn green without explaining anything.

When that happens often enough, people pin to an older version, add continue-on-error, exclude the rule, or remove the analyser from the pipeline. That is what I did first: I reverted to 1.24 and continued with other work. People rarely file an issue to say they have stopped using your tool. An unreliable check can leave you with silence and a linter nobody trusts.

Logic and semantic bugs

Script-defined functions did not shadow installed commands. If a script defined its own Get-ChildItem, the analyser resolved that name against whatever was installed on the machine running the analysis. UseCorrectCasing would then recommend casing taken from a cmdlet the script deliberately overrides. The analyser was modelling the host instead of the code.

Dot-sourced siblings were invisible. The same bug at module scale. A library file calling a function defined in a sibling file — the normal way a PowerShell module composes itself — resolved against the host rather than the module. Fixed by giving each file the functions reachable through its directed dot-source closure, deliberately without merging whole connected components, so a test script that dot-sources one library does not inherit unrelated siblings' functions.

Member names were resolved as commands. UseShouldProcessCorrectly created a call-graph vertex for every invocation member, so $x.Substring(1) produced a Get-Command Substring. The real damage was the false negative: $dir.mkdir() was credited with SupportsShouldProcess, because mkdir exists as a PowerShell function on Windows. A method call was silently scored against an unrelated cmdlet, and a rule violation went unreported. Only vertices reached through genuine command invocation are resolved now.

The Get- prefix probe applied to hyphenated names. The fallback exists to turn process into Get-Process. Applied to an unresolved Test-ActionInput it asked about Get-Test-ActionInput, which matches none of the 116,289 discoverable commands on my machine.

A rule's own test fixture was passing for the wrong reason. A fixture declared SupportsShouldProcess and never called ShouldProcess, which should be a violation. But it defined a local Set-Service that was resolved as the system cmdlet, masking the result. The shadowing fix exposed that false negative in the test suite.

Performance problems

Failed lookups dominated the fresh-process run. 59 ms for a miss against 0.8 ms for a hit, because a miss sends PowerShell autodiscovering modules across the whole PSModulePath. On a real 111-file repository, 170 of 223 lookups resolved to nothing — 76% of the calls, roughly 77% of the runtime.

Every semantic fix above is also a performance fix. Script-local and dot-source resolution answer from the syntax tree instead of the engine. Dropping the Get- probe halves the cost of the misses that remain. Not resolving member names removes an entire category of lookup that was never meaningful. The speedup comes from asking fewer questions, not from answering them faster.

Files are parsed once per run. The dot-source pre-pass retains the syntax trees it builds rather than discarding them, so a recursive analysis no longer reparses. -Fix opts out, because it rewrites each file before analysing it.

Serialisation stopped mattering. The lock cost that made fresh-process runs three times slower in September is still there — I did not remove it, it is what makes the analysis deterministic. It simply stopped being on the hot path once lookup volume collapsed.

One thing is deliberately not changed: parameter and metadata re-evaluation. I assumed it was a bottleneck, but instrumentation showed around 97 lock acquisitions per pass. It was not worth optimising.

The trade I nearly accepted

By this point I was pleased with the progress.

The pipeline case had gone from thirteen and a half seconds to under three. That was the number I had been chasing since the maintainer's review, it was the number in the issue that started all this, and it was the number almost every user would feel. I had been staring at it for days.

The editor case, meanwhile, was a couple of seconds slower on repeat runs. I started telling myself that was fine: CI pays for the first run on every push, while a second or two in a long-lived editor session is barely noticeable. A four-times improvement in CI for a small editor regression seemed acceptable.

But it was the trade I had told myself not to make when I decided to measure both cases. It is easy to ignore a principle when one of the numbers looks so good.

I would have had to explain that trade-off to a maintainer who had already warned me about regressions in this area. Instead of arguing it away, I investigated it.

The benchmark was lying to me, twice

Two things then made the harness itself the suspect.

The repeat-run numbers looked like a regression, but the first run wrote its findings to disk before the second measurement began. Garbage collection for those allocations landed on the second run's clock. Running a collection before each timed run removed this effect.

The second problem was less obvious. Once the fresh-process run became three times faster, the second run was no longer a steady state. .NET starts with quick, unoptimised code and recompiles methods after enough use. That work used to fit inside a 13-second first pass. Now it continued into passes two and three, so my "fully warmed up" run was still warming up. A few discarded analyses before the measurement also removed this effect. Disabling tiered compilation confirmed the cause: in that case the second pass was already stable.

The lesson applies beyond this project: a benchmark is software, with its own bugs. Improving the product had invalidated assumptions in the measurement.

Chasing the repeat-run numbers also revealed work the analyser repeated after everything was cached: files were reparsed and lookups repeated. Almost all of that work also happened on the first run, so improving the editor case improved the pipeline case too.

The regression was not the price of a faster first run. It showed that I had more work to remove. I did not foresee that; I found it because I was unwilling to accept the regression.

Evidence reviewers can check

My concern with AI-assisted open source contributions is not the code but the claims. A pull request saying "fixes a race condition, improves performance" gives a maintainer little they can verify.

So I set out to produce evidence that a reviewer could check without trusting me.

The final cross-platform CI results. "First run" is a freshly started process analysing the project once; "repeat run" is a second analysis immediately afterwards in that same process.

The pipeline case is two to three times faster, while the editor case is unchanged or slightly better. What looked like a trade-off was work I had not removed yet.

On the 111-file project across two CI attempts, 1.25 crashed 27 times. The runs that survived reported 154, 155, or 157 findings for the same input. The proposed changes completed all 40 measurements and always produced the same fingerprint and 154 findings. One caveat: the benchmark retries 1.25's failures which also means the timings exclude its worst runs and make it look faster than it actually is.

I proceeded with local stress testing, 100 freshly started processes per configuration: 1.25 succeeded 59 times out of 100 on PowerShell 7.6.6, and 55 out of 100 on Windows PowerShell 5.1. With the proposed changes applied: 100 out of 100 in both. Across 400 analyses in total, 86 runs of 1.25 died outright — 58 command-not-found errors, 22 null references, several pipeline creation errors, two process aborts and two timeouts. All of those signatures went into the PR description, because "it crashes sometimes" is an anecdote and a list of distinct failure modes is evidence.

One experiment also justified deleting the retry code. In 100 freshly started PowerShell 7 sessions, 35 hit a caught failure on their first attempt. Twelve recovered later in the same session. Twenty-three kept failing until a 120-second cut-off. Retrying is not recovery when the failure may have corrupted the state you are retrying in. Twelve recoveries out of thirty-five failures is not a reliable strategy. I only know that because I measured it.

Build and test problems along the way

This project also took many hours on things that had nothing to do with the bug but were necessary to work on it.

The build pins .NET SDK 8.0.4xx, and having SDK 10 installed does not help. Building with -PSVersion 7 produces only the net8 output; the module root DLL is the net462 build that Windows PowerShell 5.1 loads, and it stays stale until you build -All. I spent some time investigating why fixed code still failed before I understood that.

My editor's built-in test runner was worse than useless here, because it runs Pester without importing the repository build — so Invoke-ScriptAnalyzer silently resolved to the globally installed PSScriptAnalyzer. The symptom is beautifully misleading: your new behaviour tests fail while every control test passes. The fix is to replicate the harness and prepend out to PSModulePath explicitly.

PSUseCorrectCasing is a configurable rule and is off by default, so -IncludeRule alone does not switch it on. For A/B comparisons, build the baseline in a git worktree rather than stashing; otherwise it is easy to lose track of which build is running. Test fixtures that use Add-Type must also be C# 5 compatible, because the compiler Windows PowerShell 5.1 uses rejects null-conditional operators and expression-bodied members.

These detours are often called yak shaving. Copilot handled much of it: it read build errors, found the SDK mismatch and noticed the stale module. Work that might have taken my evening happened while I did something else. But each loop used tokens, and the agent focused on the immediate problem rather than the state of the machine. It installed .NET 8 into a temporary folder at least five times before I noticed and installed it system-wide in about a minute. It had solved the local problem each time without asking why it kept returning.

It also changed how I worked. I could start a task, do other work, check the result and correct the direction. I steered part of this project from my phone in a school car park while waiting to collect my kids.

Those detours still had to be correct. The test runner using the globally installed module could have sent me changing code that was already right. This work is mostly invisible in the finished pull request, but it is a normal part of contributing to an established project. AI can handle much of it, but somebody still has to notice when it solves the same problem five times.

Where AI helped

Copilot was very good at bounded, well-specified work. Build an isolated benchmark matrix. Make this fixture cross-platform. Trace where these 106 unresolved lookups come from. Port this test to C# 5 syntax. With a clear brief and a way to verify the result, the agents produced work I merged with minor edits. They were also useful for the tedious middle of the investigation: another stress-test variation, readable result formatting, or broken links in a pull request description. The instrumentation that produced the 59 ms figure took minutes rather than an afternoon. That made the difference between running the experiment and deciding it was too much work.

What it did not do was tell me when to stop. It did not question the four layers of retry machinery, notice that a local Set-Service made a test pass for the wrong reason, or ask whether the repeat-run slowdown came from the product or the benchmark. I changed direction because the numbers looked wrong.

Could an agent have got there by itself? Possibly. An agent reviewing a diff can identify these problems, and repeated review and revision can improve the result. I have not found a reliable way to know when that loop is finished. It may stop at a plausible answer or continue until it loses direction.

That is why I kept steering it by hand. A few minutes deciding where to look is cheaper than asking the machine to discover the direction. "Trace where these unresolved lookups come from" is targeted and verifiable. "Find out why this is slow and fix it" is open-ended and can use many tokens exploring ideas I could reject quickly. Often the human contribution is simply narrowing the search before the machine starts.

This is the difference between vibe-coded code and code written with the same tools but properly reviewed. It is not the amount of code the model produced. It is whether the person directing it can ask a precise question. "Is the dot-source closure transitive, and should it be?" requires enough knowledge to understand why transitivity could be dangerous. Without that knowledge, you are left with "make it faster" and "fix the failing test", and may not be able to spot a plausible but wrong answer.

Expertise can also narrow the questions you ask. I spent this project asking "why is this lookup happening?" and much less time asking "should command resolution work this way at all?" or "how would I design this today?". People familiar with a codebase tend to accept its existing shape. An agent does not, which can be annoying but also useful. Next time I want to spend some time asking the basic questions before I choose a direction.

There is another limit: what an agent can know. An assistant can read an issue tracker and follow links into another repository. That is how the PowerShell runtime issues behind this bug came to light.

It cannot see what was never committed: an approach tried two years ago and abandoned because it halved throughput, or a conclusion a maintainer never wrote down. A repository records what survived, not everything that was attempted. A model can compare an approach with the code that exists. A maintainer can also tell you which ideas have failed before.

AI made me much faster at producing possible solutions and evidence. It did not decide which agent branches should survive, what to remove, or when the numbers were misleading. Without these tools, however, I would almost certainly never have done the work.

This is not my job. It is a side project on somebody else's repository, fitted between client work, family and everything else. I was never going to spend three uninterrupted evenings fixing the build, writing a benchmark harness and running four hundred stress analyses by hand. I would have pinned to 1.24 and moved on. Instead I could work in twenty-minute pieces, with an agent continuing while I did something else.

So the wasted tokens and repeated SDK installs need that context. The alternative was not a cleaner version of this pull request. It was no pull request, an unfixed bug, and a linter that a few more people might stop trusting.

The hard part comes next

The technical work is nearly done. The crashes are gone, a first analysis in a fresh process is two to three times faster, and there is a benchmark anyone can rerun. Now it needs to be released.

That carries risk. Three of the four main improvements are behavioural changes to command resolution. Functions defined by a script now shadow installed cmdlets. Member invocations are no longer resolved as commands. Dot-sourced files now share their function definitions. These are fixes, but they also mean that some users will see different findings.

Some of those differences are the whole point — the false SupportsShouldProcess credit from $dir.mkdir() should never have been there. But "we removed a false negative" reads very differently when it lands in your pipeline as a new build failure on a Friday afternoon.

Then there are custom rules that I cannot see or test. I have checked the projects I can reach: internal ones, actions-semver-checker, PSScriptAnalyzer's own build and test suite, and a handful of open source repositories. The results are consistent, but that is not proof.

So please go and break it

This is where I need your help.

If you use PSScriptAnalyzer in CI, in your editor, or with custom rules, please run the proposed build against your code and tell me what you find. Ten minutes of your time is worth more than another day of mine because I cannot test your code myself.

Build it from the pull request:

# Install the .NET 8 SDK if you do not have it (on Linux/macOS: https://dot.net/download)
winget install Microsoft.DotNet.SDK.8

# Requires the GitHub CLI
git clone https://github.com/PowerShell/PSScriptAnalyzer.git
Set-Location PSScriptAnalyzer
gh pr checkout 2206

# -All also builds the net462 output that Windows PowerShell 5.1 loads
./build.ps1 -Configuration Release -All

$proposed = (Resolve-Path ./out/PSScriptAnalyzer/1.25.0/PSScriptAnalyzer.psd1).Path
$project  = 'C:\src\your-project'

Then analyse your project twice, once with the version you use today and once with the proposed one, and compare the results. Use a fresh process for each so neither run can contaminate the other:

$analysis = {
    param($ModulePath, $ProjectPath, $OutputPath)

    $ErrorActionPreference = 'Stop'
    $temporaryPath = "$OutputPath.tmp"
    try {
        Import-Module $ModulePath -Force -ErrorAction Stop
        Write-Host "Analysing with PSScriptAnalyzer $((Get-Module PSScriptAnalyzer).Version)"

        # Collect everything before writing anything, so a failed run cannot leave a partial CSV.
        $findings = @(Invoke-ScriptAnalyzer -Path $ProjectPath -Recurse -ErrorAction Stop)
        try {
            New-Item $temporaryPath -ItemType File -Force | Out-Null
            $findings |
                Select-Object RuleName, Severity, ScriptName, Line, Column, Message |
                Sort-Object RuleName, ScriptName, Line, Column |
                Export-Csv $temporaryPath -NoTypeInformation -ErrorAction Stop
            Move-Item $temporaryPath $OutputPath -Force
        }
        finally {
            if (Test-Path $temporaryPath) {
                Remove-Item $temporaryPath -Force -ErrorAction SilentlyContinue
            }
        }
    }
    catch {
        $details = $_ | Out-String
        [Console]::Error.WriteLine($details)
        exit 1
    }
    exit 0
}

function New-AnalyzerSnapshot {
    param($ModulePath, $OutputPath)

    Remove-Item $OutputPath, "$OutputPath.tmp" -Force -ErrorAction SilentlyContinue
    & pwsh -NoProfile -Command $analysis -args $ModulePath, $project, $OutputPath
    if ($LASTEXITCODE -ne 0) {
        Remove-Item $OutputPath, "$OutputPath.tmp" -Force -ErrorAction SilentlyContinue
        throw "PSScriptAnalyzer failed with exit code $LASTEXITCODE. No comparison was made."
    }
}

# What you get today, followed by what you get with this pull request
New-AnalyzerSnapshot PSScriptAnalyzer "$PWD/before.csv"
New-AnalyzerSnapshot $proposed "$PWD/after.csv"

# Anything listed here is a behaviour change worth knowing about
Compare-Object (Import-Csv ./before.csv) (Import-Csv ./after.csv) `
    -Property RuleName, ScriptName, Line, Column, Message

Add -CustomRulePath and -Settings to the Invoke-ScriptAnalyzer call if you use them. Those are the cases I can predict least and would most like to hear about.

The longer script is deliberate. When I tested a simple pipeline against actions-semver-checker using my installed 1.25 version, the baseline printed The term 'Get-Command' is not recognized, exited with code 1, and still wrote 48 findings to before.csv. The proposed build wrote 154. The apparent 106 additions were not behaviour changes; the failed run had dropped those findings. This script collects the complete result before replacing the CSV, deletes stale output, and stops if either analysis fails.

Some differences are expected: a finding may disappear because a member call is no longer resolved as a command, casing advice may stop firing on a function defined by the script, or a new PSUseShouldProcess warning may reveal a previous false negative. If something else changes — a rule stops working, a custom rule throws, a finding moves without a clear reason, or the analyser crashes — please report it on PR #2206. The rule name and a small example are enough; you do not need to diagnose it.

An identical result is useful too. "No change on a 300-file module with four custom rules" is real evidence, and I need more of it.

Being able to suggest a fix to an open source project also means taking responsibility for what the fix does to its users. I can give the maintainers evidence, explain the behaviour changes, and respond when something breaks. I cannot test code I do not have.

Generating code has become cheap. Earning a maintainer's trust has not. I would not want it to.

PR #2206 is open at the time of writing. The benchmark workflow is on the branch, so you can run it yourself — and I would rather you did than take my word for any of the above.