Review Pull Requests on Azure DevOps¶

lgtmaybe posts reviews to Azure DevOps Repos as well as GitHub, GitLab and Gitea. The review itself is identical, because none of it depends on the host. It runs the same lenses, the same reflection pass, and produces the same findings. Only the adapter that reads the change and posts the comments differs.

Azure DevOps Services is supported: dev.azure.com/{organisation} and the legacy {organisation}.visualstudio.com. Azure DevOps Server (on-premises) is not supported yet.

Set It Up¶

Azure Pipelines runs a pull request build from the YAML in the pull request itself. So the job that holds the model key must not live in the repository being reviewed. The setup below keeps it in a template in a separate, protected repository, and the reviewed repository only extends it. Read Security: who can change the pipeline before choosing anything simpler.

Create a protected template repository. For example Platform/pipeline-templates. Add branch policies to its main branch, so nobody can change it without a review from someone trusted with the secrets. Commit examples/azure-devops/lgtmaybe-review.yml to it.
Store the model key in a variable group. In Pipelines → Library, create a variable group named lgtmaybe-secrets holding the provider key as a secret, for example ANTHROPIC_API_KEY. Use a key with a spend limit.
Require the template. On that variable group, open Approvals and checks, add a Required template check, and name the template repository, refs/heads/main and lgtmaybe-review.yml. A pipeline that does not extend exactly that template is refused the key, whatever its YAML says.
Let the pipeline comment. lgtmaybe posts with the job's own token, System.AccessToken. In Project settings → Repositories → Security, select the project's build service identity (<Project> Build Service (<Organisation>)) on the reviewed repository. Allow Contribute to pull requests, and nothing broader.
Add the pipeline to the reviewed repository. Copy examples/azure-devops/azure-pipelines.yml, and point its resources.repositories entry at your template repository. Create a pipeline from it. Approve its access to the template repository and the variable group on the first run.
Run it on pull requests. Azure Repos starts pull request builds from a branch policy, not from a pr: trigger. In Project settings → Repositories → Policies, open the target branch and add the pipeline under Build validation.

The reviewed repository's pipeline holds no steps and no secrets:

trigger: none

resources:
  repositories:
    - repository: trusted
      type: git
      name: Platform/pipeline-templates
      ref: refs/heads/main

extends:
  template: lgtmaybe-review.yml@trusted

The template in the protected repository runs the GHCR image with the azure-pipelines command:

jobs:
  - job: lgtmaybe
    pool:
      vmImage: ubuntu-latest
    variables:
      - group: lgtmaybe-secrets
    steps:
      - checkout: none

      - script: |
          docker run --rm \
            -e SYSTEM_COLLECTIONURI -e SYSTEM_TEAMPROJECT \
            -e BUILD_REPOSITORY_NAME -e BUILD_REPOSITORY_PROVIDER \
            -e SYSTEM_PULLREQUEST_PULLREQUESTID \
            -e SYSTEM_ACCESSTOKEN -e ANTHROPIC_API_KEY \
            ghcr.io/mattjcoles/lgtmaybe:v2 \
            azure-pipelines --provider anthropic --model claude-sonnet-4-6
        displayName: lgtmaybe review
        env:
          SYSTEM_ACCESSTOKEN: $(System.AccessToken)
          ANTHROPIC_API_KEY: $(ANTHROPIC_API_KEY)

Secret variables, System.AccessToken included, reach a script only when they are mapped under env:. That is why both are listed there. The template takes no parameters on purpose: a parameter would let the pull request's YAML pass steps into the job that holds the secrets.

azure-pipelines finds the pull request from these predefined variables:

SYSTEM_COLLECTIONURI;
SYSTEM_TEAMPROJECT;
BUILD_REPOSITORY_NAME;
SYSTEM_PULLREQUEST_PULLREQUESTID.

A build that is not validating a pull request has no id. There, lgtmaybe fails with an error naming the missing variable rather than guessing.

A pipeline building a GitHub repository is refused too. Those pull requests live on GitHub, so use the GitHub Action.

Azure Pipelines always sets BUILD_REPOSITORY_PROVIDER. When it is unset (running the command by hand), lgtmaybe treats the repository as Azure Repos.

Configure the review with the command's flags: --provider, --model, --fallback-model, --api-key, --api-base and --config.

Without Docker¶

On a self-hosted agent without Docker, install the CLI instead. These steps replace the steps: of the protected template, never the reviewed repository's pipeline:

    steps:
      - checkout: none
      - task: UsePythonVersion@0
        inputs:
          versionSpec: "3.12"
      - script: |
          pip install lgtmaybe
          lgtmaybe azure-pipelines --provider anthropic --model claude-sonnet-4-6
        displayName: lgtmaybe review
        env:
          SYSTEM_ACCESSTOKEN: $(System.AccessToken)
          ANTHROPIC_API_KEY: $(ANTHROPIC_API_KEY)

Tokens¶

lgtmaybe reads the Azure DevOps token from one of two variables:

Variable	What it is	Sent as
`AZURE_DEVOPS_TOKEN`	A personal access token with Code (Read & write) scope	Basic auth, empty user name
`SYSTEM_ACCESSTOKEN`	The job's own token, mapped from `$(System.AccessToken)`	Bearer token

When both are set, AZURE_DEVOPS_TOKEN wins. Use a personal access token when the comments should come from a named account rather than the build service. Either way, the token travels only in the Authorization header and is never logged.

Try It Locally First¶

Reviewing a pull request by URL happens in the pipeline, where the token and the pull request live. Locally, lgtmaybe review reviews the branch you have checked out against the remote primary branch. It prints the findings instead of posting them:

git switch my-feature-branch
export ANTHROPIC_API_KEY=...
lgtmaybe review --provider anthropic --model claude-sonnet-4-6

How It Differs from GitHub¶

Azure DevOps serves no unified diff (the standard text format for a change), so lgtmaybe builds the diff itself:

It reads the pull request's latest iteration (Azure DevOps's name for one version of the pull request: one when it opens, then one per push).
It lists the files that iteration changes against the merge base.
It fetches each file's text before and after.
It builds the diff from the two.

Its limits and failure handling:

Lockfiles, vendored and binary files are listed as changed but never fetched.
A file side over 1 MB is treated like a binary.
A pull request changing more than 300 files is not reviewed. lgtmaybe says so instead of reviewing part of it.
When Azure DevOps reports no merge base for the pull request, the same applies: lgtmaybe says so and does not review it.
lgtmaybe retries a file Azure DevOps keeps throttling or failing to return. If it still fails, lgtmaybe leaves that file out on its own and names it under Files not reviewed in the summary.
A 401 or 403 fails the review, since that means the token is wrong.

Each finding is its own comment thread on the changed line. lgtmaybe leaves it active, so a "resolve all comments" branch policy can hold the merge until someone deals with it. The summary is a general thread that is edited in place on each run. It is created closed, so it never holds a merge on its own. lgtmaybe does not post a finding again when its thread is already open, or marked Won't fix or Closed. It does post a finding again when its thread was marked Resolved and the finding comes back.

On a later run, lgtmaybe marks a finding thread it opened Resolved, with a ✅ Looks resolved. reply. It does this when the finding is no longer reported and the line it was posted on has changed since. Azure DevOps has no "this code moved" flag. So lgtmaybe records the iteration each thread was posted at, and compares the file then with the file now.

Threads anyone else opened are never touched.
Nothing is resolved after a run that failed or finished only partly.
The token needs the same Contribute to pull requests permission it already uses to post.
Set resolve_fixed: false in .lgtmaybe.yml to resolve threads by hand.
A thread posted by an earlier lgtmaybe version may carry no iteration. One that doesn't stays open for a person to resolve.

These are not available yet:

Not available	Why	What happens instead
Incremental review	Not built yet	Every run is a full review
Slash commands (`/review`, `/ask`, …)	`azure-pipelines` reviews the pull request; nothing routes comments to it	Queue the build again to review again
Change overview comment	`azure-pipelines` runs the review only	No description or diagram thread is posted
PR labels and the `fail_on` merge gate	Not built yet in the Azure DevOps adapter	No labels or pull request status are set
👎 feedback (`learn_feedback`)	Not built yet	A downvoted finding can be posted again on a later run
Symbol resolution	No base-branch clone in the Azure DevOps adapter	Reflection judges cross-file findings without the symbol's definition
One-click suggestions	Azure DevOps has no suggestion blocks	A suggested fix shows as a code block
Azure DevOps Server	Only Azure DevOps Services URLs are recognised	The URL is refused with an error naming Azure DevOps Server

Keyless model auth (Bedrock, Vertex, Azure OpenAI through GitHub OIDC) is built into the GitHub composite action, not the azure-pipelines command. Use an API key, or ollama against a model on the agent.

Security: Who Can Change the Pipeline¶

A pull request build in Azure Pipelines runs the YAML from the pull request's own merge result. This is the opposite of GitHub's pull_request_target, where the base branch's workflow runs. So anyone who can push a branch and open a pull request can rewrite a pipeline file in that repository. They can keep its variable mappings, and send every secret those steps can read to a server of their choosing. checkout: none does not help: the YAML itself is the part the pull request controls.

So a pipeline file in the reviewed repository that maps a secret is only safe when everyone who can open a pull request there is already trusted with that secret. Otherwise use the setup above:

The secret-bearing job lives in a template in a separate repository with branch policies, and the reviewed repository only extends it.
The model key sits in a variable group guarded by a Required template check, so a pipeline that drops the extends, or points it at another branch or file, is refused the key.
Builds of pull requests from forks get no secrets by default. Keep that setting off.
Grant the build service only Contribute to pull requests on the reviewed repository. Keep Limit job authorization scope to current project turned on. Any pipeline can map its own System.AccessToken, so its permissions are what a rewritten pipeline could use.
Prefer System.AccessToken over a personal access token. If you must use a personal access token, give it only Code (Read & write) on this organisation, a short expiry, and keep it in the protected variable group.
Use a model key with a spend limit, so a leaked key has a ceiling.

lgtmaybe itself never checks out or executes pull request code. It reads the change through the API and treats every byte of it as untrusted input.