Is there a tool to validate an Azure DevOps Pipeline locally?

Solution 1:

UPDATE: This functionality was removed in Issue #2479 in Oct, 2019

You can run the Azure DevOps agent locally with its YAML testing feature.

From the microsoft/azure-pipelines-agent project, to install an agent on your local machine.

Then use the docs page on Run local (internal only) to access the feature that is available within the agent.

This should get you very close to the type of feedback you would expect.

Solution 2:

FYI - this feature has been removed in Issue #2479 - remove references to "local run" feature

Hopefully they'll bring it back later considering Github Actions has the ability to run actions locally

Solution 3:

Azure DevOps has provided a run preview api endpoint that takes a yaml override and returns the expanded yaml. I added this support to the AzurePipelinePS powershell module. The command below will execute the pipeline with the id of 01 but with my yaml override and return the expanded yaml pipeline.

Preview - Preview Service: Pipelines API Version: 6.1-preview.1 Queues a dry run of the pipeline and returns an object containing the final yaml.

# AzurePipelinesPS session
$session = 'myAPSessionName'

# Path to my local yaml
$path = ".\extension.yml"    

# The id of an existing pipeline in my project
$id = 01        
        
# The master branch of my repository
$resources = @{              
   repositories = @{
       self = @{
           refName = 'refs/heads/master'
        }
   }
}

Test-APPipelineYaml -Session $session -FullName $path -PipelineId $id -Resources 
$resources

Solution 4:

A pipeline described with YAML, and YAML can be validated if you have a schema with rules on how that YAML file should be composed. It will work as short feedback for the case you described, especially for syntax parsing errors. YAML Schema validation might be available for almost any IDE. So, we need:

YAML Schema - against what we will validate our pipelines
An IDE (VS Code as a popular example) - which will perform validation on the fly
Configure two of the above to work together for the greater good

The schema might be found from many places, for this case, I'll suggest using https://www.schemastore.org/json/ It has Azure Pipelines schema (this schema contains some issues, like different types of values comparing to Microsoft documentation, but still cover the case of invalid syntax)

VS Code will require an additional plug-in to perform YAML text validation, there are also a bunch of those, who can validate schema. I'll suggest try YAML from RedHat (I know, the rating of the plugin is not the best, but it works for the validation and is also configurable)

In the settings of that VS Code plugin, you will see a section about validation (like on screenshot)

YAML Schema validation plugin settings

Now you can add to the settings required schema, even without downloading it to your machine:

    "yaml.schemas": {
    
    "https://raw.githubusercontent.com/microsoft/azure-pipelines-vscode/v1.174.2/service-schema.json" : "/*"
}

Simply save settings and restart your VS Code. You will notice warnings about issues in your YAML Azure DevOps Pipeline files (if there is any). Failed validation for purpose on the screenshot below:

enter image description here

See more details with examples here as well