How to Preview and Test a Changing YAML Pipeline on Azure DevOps

Since multi-staging-pipelines with YAML have been introduced a big step has been done. They are declarative and configurable pipelines that can be versioned in a repository. One problem still comes:

Change YAML -> commit -> run -> fail -> Change YAML -> commit -> run -> fail…

Some people getting nerve wrecked because of this. Besides the fact that you have to get used to the YAML syntax of SPACES 😉

Use the REST API Endpoint

To avoid this vicious circle and get fast feedback loops during pipeline configuration, the Azure DevOps product team has released a new API endpoint The endpoint is the following:

POST dev.azure.com/<org>/<project>/_apis/pipelines/<pipelineId>/runs?api-version=5.1-preview

And you need to post the following JSON

{
  "PreviewRun": true,
  "YamlOverride": "# your new YAML here, optionally"
}

Then you basically do a “What-If” preview run without triggering any other endpoint with the pipelines.
It checks

• for correct syntax
• if tasks exist
• correct and existing properties for e.g. tasks properties
• outputs the final YAML, which replaces any shorthand YAML syntax (e.g. the ‘-script’ bash task)

But it can’t (yet) check:

• contents of your variable or if a variable is null
• check if you used correct and existing values for e.g. vmImage, branch name, connections or things like the repository name
• YAML templates. The YAML pipeline must be within one file!

User PowerShell Module VSTeam

If you want you can write your PowerShell, but there is a nice PowerShell module named VSTeams on the PowerShell Gallery which does a lot of automation tasks for you against Azure DevOps. With this module, you have the cmdlet Test-VSTeamYamlPipeline with the following PowerShell snippet.

Test-VSTeamYamlPipeline -PipelineId 29 -FilePath .\azure-pipelines.yml -ProjectName MyProject

So currently you need an existing YAML pipeline to test your changed YAML against. You need to give the ID and either the file path or directly the YAML string.

When you e.g. use the YAML code below against the cmdlet

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  Test: mytest

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |    
    echo "Hello World in a YAML Pipeline" > "$PIPELINE_WORKSPACE/outputfile.txt"
  displayName4: 'Create a test file $(Test)'

You will see that you get the following error

Test-VSTeamYamlPipeline : (Line: 16, Col: 3): Unexpected value 'displayName4'
At line:1 char:1
+ Test-VSTeamYamlPipeline -PipelineId 29 -FilePath .\azure-pipelines.er ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Test-VSTeamYamlPipeline

This tells you that you used the property ‘displayName4’ which is not known by the task ‘script’ (which is a BASH script task). But if you correct this the returned YAML looks like this

trigger:
  branches:
    include:
    - master
variables:
- name: Test
    value: mytest
stages:
- stage: __default
    jobs:
    - job: Job
    pool:
        vmImage: ubuntu-latest
    steps:
    - task: CmdLine@2
        displayName: Run a one-line script
        inputs:
        script: echo Hello, world!
    - task: CmdLine@2
        displayName: Create a test file $(Test)
        inputs:
        script: >
            echo "Hello World in a YAML Pipeline" > "$PIPELINE_WORKSPACE/outputfile.txt"

Conclusion

You are now able to start preview testing your pipelines if they work correctly. You can now even think of running a YAML build pipeline for PRs that check if your YAML code is correct. Which is really nice.

Also published on Medium.