How to Preview and Test a Changing YAML Pipeline on Azure DevOps

Sebastian Sch├╝tzeSebastian Sch├╝tze

Since multi-staging-pipelines with YAML have been introduced a big step has been done. They are declarative and configurable pipelines that can be versioned in a repository. One problem still comes:

Change YAML -> commit -> run -> fail -> Change YAML -> commit -> run -> fail…

Some people getting nerve wrecked because of this. Besides the fact that you have to get used to the YAML syntax of SPACES ­čśë

Use the REST API Endpoint

To avoid this vicious circle and get fast feedback loops during pipeline configuration, the Azure DevOps product team has released a new API endpoint The endpoint is the following:

POST dev.azure.com/<org>/<project>/_apis/pipelines/<pipelineId>/runs?api-version=5.1-preview

And you need to post the following JSON

  "PreviewRun": true,
  "YamlOverride": "# your new YAML here, optionally"

Then you basically do a “What-If” preview run without triggering any other endpoint with the pipelines.
It checks

But it can’t (yet) check:

User PowerShell Module VSTeam

If you want you can write your PowerShell, but there is a nice PowerShell module named VSTeams on the PowerShell Gallery which does a lot of automation tasks for you against Azure DevOps. With this module, you have the cmdlet Test-VSTeamYamlPipeline with the following PowerShell snippet.

Test-VSTeamYamlPipeline -PipelineId 29 -FilePath .\azure-pipelines.yml -ProjectName MyProject

So currently you need an existing YAML pipeline to test your changed YAML against. You need to give the ID and either the file path or directly the YAML string.

When you e.g. use the YAML code below against the cmdlet

- master

  vmImage: 'ubuntu-latest'

  Test: mytest

- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |    
    echo "Hello World in a YAML Pipeline" > "$PIPELINE_WORKSPACE/outputfile.txt"
  displayName4: 'Create a test file $(Test)'

You will see that you get the following error

Test-VSTeamYamlPipeline : (Line: 16, Col: 3): Unexpected value 'displayName4'
At line:1 char:1
+ Test-VSTeamYamlPipeline -PipelineId 29 -FilePath .\azure-pipelines.er ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Test-VSTeamYamlPipeline

This tells you that you used the property ‘displayName4’ which is not known by the task ‘script’ (which is a BASH script task). But if you correct this the returned YAML looks like this

    - master
- name: Test
    value: mytest
- stage: __default
    - job: Job
        vmImage: ubuntu-latest
    - task: CmdLine@2
        displayName: Run a one-line script
        script: echo Hello, world!
    - task: CmdLine@2
        displayName: Create a test file $(Test)
        script: >
            echo "Hello World in a YAML Pipeline" > "$PIPELINE_WORKSPACE/outputfile.txt"


You are now able to start preview testing your pipelines if they work correctly. You can now even think of running a YAML build pipeline for PRs that check if your YAML code is correct. Which is really nice.

Also published on Medium.

Sebastian is an Azure Nerd with focus on DevOps and Azure DevOps (formerly VSTS) that converted from the big world of SharePoint and O365. He was working with O365 since 2013 and loved it ever since. As his focus shifted in 2017 to more DevOps related topics in the Microsoft Stack. He learned to love the possibilities of automation. Besides writing articles in his blog and German magazines, he is still contributing to the SharePoint Developer Community (and PnP SharePoint) to help to make the ALM part a smoother place to live in.

Comments 2
  • Andrew Stanton
    Posted on

    Andrew Stanton Andrew Stanton

    Reply Author

    I’m still not understanding the fascination with silly-error prone text files when there was a perfectly good graphical designer available that was light years better than each of the prior technologies (XAML, more wordy and difficult XAML, that weird msbuildtfproj build thing). I can drag the usual 5-8 tasks onto a pipeline, leave mostly the defaults and only be a build pipeline config victim if something not usual is required for the build – without touching the source code.

    • Sebastian Sch├╝tze
      Posted on

      Sebastian Sch├╝tze Sebastian Sch├╝tze

      Reply Author

      It is the same as with any UI based GUI. Scripting is usually faster and more flexible without demanding a product team to create complicated and fancy UI.
      BTW. the same arguemtens to copy and paste 7-8 and use their default config works the same for YAML. you configure only properties that differ from the default settings.

      Another powerfully feature is reusability. Create a template once but reise it multiple times cross-team, -project, -organization wide.

      Last but not least: How do you test the correctness of UI configured pipelines without changing them? With YAML you can and they are declarative only! No code logic.

This site uses Akismet to reduce spam. Learn how your comment data is processed.