Scalable Container Based Azure Pipelines Pools with Azure Container Apps

Update 22.02.2022

There is a thread about this approach regarding Container Apps only supporting ScaledObjects and not ScaledJobs. Which is needed to not cancel agent jobs when container apps are scaled down. So for now scaling is not really usable for ACI (Azure Container Apps).

See below

It has been removed from the current public documentation as well but there is a feature request to support scaled jobs in the future. So use this as a basic concept and stay tuned until scaled jobs really work with Azure Pipelines.

Introduction

When Microsoft’s Ignite 2021 in November happened a new serverless container based service has been released. It allows more capabilities and control than container instances but is abstracted away from a full kubernetes cluster (like AKS in Azure).

Additionally, it is concentrating on horizontal scaling (with KEDA scalers, look on GitHub) and microservices best practices to make it easier for developers to handle multiple containers running side by side (with Dapr, look on GitHub).

Note: This blog post assumes that you can read docs (I put all the links) and you know your way around. I also provide all the code at the end of the blog post. That is why I am keeping it short here and explain only the new shit here. 😉

What is a KEDA Scaler?

KEDA scalers are actually listening to a defined metric for a resource that the scaler integrates and then horizontally scales a container template or a template for a group of containers working together (microservices).

And what is a container template?

I just called it like that, because the ARM API is defining it like that for the Container Apps. But basically, it is a blueprint which says, which docker images with what compute resource and which scaler it should run and scale.

KEDA scalers can scale with anything that is measurable like CPU and memory usage. Or Azure storage queues. Or even based on MSSQL query results. It is basically up to the user or someone who is implementing a scaler.

KEDA Azure Pipelines scaler

These two beasts, KEDA and Dapr, actually is a very good integration to make using and deploying container so easy it is unbelievable. Especially, one specific KEDA scaler is very interesting from Azure DevOps (and my) perspective: the Azure Pipelines scaler.

The scaler itself is written in Go and can be used to scale container based agents in one agent pool. This means you can spin up new container instances based on the queue length of waiting jobs in a pool.

There is one simple parameter to fine grain the scalability: How many jobs should be waiting before a new container is created?

Documentation says:

Example – If one pod can handle 10 jobs, set the queue length target to 10. If the actual number of jobs in the queue is 30, the scaler scales to 3 pods.

KEDA | Azure Pipelines

Azure Container Apps (Preview) and Azure Pipelines scaler

At the time I am writing this the service is in preview and maybe out for about three weeks.

Creating a container app is fairly easy either with the Azure portal or with the azure cli. My experience was that using an ARM template was the easiest approach (because any abstraction layer can only create what ARM supports…). But the documentation itself is (nature of previews) very, lets say spartan.

You find only scaling examples for CPU and memory. But I added a PR to improve the Azure docs for this scaler.

Let’s get to it. The scaler is defined as follow in an ARM template

"scale": {
    "minReplicas": "1",
    "maxReplicas": "5",
    "rules": [
        {
            "name": "azdo-agent-scaled",
            "custom": {
                "type": "azure-pipelines",
                "metadata": {
                    "poolID": "[parameters('azp_poolId')]",
                    "targetPipelinesQueueLength": "1"
                },
                "auth": [
                    {
                        "secretRef": "azp-token",
                        "triggerParameter": "personalAccessToken"
                    },
                    {
                        "secretRef": "azp-url",
                        "triggerParameter": "organizationURL"
                    }
                ]
            }
        }
    ]
}

I will give you just the format, because parameters like poolID, targetPipelinesQueueLength, personalAccessToken, organizationURL are explained on the KEDA docs pages. Look at the end of the article for my full code in my GitHub Repo.

Agent Container Image

There is not much to explain. Just so you don’t ask where it comes from. I got the base image from the docs. It is very basic and only good for this demonstration.

Use it. It gets the job done.

Additionally you want to save the image to an Azure Container Registry. You can also find documentation on how to create one and push your agent image.

Resources on Azure DevOps Services

Here is also not much needed. You need basically:

• An Azure DevOps organization
• An agent pool
• a PAT to save it to your container app for the container agents.

Full Code Sample

You can find a full code on GitHub for:

• creating the basic agent image
• pushing it to your existing ACR
• creating an Agent Pool
• deploying the Container App with a connected Analytics Workspace

https://github.com/RazorSPoint/azure-pipeline-agent-scaler

Limitations and wishes

So there are some limitations.

• ScaledJobs are not supported in ACA. But are needed so that jobs get not cancelled when scaled down
• You can scale to zero but there is a problem with Azure DevOps Pools which is explained in a KEDA blog post and how to work around it
• The container app says you can scale up to 25 (the tool tip says it somewhere on the Azure Portal) but 10 is the current actual limit
• KEDA and Dapr exception are currently not surfaced to the Analytics Workspace. So if you mistype you never now it because provision of the container can still be successful
• KeyVault reference syntax is not working and it will hopefully in the future
• VNet integration does not exist but will come soon

Also published on Medium.