skip to content
System Design in the Smoking Room

Scaling Temporal workers with Keda and why it might not work

/ 2 min read

k8s, guide
Table of contents

TLDR

Temporal should be v1.27+ and quadruple check queue names.

Terminal window
temporal worker deployment list

should return

Name  CreateTime  CurrentVersionDeploymentName  CurrentVersionBuildID  RampingVersionDeploymentName  RampingVersionBuildID  RampingVersionPercentage

Temporal + Keda = ❤️

If you ended up here then you must love Temporal or your annoying coworker persuaded everyone around you to finally ditch Kafka with all its dead letter queues and yada yada.

Or you can be like me who spent almost a week of constant debugging bashing your head against the terminal questioning your own sanity because your pods were not scaling. Even though there were like 11k workflows waiting to be processed.

Without further ado, let’s dive in.

Setup

Usual Keda installation with Helm. At the time of writing the version was 2.20.1. According to the docs it was a more than enough version to use with native Temporal queue metrics.

Simple ScaledObject to go with my deployment (almost entirely built from keda-temporal example):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: some-worker-that-needed-some-help
  namespace: my-favourite-namespace
spec:
  scaleTargetRef:
    name: some-worker-that-needed-some-help
  minReplicaCount: 1
  maxReplicaCount: 3
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 30
          policies:
          - type: Pods
            value: 1
            periodSeconds: 60
        scaleDown:
          stabilizationWindowSeconds: 60
          policies:
          - type: Pods
            value: 1
            periodSeconds: 60
  triggers:
    - type: temporal
      metadata:
        namespace: default
        taskQueue: my-queue
        targetQueueSize: "10"
        endpoint: temporal-frontend.temporal:7233

Like nothing special, just the simplest example to scale from one when there are some spikes in workflow creation.

But no, it’s not working. At some point I was questioning whether my QA even saw it working (sorry for doubting you), because it was working properly on dev, on stage but prod had a really old Temporal installation. Like really old. Like DevOps from the one before last deployed it and it has been working tirelessly ever since. And not even one article or docs page said that Keda relies on some API that only appeared in v1.27

And it is pretty easy to check:

Terminal window
temporal worker deployment list

Proper version should return something like this:

Name  CreateTime  CurrentVersionDeploymentName  CurrentVersionBuildID  RampingVersionDeploymentName  RampingVersionBuildID  RampingVersionPercentage

Even though we don’t have any, we can see that the API is working and at least returning something. Otherwise you will get something like:

Error: unknown method ListWorkerDeployments for service temporal.api.workflowservice.v1.WorkflowService

And it is never stated anywhere. And nobody will return all that wasted time. I’m writing this to help a lost soul, should they encounter the same problem on their path.

After that you can check that whenever there are newer workflows in Temporal either the ScaledObject changes its status or you can query the related HPA resource (k get hpa -w) and see that it changes its values.

And please, for the love of God, if you have multiple environments, synchronize all versions between them.

Take care of yourself and your loved ones.