Temporal worker owned resource

[carlydf]

What was changed

New CRD: TemporalWorkerOwnedResource (TWOR)

A new TemporalWorkerOwnedResource CRD that lets users attach arbitrary namespaced Kubernetes resources (HPAs, PDBs, KEDA ScaledObjects, etc.) to a TemporalWorkerDeployment. The controller creates one copy of the attached resource per active worker version, with auto-injection of scaleTargetRef and selector.matchLabels to point at the correct versioned Deployment.

Key behaviors:

• One copy per active Build ID, named {twdName}-{tworName}-{buildID} (uniquely truncated to 47 chars, DNS-safe)
• Auto-injects spec.scaleTargetRef (when explicitly null) to reference the versioned Deployment -> enables per-version HPA autoscaling
• Auto-injects selector.matchLabels (when explicitly null) with the correct per-version labels -> enables per-version PDB targeting, and arbitrary CRDs that use selector.matchLabels to target versioned Deployments
• Applied via Server-Side Apply with a stable per-TWOR field manager
• Owner ref on each resource copy points to the versioned Deployment → k8s GC deletes copies when a version sunsets
• Owner ref on the TWOR itself points to the TWD → GC cascades on TWD deletion
• Apply status written back to TemporalWorkerOwnedResource.status.versions[*] (Applied, Message, BuildID)
• Template variable support: {{ .DeploymentName }}, {{ .TemporalNamespace }} in any string field

Validating Webhook

A TemporalWorkerOwnedResourceValidator webhook enforces:

• apiVersion and kind required; metadata.name/metadata.namespace forbidden (controller sets these)
• Banned resource kinds (Deployment, StatefulSet, Job, Pod, CronJob by default; configurable via BANNED_KINDS env var)
• minReplicas ≠ 0 (required for approximate_task_queue_backlog metric-based autoscaling to work when queue is idle)
• scaleTargetRef and selector.matchLabels must be absent or null (controller owns injection)
• SAR check: requesting user must be able to create/update the embedded resource type
• SAR check: controller service account must be able to create/update the embedded resource type
• workerRef.name is immutable after creation

Helm chart updates

• helm/crds/temporal.io_temporalworkerownedresources.yaml (new CRD manifest)
• helm/templates/webhook.yaml (always-on TWOR ValidatingWebhookConfiguration; TWD webhook now behind webhook.enabled)
• helm/templates/certmanager.yaml (cert-manager Issuer + Certificate for TLS, default enabled)
• helm/templates/manager.yaml (cert volume/port always present; BANNED_KINDS, POD_NAMESPACE, SERVICE_ACCOUNT_NAME env vars)
• helm/templates/rbac.yaml (TWOR + SAR rules in manager ClusterRole; editor/viewer roles; configurable attached-resource RBAC)
• helm/values.yaml (ownedResourceConfig.bannedKinds, ownedResourceConfig.rbac.wildcard, default false, ownedResourceConfig.rbac.rules, default: HPA + PDB)

Integration tests

New integration test subtests added to the existing envtest suite, all running through the shared testTemporalWorkerDeploymentCreation table-test runner:

• TemporalWorkerOwnedResource (7 tests): Deployment owner ref, matchLabels injection, multiple TWORs on same TemporalWorkerDeployment, template variable rendering, multiple active versions, apply failure → Applied:false, SSA idempotency
• Rollout gaps (5 tests): Progressive ramp to Current, ConnectionSpecHash annotation repair, gate input from ConfigMap, gate input from Secret, multiple deprecated versions
• Webhook admission (5 tests, separate Ginkgo suite): Spec rejection, SAR pass, SAR fail (user), SAR fail (controller SA),
  workerRef.name immutability

Why?

HPA autoscaling for versioned Temporal workers requires a separate HPA per worker version, each targeting only that version's Deployment with the correct scaleTargetRef and label selectors. Without this CRD, users have no way to create per-version resources that the controller lifecycle-manages alongside the versioned Deployments.

Checklist

1. Closes Enable CRUD of controller-managed scaling objects (and other custom scalers) #207

2. How was this tested:

   • Full envtest integration test suite: new subtests covering TWOR lifecycle, previously uncovered rollout scenarios, and webhook admission via the real HTTP admission path
   • Unit tests: webhook validator, SSA naming/injection helpers, planner integration
   • All tests pass: KUBEBUILDER_ASSETS=.../bin/k8s/1.27.1-darwin-arm64 go test -tags test_dep ./...

3. Any docs updates needed?

   • docs/test-coverage-analysis.md updated to reflect all newly covered test scenarios*
     • probably will delete this before merging, but leaving it up during PR review
   • docs/twor.md added: concept overview, HPA example with cert-manager setup, RBAC configuration guide

[carlydf]

I will be deleting this!

Also, it is AI generated, and I have only read the top half carefully. Use it for your information during review ("The Controller's Contract" might be an especially helpful section for us all to understand and agree on), but don't spend too much time on the details here. Not intended to be a part of the main repo.

[jaypipes]

@carlydf I realize Claude wrote the majority of this code, so take this review partly as a critique of the style of code that Claude produces :)

[jaypipes]

Temporal namespace or Kubernetes namespace?

[jaypipes]

Should be WorkerDeploymentRef since this refers to the deployment not the worker.

[jaypipes]

The naming of objects in these new files is inconsistent in a couple ways. Some of the resources are prefixed with Temporal while others are not. Other resources are prefixed with TemporalWorkerOwned but actually refer to things that are owned by the TemporalWorkerDeployment, not the Worker.

[jaypipes]

If it's a template, this field really should be called Template :)

[jaypipes]

Allowing template variable interpolation is a dangerous game to get into because you now need to guard on the controller side (as opposed to the API machinery/schema-side) against malformed or malicious content. I'd recommend not supporting template variable interpolation for this first PR.

[jaypipes]

Should DaemonSet be excluded as well? What about Service? At some point, you might find that the exclusion list basically includes a bunch of things and that specifying particular supported resource kinds (plus, perhaps CRDs) is a better approach...

[jaypipes]

Note that you can add cert-manager as a conditional Helm chart dependency.

https://helm.sh/docs/chart_best_practices/dependencies/#conditions-and-tags

[jaypipes]

Why not just have the field manager be a const temporal-worker-controller? The general pattern is for the field manager to be a constant string named after the controller itself.

Examples:

• ResourceClaim controller
• Sample controller

[jaypipes]

I recommend using the unstructured library instead of manually manipulating map[string]interface{}.

For example, things like this:

	if spec, ok := raw["spec"].(map[string]interface{}); ok {
		autoInjectFields(spec, data.DeploymentName, selectorLabels)
	}

can be this:

spec, ok, _ := unstructured.NestedMap(raw, "spec")
if ok {
    autoInjectFields(spec, data.DeploymentName, selectorLabels)
}

things like this:

raw["metadata"] = meta

can be this:

unstructured.SetNestedMap(raw, "metadata", meta)

etc...

[jaypipes]

The unstructured.Unstructured struct already has methods for things like SetNamespace() or SetName(), SetLabels() along with things like SetOwnerReferences(). So you might consider creating obj at the top of this function and using those object methods instead of the manual construction of various fields.