Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,13 @@ kubectl -n openshell get secret \
openshell-jwt-keys
```

When `server.tls.clientCaSecretName=""`, the chart intentionally omits
`client_ca_path` and the `tls-client-ca` mount, even with built-in PKI or
cert-manager. That is expected; do not treat a missing `tls-client-ca` pod
mount as a defect (`openshell-server-client-ca` may still exist from PKI).
User auth is OIDC or trusted proxy (`server.auth.allowUnauthenticatedUsers=true`);
supervisor transport still uses `openshell-client-tls`.

In cert-manager installs, `certManager.enabled=true` makes cert-manager own TLS
generation. The Helm chart should still render the `openshell-certgen`
pre-install/pre-upgrade hook in JWT-only mode to create `openshell-jwt-keys`,
Expand Down Expand Up @@ -350,6 +357,7 @@ openshell logs <sandbox-name>
| `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled |
| HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://<host>` |
| HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https://<host> --oidc-issuer <url>`, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster |
| Browser `ERR_BAD_SSL_CLIENT_AUTH_CERT` or gateway logs show client cert verification when OIDC or direct HTTPS is expected | Listener client-CA verification still enabled (`clientCaSecretName` unset or `client_ca_path` in ConfigMap) | Set `server.tls.clientCaSecretName=""`, upgrade chart, confirm ConfigMap omits `client_ca_path` |

## Reporting

Expand Down
2 changes: 1 addition & 1 deletion deploy/helm/openshell/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files.
| server.sandboxJwt.ttlSecs | int | `3600` | Token TTL in seconds. Defaults to 3600 (1h). |
| server.sandboxNamespace | string | `""` | Namespace where sandbox pods are created. Defaults to the Helm release namespace (.Release.Namespace) when left empty. |
| server.tls.certSecretName | string | `"openshell-server-tls"` | K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. |
| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). |
| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). Do not set to null; omit the key to use the default secret name above. |
| server.tls.clientTlsSecretName | string | `"openshell-client-tls"` | K8s secret mounted into sandbox pods for mTLS to the server. |
| server.workspaceDefaultStorageSize | string | `""` | Default storage size for the workspace PVC in sandbox pods. Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). Empty = built-in default (2Gi). |
| service.healthPort | int | `8081` | Gateway health service port. |
Expand Down
4 changes: 2 additions & 2 deletions deploy/helm/openshell/templates/_gateway-workload.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ spec:
- name: tls-cert
mountPath: /etc/openshell-tls/server
readOnly: true
{{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }}
{{- if eq (include "openshell.gatewayClientCaEnabled" .) "true" }}
- name: tls-client-ca
mountPath: /etc/openshell-tls/client-ca
readOnly: true
Expand Down Expand Up @@ -144,7 +144,7 @@ spec:
- name: tls-cert
secret:
secretName: {{ .Values.server.tls.certSecretName }}
{{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }}
{{- if eq (include "openshell.gatewayClientCaEnabled" .) "true" }}
- name: tls-client-ca
secret:
{{- if or (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }}
Expand Down
17 changes: 17 additions & 0 deletions deploy/helm/openshell/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,20 @@ so a released chart automatically pulls the matching image without extra overrid
ghcr.io/nvidia/openshell/supervisor
{{- end }}

{{/*
Whether the gateway listener should verify client certificates (mTLS).
An explicit empty server.tls.clientCaSecretName disables client-CA wiring in
both gateway.toml and the workload, overriding built-in PKI and cert-manager
defaults.
*/}}
{{- define "openshell.gatewayClientCaEnabled" -}}
{{- if .Values.server.disableTls -}}
{{- else if eq .Values.server.tls.clientCaSecretName "" -}}
{{- else if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) -}}
true
{{- end -}}
{{- end -}}

{{/*
Whether Helm must propagate a supervisor image override into gateway.toml.
The chart's documented repository and empty tag are the gateway-owned default.
Expand Down Expand Up @@ -213,4 +227,7 @@ Validate chart values that Helm would otherwise accept silently.
{{- if and (eq $workloadKind "statefulset") (gt $replicaCount 1) (not (get $workload "allowMultiReplicaStatefulSet" | default false)) -}}
{{- fail "replicaCount > 1 with workload.kind=statefulset requires workload.allowMultiReplicaStatefulSet=true; use workload.kind=deployment for external database-backed multi-replica gateways." -}}
{{- end -}}
{{- if kindIs "invalid" .Values.server.tls.clientCaSecretName -}}
{{- fail "server.tls.clientCaSecretName cannot be null; omit the key to use the chart default (openshell-server-client-ca), or set to \"\" to disable client certificate verification for HTTPS-only mode" -}}
{{- end -}}
{{- end }}
2 changes: 2 additions & 0 deletions deploy/helm/openshell/templates/gateway-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,10 @@ data:
[openshell.gateway.tls]
cert_path = "/etc/openshell-tls/server/tls.crt"
key_path = "/etc/openshell-tls/server/tls.key"
{{- if eq (include "openshell.gatewayClientCaEnabled" .) "true" }}
client_ca_path = "/etc/openshell-tls/client-ca/ca.crt"
{{- end }}
{{- end }}

{{- if .Values.server.auth.allowUnauthenticatedUsers }}

Expand Down
74 changes: 74 additions & 0 deletions deploy/helm/openshell/tests/gateway_config_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,72 @@ tests:
path: data["gateway.toml"]
pattern: '\[openshell\.gateway\.tls\]'

- it: renders client_ca_path for built-in PKI by default
template: templates/gateway-config.yaml
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: 'client_ca_path\s*=\s*"/etc/openshell-tls/client-ca/ca\.crt"'

- it: omits client_ca_path when clientCaSecretName is empty for HTTPS-only mode
template: templates/gateway-config.yaml
set:
server.tls.clientCaSecretName: ""
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: '\[openshell\.gateway\.tls\]'
- matchRegex:
path: data["gateway.toml"]
pattern: 'cert_path\s*=\s*"/etc/openshell-tls/server/tls\.crt"'
- matchRegex:
path: data["gateway.toml"]
pattern: 'key_path\s*=\s*"/etc/openshell-tls/server/tls\.key"'
- notMatchRegex:
path: data["gateway.toml"]
pattern: 'client_ca_path\s*='

- it: omits client_ca_path for cert-manager shared CA when clientCaSecretName is empty
template: templates/gateway-config.yaml
set:
certManager.enabled: true
certManager.clientCaFromServerTlsSecret: true
server.tls.clientCaSecretName: ""
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: '\[openshell\.gateway\.tls\]'
- matchRegex:
path: data["gateway.toml"]
pattern: 'cert_path\s*=\s*"/etc/openshell-tls/server/tls\.crt"'
- matchRegex:
path: data["gateway.toml"]
pattern: 'key_path\s*=\s*"/etc/openshell-tls/server/tls\.key"'
- notMatchRegex:
path: data["gateway.toml"]
pattern: 'client_ca_path\s*='

- it: omits client_ca_path when cert-manager owns TLS and no client CA secret is set
template: templates/gateway-config.yaml
set:
certManager.enabled: true
certManager.clientCaFromServerTlsSecret: false
pkiInitJob.enabled: true
server.tls.clientCaSecretName: ""
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: '\[openshell\.gateway\.tls\]'
- matchRegex:
path: data["gateway.toml"]
pattern: 'cert_path\s*=\s*"/etc/openshell-tls/server/tls\.crt"'
- matchRegex:
path: data["gateway.toml"]
pattern: 'key_path\s*=\s*"/etc/openshell-tls/server/tls\.key"'
- notMatchRegex:
path: data["gateway.toml"]
pattern: 'client_ca_path\s*='

- it: renders server_sans from certManager.serverDnsNames
set:
certManager.enabled: true
Expand All @@ -328,6 +394,14 @@ tests:
path: spec.template.spec.containers[0].args
content: "sqlite:/var/openshell/openshell.db"

- it: fails when clientCaSecretName is null
template: templates/statefulset.yaml
set:
server.tls.clientCaSecretName: null
asserts:
- failedTemplate:
errorMessage: "server.tls.clientCaSecretName cannot be null; omit the key to use the chart default (openshell-server-client-ca), or set to \"\" to disable client certificate verification for HTTPS-only mode"

- it: fails when legacy postgres.enabled is set
template: templates/statefulset.yaml
set:
Expand Down
36 changes: 36 additions & 0 deletions deploy/helm/openshell/tests/statefulset_client_ca_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,39 @@ tests:
name: tls-client-ca
mountPath: /etc/openshell-tls/client-ca
readOnly: true

# Explicit HTTPS-only opt-out must suppress built-in PKI client-CA wiring even
# when pkiInitJob remains enabled (the default install path from #2095).
- it: omits the client CA volume for built-in PKI when clientCaSecretName is empty
template: templates/statefulset.yaml
set:
pkiInitJob.enabled: true
certManager.enabled: false
server.tls.clientCaSecretName: ""
asserts:
- lengthEqual:
path: spec.template.spec.volumes
count: 3
- notContains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: tls-client-ca
mountPath: /etc/openshell-tls/client-ca
readOnly: true

- it: omits the client CA volume when cert-manager shares server CA and clientCaSecretName is empty
template: templates/statefulset.yaml
set:
certManager.enabled: true
certManager.clientCaFromServerTlsSecret: true
server.tls.clientCaSecretName: ""
asserts:
- lengthEqual:
path: spec.template.spec.volumes
count: 3
- notContains:
path: spec.template.spec.containers[0].volumeMounts
content:
name: tls-client-ca
mountPath: /etc/openshell-tls/client-ca
readOnly: true
1 change: 1 addition & 0 deletions deploy/helm/openshell/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,7 @@ server:
certSecretName: openshell-server-tls
# -- K8s secret with ca.crt for client certificate verification (mTLS).
# Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead).
# Do not set to null; omit the key to use the default secret name above.
clientCaSecretName: openshell-server-client-ca
# -- K8s secret mounted into sandbox pods for mTLS to the server.
clientTlsSecretName: openshell-client-tls
Expand Down
8 changes: 7 additions & 1 deletion docs/kubernetes/access-control.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,12 @@ helm upgrade openshell \
--version <version> \
--namespace openshell \
--set server.oidc.issuer=https://your-idp.example.com/realms/openshell \
--set server.oidc.audience=openshell-cli
--set server.oidc.audience=openshell-cli \
--set server.tls.clientCaSecretName=""
```

Set `server.tls.clientCaSecretName=""` when the gateway terminates TLS directly and browsers or CLI clients connect without client certificates. The chart omits `client_ca_path` from `gateway.toml` and does not mount the client-CA volume, leaving HTTPS-only transport with OIDC for user authentication. Do not set the value to `null`; omit the key to use the chart default, or set it to `""` to disable client certificate verification.

The `audience` value must match the client ID configured in your identity provider for the OpenShell resource server.

### OIDC values reference
Expand Down Expand Up @@ -67,6 +70,7 @@ helm upgrade openshell \
--namespace openshell \
--set server.oidc.issuer=https://your-idp.example.com/realms/openshell \
--set server.oidc.audience=openshell-cli \
--set server.tls.clientCaSecretName="" \
--set server.oidc.rolesClaim=realm_access.roles \
--set server.oidc.adminRole=openshell-admin \
--set server.oidc.userRole=openshell-user
Expand Down Expand Up @@ -96,6 +100,8 @@ helm upgrade openshell \

The gateway still serves TLS and sandbox supervisors still authenticate with gateway-minted sandbox JWTs. User-facing CLI/API calls without OIDC or mTLS credentials are accepted as an unauthenticated local developer principal. The proxy is responsible for authenticating callers and forwarding only authorized traffic.

When the gateway terminates TLS directly and callers connect without client certificates, also set `server.tls.clientCaSecretName=""` as described in the OIDC section above.

To also disable TLS entirely (when the proxy terminates TLS before the request reaches the gateway):

```shell
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/gateway-config.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ grpc_rate_limit_requests = 120
grpc_rate_limit_window_seconds = 60

# Gateway listener TLS (distinct from the per-driver guest_tls_*).
# client_ca_path is optional; omit it for HTTPS-only listeners that do not
# verify client certificates.
[openshell.gateway.tls]
cert_path = "/etc/openshell/certs/gateway.pem"
key_path = "/etc/openshell/certs/gateway-key.pem"
Expand Down
Loading