Skip to content

ADR-012: TLS for Keycloak and Grafana Ingresses

Status

Accepted

Date

2026-03-21

Context

After Phase 3 (ADR-009, ADR-011), Keycloak and Grafana were both reachable over plain HTTP only:

  • http://keycloak.local — OIDC login page, admin console
  • http://grafana.local — Grafana dashboard, OIDC redirect target

This meant that browser-to-service traffic (OIDC tokens, session cookies, admin credentials) crossed the network in plaintext. Even on a local network, this is exploitable via passive sniffing on shared Wi-Fi or a corporate LAN.

The infrastructure to issue certificates was already in place (cert-manager + local-ca-issuer ClusterIssuer from ADR-006) but the ingresses were never configured to use it.

Two related issues were also addressed in the same change:

  1. HTTP still accepted alongside HTTPS — adding a TLS block to an Ingress enables HTTPS but leaves HTTP open unless an explicit redirect is configured.
  2. Hardcoded KEYCLOAK_ADMIN username — the admin username admin was a literal value: in the Deployment manifest rather than sourced from the encrypted secret alongside the password.

Decision

TLS on Both Ingresses

Enable TLS via cert-manager on both the Keycloak and Grafana ingresses using the existing local-ca-issuer ClusterIssuer:

annotations:
  cert-manager.io/cluster-issuer: local-ca-issuer
  nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
    - hosts:
        - keycloak.local  # or grafana.local
      secretName: keycloak-local-tls  # or grafana-local-tls

cert-manager automatically issues a certificate signed by the local CA and stores it in the named Secret. ingress-nginx uses that Secret for TLS termination.

HTTP → HTTPS Redirect

nginx.ingress.kubernetes.io/force-ssl-redirect: "true" is added explicitly to both ingresses. ingress-nginx already redirects HTTP → HTTPS by default when a TLS block is present (ssl-redirect: true), but the explicit annotation makes the intent clear and enforces the redirect even if the controller default is overridden.

Result: any request to http://keycloak.local or http://grafana.local receives a 308 Permanent Redirect to the HTTPS equivalent.

OIDC URL Updates

Grafana's Keycloak integration uses two URL categories with different requirements:

URL Value Reason
auth_url https://keycloak.local/... Browser-facing: user is redirected here, must match the cert
token_url http://keycloak.identity.svc.cluster.local/... Server-to-server inside cluster: HTTP is fine
api_url http://keycloak.identity.svc.cluster.local/... Server-to-server inside cluster: HTTP is fine
root_url https://grafana.local OIDC redirect URI base: must match the ingress scheme

The token and API URLs deliberately stay HTTP on the in-cluster DNS name. These calls never leave the cluster, so encrypting them would add overhead with no security benefit.

Admin Username in Secret

KEYCLOAK_ADMIN was moved from a hardcoded value: admin in the Deployment to a secretKeyRef pointing to keycloak-admin-secret. The key admin-username was added to identity/keycloak-secret.sops.yaml alongside the existing admin-password:

# keycloak-deployment.yaml (after)
- name: KEYCLOAK_ADMIN
  valueFrom:
    secretKeyRef:
      name: keycloak-admin-secret
      key: admin-username

Alternatives Considered

Option Reason Not Chosen
Self-signed cert per ingress (no CA) Browser shows a different warning per hostname; the local CA approach requires trusting once and works for all *.local hostnames
Let's Encrypt / ACME Requires a publicly routable domain and internet access — not applicable for *.local hostnames
Disable HTTP entirely at pod level KC_HTTP_ENABLED: "true" must remain for Grafana's server-to-server token exchange on the in-cluster DNS name; removing it would break OIDC
Wildcard cert for *.local cert-manager supports wildcard certs with DNS-01 challenge; not needed here since we have only two hostnames

Consequences

Positive

  • All browser-facing traffic (tokens, cookies, credentials) is encrypted in transit
  • OIDC redirect URIs are consistent with the https:// scheme
  • force-ssl-redirect ensures HTTP is never accidentally used by clients
  • Admin username is no longer visible in plaintext in git log
  • cert-manager handles certificate renewal automatically

Negative

  • Browsers will show an "untrusted certificate" warning until the local CA is added to the system trust store (expected for a self-signed local CA)
  • kubectl and curl commands referencing keycloak.local or grafana.local need -k / --insecure or the CA cert added to the trust store
  • The local-ca-issuer CA keypair must be restored after minikube delete (same as any cert-manager state)

Trust Store Setup

To avoid browser warnings, add the local CA to your macOS trust store:

# Export the CA cert from the cluster
kubectl get secret local-ca-key-pair -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/homekube-ca.crt

# Trust it system-wide (macOS)
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain /tmp/homekube-ca.crt

After trusting the CA, both https://keycloak.local and https://grafana.local will show a padlock without warnings.