Skip to content

ADR-017: Kyverno as Policy Engine for Admission Control

Status

Accepted

Date

2026-03-21

Context

As the cluster grows with more workloads, it becomes necessary to enforce baseline standards automatically rather than relying on manual code-review. Three categories of risk have been identified:

  1. Untagged or mutable images — deploying :latest makes rollbacks unreliable and breaks reproducibility.
  2. Missing workload labels — without consistent labels, NetworkPolicies, Prometheus ServiceMonitors, and future Kyverno policies cannot reliably select resources.
  3. Unbounded resource consumption — containers without resources.limits can starve the single-node Minikube host.

A policy engine integrated with the Kubernetes admission webhook is the standard approach for enforcing these rules at creation time.

Decision

Kyverno v3 is selected as the cluster policy engine.

Kyverno is installed as a HelmRelease inside infrastructure/controllers/ so it starts before any downstream workloads. The policies/ Flux Kustomization (depends on infrastructure-controllers) applies three components:

1. Pod Security Standards library (kyverno-policies chart)

The official Kyverno PSS chart deploys a curated set of community-maintained policies that implement the Kubernetes Pod Security Standards. The baseline profile is applied in Audit mode — it covers the most critical misconfigurations (hostPID, hostNetwork, privileged containers, dangerous capabilities) without blocking existing workloads.

2. Policy Reporter UI (policy-reporter chart)

The Kyverno Policy Reporter provides a web dashboard that reads PolicyReport and ClusterPolicyReport objects and visualises compliance status across namespaces. It is exposed at https://polr.local via ingress-nginx with a cert-manager TLS certificate, following the cluster convention (ADR-012).

3. Custom ClusterPolicies

Policy Mode What it checks
disallow-latest-tag Enforce Rejects Pods whose containers use :latest or no tag
require-labels Audit Deployments must have app.kubernetes.io/name
require-resource-limits Audit Containers must set resources.limits.cpu and .memory

Enforce is used for disallow-latest-tag because it is an unambiguously bad practice with no valid exceptions in this cluster. The other two policies start in Audit mode so that existing resources can be brought into compliance incrementally without causing disruption.

Policy violations are recorded as PolicyReport objects (CNCF Policy Working Group standard CRD) and are visible with:

kubectl get polr -A
kubectl describe polr -n <namespace> <name>

The Policy Reporter UI at https://polr.local provides the same data in a browser-friendly dashboard.

Alternatives Considered

OPA / Gatekeeper

OPA Gatekeeper is the other widely-adopted CNCF policy engine. It is more flexible (Rego covers arbitrary logic) but requires learning the Rego language. For the policies needed here — pattern matching on resource fields — Kyverno's YAML-native rules are significantly simpler to read and maintain. Gatekeeper also does not natively produce PolicyReport objects (it uses its own ConstraintViolation CRD), making kubectl get polr -A impossible without additional tooling.

Validating Admission Webhooks (custom)

Writing a custom webhook is the most flexible option but requires a running Go or Python service, TLS certificates, and significant maintenance overhead. It is not appropriate for a personal sandbox cluster.

No policy engine

Accepting the risk and relying on PR review was considered. Given that the cluster is a learning environment designed to mirror production patterns, enforcing standards at admission time is more valuable than convenience.

Consequences

Positive

  • Non-compliant resources are blocked (Enforce) or flagged (Audit) at admission time, before they reach the cluster.
  • PolicyReport objects integrate with the Kubernetes Policy WG ecosystem.
  • The Policy Reporter UI at https://polr.local makes policy compliance browsable without kubectl.
  • The PSS baseline library provides a well-tested community ruleset; custom policies extend it for project-specific standards.
  • Kyverno policies are YAML — the same language used everywhere else in this repo — keeping the tooling surface small.
  • The background: true setting means Kyverno scans already-running resources and flags existing violations.

Negative / Trade-offs

  • Kyverno adds ~4 pods (admission-controller, background-controller, cleanup-controller, reports-controller) plus the Policy Reporter pod — approximately 600 MiB of additional memory overhead on the MacBook.
  • The Enforce policy for disallow-latest-tag will block any direct kubectl apply or Helm install that references :latest. Authors must pin image tags.
  • Upgrading Kyverno major versions can require policy syntax changes.