Skip to content

ADR-008: Use Cilium as the CNI and Network Policy Engine

Status

Accepted

Date

2026-03-07

Context

The default Minikube bridge CNI provides basic pod networking but has no network policy support and no observability. A production-grade CNI is needed that supports: - Kubernetes NetworkPolicy enforcement - Layer 7 (HTTP/gRPC-aware) policies - Network flow observability - eBPF-based performance

Decision

Use Cilium (CNCF Graduated) as the CNI plugin, with Hubble enabled for network observability.

Installation approach

Cilium is installed via the cilium CLI before Flux bootstrap. This is a deliberate exception to the GitOps-everything rule, caused by a bootstrapping constraint: without a working CNI, no pods can run — including Flux itself.

NetworkPolicy resources are managed by Flux after bootstrap.

Cluster setup: 

minikube delete
minikube start --driver=docker --network-plugin=cni --addons=metrics-server
cilium install
cilium status --wait
cilium hubble enable --ui

Re-bootstrap Flux (repo already contains all manifests): 

flux bootstrap github \
  --token-auth \
  --owner=MarcSpeckmann \
  --repository=HomeKube \
  --branch=main \
  --path=clusters/local \
  --personal

Alternatives Considered

Tool Reason Not Chosen
Default bridge CNI No NetworkPolicy support; no observability; not representative of production
Calico CNCF project, strong NetworkPolicy support; but iptables-based (not eBPF); no L7 policies; less modern
Flannel Simple overlay; zero NetworkPolicy support
Weave Net Largely unmaintained; not CNCF
minikube --cni=cilium Installs a stripped-down Cilium manifest without Hubble UI or Helm management

Consequences

Positive

  • eBPF-based data plane — faster than iptables, better observability hooks
  • Hubble UI shows real-time network flows between pods (invaluable for learning)
  • Supports both standard NetworkPolicy and extended CiliumNetworkPolicy (L7 HTTP/gRPC rules)
  • CNCF Graduated — production standard at major cloud providers
  • cilium connectivity test validates the entire CNI setup end-to-end

Negative

  • Cilium itself is not managed by Flux (bootstrapping constraint) — upgrades done via cilium upgrade
  • eBPF requires a relatively recent kernel — minikube's kernel is sufficient, but worth noting
  • Higher complexity than Flannel/bridge for initial setup
  • minikube delete required when changing CNI — destroys all cluster state

Bootstrapping Exception (documented)

This is the only component in HomeKube not managed by Flux. The rationale:

Flux needs pods to run → pods need a CNI → CNI must exist before Flux

All NetworkPolicy resources (the output of Cilium) ARE managed by Flux. Future option: use a GitOps-compatible pre-bootstrap hook (e.g. ArgoCD App-of-Apps with init containers) — considered over-engineered for Phase 2.

Trade-offs

eBPF observability and L7 policy capability are prioritised over strict GitOps purity for the CNI installation step.