Policy as Code for Federated Kubernetes Clusters

If I run more than one Kubernetes cluster, I need policy in Git, checked in CI, and enforced by admission controls - or drift, risk, and waste creep in fast.

Here’s the short version: I would set one source of truth for policy, split rules into global, cluster, and tenant layers, and roll changes out through GitOps in Audit before Enforce. I would start with a small set of rules that cut risk first: approved registries, no :latest, resource requests and limits, required labels, and non-root containers.

At a glance, the article boils down to this:

• Why it matters: multi-cluster Kubernetes makes security, compliance, data location, and spend harder to control.
• Where policy is enforced: mainly through mutating and validating admission webhooks.
• Engine choice: Gatekeeper suits teams that need Rego and more complex rule logic; Kyverno suits teams that want YAML and fast uptake.
• How to structure rules:
  • Global for fleet rules like Pod Security Standards and audit access
  • Cluster for region, tier, or environment rules
  • Tenant for namespace quotas, limits, and app network rules
• How to roll out: use one Git repo, test in pull requests, deploy with Flux, Argo CD, or Fleet, then move from one namespace to fleet-wide enforcement.
• How to track success: watch violation count, median fix time, change failure rate, waiver rate, and audit evidence.
• How to handle exceptions: keep waivers time-boxed, tied to one workload, with an owner, reason, and expiry date.

A few points stand out. The article recommends keeping new rules in audit mode for at least two weeks before blocking anything. It also sets a hard line on webhook setup: use failurePolicy: Fail only when the policy engine runs in HA with at least three replicas. And for audit work such as NIS2, Git history plus signed policy reports give proof that rules were enforced, not just written down.

If I had to sum it up in one sentence, it would be this: standardise first, test in Git, roll out in stages, and keep exceptions tight and short-lived.

Core building blocks and policy engine choices

Kubernetes controls used for policy enforcement

These are the parts of Kubernetes that turn fleet-wide governance into rules the cluster can act on. Before you choose a policy engine, it helps to know what Kubernetes already has out of the box.

The main piece is the admission controller. It sits between the API server and etcd. Every request for a resource passes through it after authentication and authorisation, but before anything is written to the cluster. That makes it the point where policy gets checked and enforced.

Two webhook types handle that work:

• Mutating admission webhooks change resources on the way in. They can inject sidecars, set default security contexts, or add missing labels.
• Validating admission webhooks reject any object that breaks policy.

RBAC comes earlier in the flow and controls who can do what on which resources. It is your first layer of control. But it cannot enforce content rules such as no privileged containers or images must come from an approved registry. That is where a policy engine comes in.

Namespaces set the scope for most policies. Labels and annotations help policies target workloads, owners, and cost centres.

These controls are the enforcement points. From there, the next job is picking the engine that matches your governance model.

Choosing between OPA Gatekeeper and Kyverno

Pick the engine that helps your team enforce policy the same way across the fleet. The choice between OPA Gatekeeper and Kyverno usually comes down to team skills and how deep your compliance needs go.

Gatekeeper uses Rego, a domain-specific language built for policy. It works well for complex logic, especially when policies need to look across resources. Kyverno takes a more Kubernetes-native route: policies are written in YAML, which your engineers already use for manifests every day.

Put simply, Gatekeeper fits complex, cross-resource policies written in Rego. Kyverno fits teams that want Kubernetes-native YAML and faster adoption.

Some teams split the job: Kyverno for day-to-day guardrails, and Gatekeeper for heavier compliance rules.

Once the engine is set, the next move is simple: codify the policies that cut risk first.

Which policies to codify first

Start with the rules that give you the most risk reduction for the least effort. In practice, that means approved registries, no :latest, resource requests and limits, required labels, and non-root containers.

Block the :latest tag in production. It makes reproducibility and audit trails weaker. Enforce mandatory resource requests and limits for CPU and memory, because one badly set workload can starve nearby services. Add required labels such as team, environment, and cost-center so each workload is tied to an owner and a cost centre. If data residency matters, add region-based placement labels like region=eu early so workloads stay aligned with local data-residency laws. Then set your security baseline with runAsNonRoot and by disallowing privileged containers.

Keep new policies in audit mode for two weeks before you enforce them. That gives you time to spot current violations without breaking production. It also gives teams room to fix issues calmly instead of scrambling during an incident. [2][6][7]

After these controls are in audit mode, move on to rollout and ownership.

Policy as Code: Kyverno and Securing Kubernetes at Scale with Jim Bugwadia

Designing a governance model for federated clusters

Once the policy engine is in place, the next step is simple in theory but easy to muddle in practice: decide who owns each rule layer. A three-layer setup works well here - global, cluster-specific, and tenant-specific. But it only holds together if each layer maps cleanly to ownership and delivery workflows.

Global, cluster-specific and tenant-specific policy layers

The global layer covers mandatory rules that apply across the full fleet. Pod Security Standards set to Restricted, global RBAC for auditors, and fleet-wide ownership and traceability labels sit here. These rules are owned centrally by security, compliance, and platform teams. If an exception is needed, it should go through a controlled override.

The middle layer handles cluster-specific rules. Define these centrally, then target them with labels such as environment, tier, region, or cluster-type. That way, new clusters pick up the right controls by default. No one has to scramble to update policies by hand every time the fleet expands [1][10].

The tenant layer is where application teams manage namespace-level rules. That includes resource quotas, LimitRanges, and app-specific ingress and egress rules. Instead of forking the entire policy set, use a versioned override such as a Kustomize or JSON patch [5][3].

Policy categories mapped to scope and ownership

A policy stack only works when each category has a clear scope and a clear owner. If that line is blurred, policies drift, get copied across layers, or fall into the classic gap where everyone assumes someone else is handling it.

This split lets central teams keep baseline standards tight without slowing down application teams.

Aligning governance with cost and operating models

This model also ties directly to cost control. Enforcing resource requests, limits, and namespace quotas helps stop noisy-neighbour workloads and cuts down on over-provisioning. Mandatory cost-centre and owner labels make chargeback and showback possible. If spend cannot be linked to a team or product, it cannot be allocated with confidence [5][11].

Namespace quotas matter most in development environments, where a bit of testing can quietly turn into a surprisingly high bill.

Once ownership is settled, encode it in Git and roll it out through testing and staged enforcement.

How to implement policy as code across federated Kubernetes clusters

::: @figure {Policy as Code Rollout Process for Federated Kubernetes Clusters} :::

Set up repositories, testing and promotion workflows

Once ownership is clear, the next step is to put the policy workflow into Git and GitOps.

Use one fleet-wide Git repository for policy definitions. If you skip this, clusters drift and start behaving differently over time. A simple pattern works well: map base/ to global policy, then use overlays/ for cluster-level or environment-level changes. That lines up with the global, cluster, and tenant layers in the governance model.

Overlays also give you a clean way to change enforcement by environment. For example, you can set staging to warn and production to deny.

Policy checks should run in pull requests before anything is merged. That way, bad policy changes are caught early instead of after they hit a cluster. Add conftest for Rego or the kyverno CLI to the pull request pipeline so manifests are checked before deployment.

Deploy policy engines and connect them to GitOps

After policies pass CI, deploy them the same way across the fleet.

Policy engines should go out through GitOps, with Git kept as the source of truth. Keep the same global, cluster, and tenant boundaries used in the governance model [2][4]. Turn on reconciliation so manual changes made in a cluster are rolled back on the next sync [4][11].

You can also use variables to switch enforcement by environment. A common setup is staging with warn and production with deny [8].

Roll out enforcement gradually and assign responsibilities

New policies should begin in Audit mode. Leave them there for at least two weeks before moving to enforcement. That gives teams time to spot workloads that would be blocked later.

The rollout path should stay controlled:

single namespace → single cluster (Audit) → single cluster (Enforce) → fleet-wide (Audit) → fleet-wide (Enforce) [2][12][8]

Here’s a clear handoff model for the rollout process:

For production, the admission webhook needs a safe failure setup. Set failurePolicy to Fail only if the policy engine is running in HA with at least three replicas. If not, use Ignore so normal cluster operations are not blocked [2][11][12].

Monitoring, exceptions and continuous improvement

Track violations, drift and audit evidence across clusters

Once enforcement is live, keep a close eye on the fleet. Kyverno’s built-in PolicyReport and ClusterPolicyReport CRDs let you pull violation data together by namespace and cluster [8].

It also helps to pair Gatekeeper’s audit controller with regular drift checks through the Kyverno Reports API. That makes it easier to spot changes that slipped in through manual “break-glass” actions [2].

Track a small set of metrics that tell you what’s happening:

• Violation count per cluster
• Median remediation time
• Change failure rate
• Waiver rate
• Audit evidence

For NIS2 audits, Git logs and signed policy reports show proof of enforcement, not just intent [2]. The target is simple: 100% coverage across all production clusters.

Use these reports to spot cases that need a formal waiver.

Handle waivers and temporary overrides safely

Use a time-boxed exception resource tied to a single workload. In Kyverno, the PolicyException CRD limits the bypass to a named workload through a match block - namespace, name, or label - instead of switching the policy off for everyone [8].

Every exception needs three things: a named owner, a documented reason, and a clear expiry date [9][8]. Keep all exceptions in the same Git repository as the policies.

Review active exceptions every month. If a waiver has expired, renew it or remove it at once. Waiver rate is a useful signal over time. If that ratio starts climbing, teams are likely working around governance instead of following it.

Conclusion: start small, standardise first, then tighten enforcement

Start with one policy engine. Then set up a layered governance model that covers global, cluster, and tenant scopes. After that, codify the rules with the biggest payoff first: privileged containers, missing resource limits, and unlabelled namespaces.

Roll changes out through GitOps in Audit mode for at least two weeks before switching to Enforce [2][8]. Monitor violations in one place, then adjust rules based on what the reports show. The path is straightforward: codify, monitor, remediate, then tighten enforcement.

FAQs