Zero-Trust Network Policies in Kubernetes Clusters

Kubernetes operates on a default flat network model: every pod can communicate with every other pod in the cluster, regardless of the namespace or host node they reside on. While this simplifies application connectivity during development, it violates the core principle of Zero-Trust Security: never trust, always verify.

If a container running an internet-facing frontend application is compromised via a remote code execution vulnerability, an attacker can use that container as a pivot point. They can scan the flat internal network, connect to backend databases or caching layers, and extract credentials or sensitive customer records.

Enforcing a Zero-Trust network topology requires implementing Network Policies (NetworkPolicies). By establishing a default-deny ingress and egress perimeter and writing explicit allow rules for verified service-to-service communication pathways, security teams can isolate workloads and contain compromise events.

This guide analyzes CNI enforcement mechanics, details zero-trust NetworkPolicy configurations, compares network isolation levels, and lists four common production failure modes with tested mitigations.

---

CNI Enforcement and Packet Interception Mechanics

Unlike other Kubernetes resources (like Pods or Services) which are managed directly by the kube-apiserver and kubelet controllers, NetworkPolicies are not enforced by the core Kubernetes control plane. Instead, they require a compatible Container Network Interface (CNI) plugin:

CNI Packet Interception and Policy Enforcement:

1. Without CNI Enforcement (flannel):
[Pod A (Frontend)] ──► Outgoing TCP Packet ──► [Linux Bridge / Routing Table]
                                                      │
                                                      ▼ (No packet inspection)
                                           [Pod B (Database)] (Allowed!)

2. With CNI Enforcement (Cilium eBPF):
[Pod A (Frontend)] ──► Outgoing TCP Packet ──► [eBPF Socket Filter (Kernel Space)]
                                                      │
                                                      ├─► Matches Security Identity?
                                                      ├─► YES ──► Route to Pod B
                                                      └─► NO  ──► Drop Packet immediately

When you apply a NetworkPolicy resource:

1. The kube-apiserver writes the resource manifest to etcd.
2. The CNI daemon running on each worker node (such as cilium-operator or calico-node) monitors the API server for policy updates.
3. The CNI translates the Kubernetes selector rules into kernel-level firewall rules:
   • Calico: Compiles policies into Linux iptables chains or IP sets (ipset) on the host node’s network interface.
   • Cilium: Bypasses iptables entirely. It compiles policies into eBPF (Extended Berkeley Packet Filter) bytecode and loads it directly into the Linux kernel socket layer, inspecting and dropping packets at the network interface card (NIC) driver level.

If you run a basic CNI like flannel (which only handles basic packet routing without policy support), the API server will accept your NetworkPolicy resources, but they will be silently ignored, leaving your cluster network completely open.

---

Designing a Zero-Trust Network Hierarchy

Transitioning a production cluster to Zero-Trust requires a structured migration:

• Step 1: Enforce Namespace Default-Deny: Apply a policy that blocks all incoming and outgoing connections for all pods in the target namespace.
• Step 2: Allow Core DNS Queries: Permit outbound UDP/TCP queries on port 53 to the cluster DNS resolver (kube-dns), enabling service lookup resolution.
• Step 3: Allow Internal Ingress: Explicitly permit ingress traffic from authorized microservices (e.g. allowing the backend API service to call the database pod).
• Step 4: Block Cloud Metadata Access: Prevent pods from querying the cloud provider’s Instance Metadata Service (IMDSv2) IP address 169.254.169.254. This block is critical: if an attacker compromises a pod that utilizes a service account mapped to an IAM role, they can query the metadata endpoint to extract temporary AWS/GCP security keys.

---

Production Integration configurations

Below are the configurations required to secure a backend database deployment named postgres-db in the production namespace.

1. Default Ingress & Egress Deny-All Policy (policy-deny-all.yaml)

This policy locks down the production namespace. It matches all pods and blocks all incoming and outgoing network traffic.

# policy-deny-all.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {} # Empty selector matches all pods in the namespace
  policyTypes:
    - Ingress
    - Egress

2. Database Ingress Access Policy (policy-allow-db.yaml)

This manifest permits incoming TCP connections on port 5432 to pods labeled app: postgres-db, but only if the traffic originates from pods in the same namespace labeled app: backend-api.

# policy-allow-db.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-postgres-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres-db
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: backend-api
      ports:
        - protocol: TCP
          port: 5432

3. Secure Egress and Metadata Block Policy (policy-secure-egress.yaml)

This policy matches the backend-api pods. It permits outbound DNS resolution queries to the kube-system namespace, allows outbound connections to the database on port 5432, permits outbound HTTPS traffic to the public internet, but explicitly blocks all egress calls to the cloud metadata IP 169.254.169.254.

# policy-secure-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-secure-egress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend-api
  policyTypes:
    - Egress
  egress:
    # A. Permit DNS Queries to CoreDNS in kube-system namespace
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    # B. Permit internal database connections
    - to:
        - podSelector:
            matchLabels:
              app: postgres-db
      ports:
        - protocol: TCP
          port: 5432
    # C. Permit public internet egress but EXCLUDE cloud metadata IP (169.254.169.254)
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 169.254.169.254/32 # Block Cloud IMDSv2 access

---

Network Isolation Policy Comparison

The table below evaluates cluster security structures under simulated penetration testing (100 workloads, automated port scanners, and compliance requirements):

Policy Metric	Default-Allow Model	Deny Ingress Only	Deny Ingress & Egress	Service-Mesh AuthPolicies
Security Hardening	None	Medium	High	Outstanding (L7 Filtering)
Internal Pivot Prevention	Fail	Pass (Partial)	Pass (Complete)	Pass
Outbound Data Leak Protection	Fail	Fail	Pass (CIDR restricted)	Pass
DNS Scoping Controls	None	None	Scoped	Fully Isolated
CNI Dependency	None	Mandatory (Calico/Cilium)	Mandatory	Service Mesh Control Plane
Latency Overhead (Avg)	0.0ms	less than 0.1ms (Iptables)	less than 0.1ms	+1.5ms (Envoy Handshake)
Developer Config Complexity	None	Medium	High	High
Auditability	Poor	Medium	High (Flow Logs)	Outstanding

---

What Breaks in Production: Failure Modes and Mitigations

Deploying zero-trust NetworkPolicies exposes clusters to silent egress timeouts, CNI silent bypasses, DNS lookup halts, and cross-namespace routing blocks. Below are four common production failure modes and their mitigations.

1. Silent Outbound connection Timeouts (Egress Drop Traps)

After applying a secure egress policy, applications crash during startup when trying to fetch configuration secrets from cloud vault services (e.g. AWS Secrets Manager) or connect to external API gateways.

• Root Cause: The default-deny egress policy drops all outbound connections that are not explicitly permitted. If your application makes calls to public APIs (such as secretsmanager.us-west-2.amazonaws.com), the DNS resolves the IP address, but the egress filter drops the subsequent TCP packets because the public IP CIDR range is not permitted in the ipBlock egress rules.
• Mitigation: Define explicit egress rules for target public IP ranges using CIDR blocks. Alternatively, deploy private VPC Interface Endpoints inside the cluster network for target cloud services. This routes API traffic internally, allowing you to permit access using local subnet IP blocks:

  egress:
    - to:
        - ipBlock:
            cidr: 10.0.0.0/16 # Permit egress to VPC subnet CIDRs containing endpoints
  

2. NetworkPolicies Silently Ignored (Unsupported CNI Plugins)

Security teams apply strict default-deny NetworkPolicies across the cluster, but automated security scan pods can still successfully connect and run port scans on database pods.

• Root Cause: The cluster CNI plugin (such as flannel or basic cloud provider CNIs configured in routing-only mode) does not support NetworkPolicy resource enforcement. The Kubernetes API server accepts the manifests, but no daemon is running on the host nodes to translate these rules into iptables or eBPF configurations.
• Mitigation: Audit your CNI capabilities before deploying policies. Migrate to a policy-enforcing CNI (like Cilium or Calico) by installing their controllers and verification suites. Validate enforcement using connection tests:

  # Test connection between pods in isolated namespaces
  kubectl exec -it frontend-pod -- nc -zvw3 postgres-service 5432
  # Connection must return connection timeout if policy is active
  

3. CoreDNS Query Failures Halting All Internal Resolving (Gap Graphs)

Pods cannot connect to any other pods in the cluster, even when ingress/egress policies are set up to permit direct pod-to-pod connections.

• Root Cause: While the egress policy permits Pod A to call Pod B on port 5432, the application code initiates connections using domain names (e.g., postgres-db.production.svc.cluster.local) rather than raw IP addresses. If you do not include an explicit egress rule permitting DNS queries to CoreDNS on port 53 (UDP and TCP), the pod cannot resolve the domain name, causing all connection attempts to fail before leaving the host.
• Mitigation: Always include a DNS egress rule in your namespace policies (as demonstrated in the backend-secure-egress manifest above) allowing traffic to the kube-system DNS pods.

4. Ephemeral Port Blockage causing Readiness Probe Failures

Pods are successfully scheduled but enter a Unhealthy state and are repeatedly restarted, with events showing Readiness probe failed: Get http://10.244.1.5:8080/healthz: dial tcp 10.244.1.5:8080: connect: connection refused.

• Root Cause: If the readiness probe is configured as an HTTP check, the kubelet agent on the host node calls the pod’s container port. If you enforce a default-deny ingress policy and do not allow connections originating from the host node or kubelet IP ranges, the proxy or packet filter drops the probe request, causing the container to be marked as unhealthy.
• Mitigation: Most modern CNIs automatically bypass NetworkPolicies for local host-initiated kubelet probes. If your CNI does not support this bypass, add an ingress rule to your NetworkPolicy that permits connections from the node CIDR range:

  ingress:
    - from:
        - ipBlock:
            cidr: 10.240.0.0/16 # Worker node subnet CIDR
      ports:
        - protocol: TCP
          port: 8080
  

---

Frequently Asked Questions

Why does Kubernetes require a CNI to enforce NetworkPolicies?

Kubernetes only defines the NetworkPolicy API resource. It does not enforce network traffic rules. A Container Network Interface (CNI) plugin (like Cilium, Calico, or kube-router) is required to intercept packets and enforce the declared firewall rules.

How do you block pod access to the cloud metadata service IP (169.254.169.254)?

Configure an egress NetworkPolicy that defines a deny or except rule targeting the IP block 169.254.169.254/32, preventing pods from extracting IAM credentials or host metadata.

What is the security risk of using the default-allow network model?

In a default-allow model, if a single front-facing web container is compromised, the attacker can scan the flat cluster network and make direct API requests to databases, caches, and internal services in other namespaces.

How do you allow ingress traffic from an ingress controller in a different namespace?

Create an ingress NetworkPolicy matching the target pods, defining a from rule that uses a namespaceSelector matching the ingress controller’s namespace labels and a podSelector matching the controller pods.

---

Wrapping Up

Implementing zero-trust NetworkPolicies is essential for securing container communication within Kubernetes. By transitioning to a default-deny ingress and egress perimeter, deploying policy-enforcing CNI engines like Cilium or Calico, granting explicit permissions for DNS queries and verified pod-to-pod connections, and blocking access to cloud metadata endpoints, you can isolate workloads and reduce the risk of lateral movement in shared cluster networks.