Calico Network Policies: Pod Labels as Smarter’s Network Boundary

Smarter uses Calico to enforce Kubernetes NetworkPolicies that start its production namespace from deny-all, in both directions, and then admit traffic by pod label. MariaDB and Redis accept connections only from pods labelled app.kubernetes.io/application-group=smarter, the app accepts web traffic only from the Traefik ingress controller, and a pod without the label can’t send a packet to anything. This post describes those policies, as smarter-infrastructure defines them in Terraform, and what it takes to live with them.

Without network policies, a Kubernetes cluster is flat. Any pod in any namespace can open a connection to any other pod’s IP address on any port. MariaDB’s password would otherwise be the only thing between the database and every workload in the cluster, such as a compromised monitoring agent, a test deployment someone forgot, a WordPress plugin in another namespace. Security groups don’t help, because they work at the level of nodes and network interfaces, and every one of those pods shares the same nodes. Network policies are the layer that sees pods.

Who may talk to whom

Every row is a rule in aws/terraform/kubernetes/network_policy.tf. The sections below go through them.

Calico, behind one switch

A Kubernetes NetworkPolicy does nothing unless the cluster’s network plugin enforces it. smarter-infrastructure installs Calico with the Tigera operator’s Helm chart, and opens the two node-to-node ports that Calico needs in the node security group, VXLAN on UDP 4789 and the kubelet on TCP 10250. Every resource in the file, from the Calico release to the last policy, has the same first line.

count = var.enable_enhanced_security ? 1 : 0

That variable comes from one line in aws/stack/stack.hcl.

enable_enhanced_security  = true

The same switch also replaces the EKS module’s recommended node security group rules with Smarter’s narrower ones. So an environment either has the whole set or none of it, and the choice is visible in one place. That matters for stability as much as for security. When a policy turns out to block something the platform needs, turning the whole layer off is one change, reviewed like any other.

Start from nothing

The first policy selects every pod in the namespace and allows nothing.

resource "kubernetes_network_policy_v1" "default_deny_all" {
  metadata {
    name      = "default-deny-all"
    namespace = var.namespace
  }

  spec {
    pod_selector {}
    policy_types = ["Ingress", "Egress"]
  }
}

An empty pod_selector matches every pod, and listing a policy type without any rules denies all traffic of that type. Kubernetes NetworkPolicies are additive. A connection is allowed if any policy that selects the pod allows it, and denied if policies select the pod but none allows it. So from here on, every rule is a deliberate grant, and anything not granted is refused. A new pod in the namespace starts with no network at all, which is the safe default for a namespace that holds a database.

Egress is the half that most installations skip, and it’s the half that makes the label meaningful. Denying ingress alone protects the database from strangers. Denying egress as well means that a pod which shouldn’t be talking to anything can’t, even to the internet.

Two ways into the app

The app’s ingress policy selects pods by the application-group label and admits two kinds of caller.

spec {
  pod_selector {
    match_labels = {
      "app.kubernetes.io/application-group" = var.platform_name
    }
  }

  policy_types = ["Ingress"]

  # every pod in this namespace
  ingress {
    from {
      pod_selector {}
    }
  }

  # the Traefik ingress controller, on the app's port only
  ingress {
    from {
      namespace_selector {
        match_labels = {
          "kubernetes.io/metadata.name" = local.ingress_namespace
        }
      }
    }
    ports {
      protocol = "TCP"
      port     = local.app_port
    }
  }
}

The difference between those two rules is a classic NetworkPolicy trap, and the file’s comment spells it out. A from with a pod_selector and no namespace_selector means “pods in this policy’s own namespace”. A from with an empty namespace_selector {} means “every pod in every namespace”, which is a long way from what it looks like. The same-namespace rule therefore uses a bare pod_selector {}, and the one caller from outside, Traefik, is named by its namespace’s built-in kubernetes.io/metadata.name label and limited to port 8000. That narrowing is commit dfe1b40.

Every request from the internet reaches Smarter through Traefik, so that one rule is all the outside world needs. Everything else in the cluster, including other applications on the same nodes, gets nothing.

The label is the grant

MariaDB and Redis each have their own policy, and both admit only the label, on their own port.

resource "kubernetes_network_policy_v1" "smarter_mariadb_allow_backend" {
  spec {
    pod_selector {
      match_labels = {
        "app.kubernetes.io/name"     = "mariadb"
        "app.kubernetes.io/instance" = var.platform_name
      }
    }

    policy_types = ["Ingress"]

    ingress {
      from {
        pod_selector {
          match_labels = {
            "app.kubernetes.io/application-group" = var.platform_name
          }
        }
      }
      ports {
        protocol = "TCP"
        port     = local.db_port
      }
    }
  }
}

On the other side, the application group’s egress policy names MariaDB, Redis and DNS in kube-system as destinations. A pod that carries the label can send to them. A pod that doesn’t carry it is selected only by the default-deny policy, and has no egress at all.

That’s the design in one sentence: in this namespace, the application-group label is a network credential. The Helm chart says so in every template that renders a pod, so that nobody removes it as clutter.

# Note:
# The app.kubernetes.io/application-group label is used to identify pods to Calico
# and kubernetes_network_policy_v1 resources in order to allow pod-level communication
# between backing services. If the label is missing then the network policies are
# assured to PREVENT communication.

From helm/charts/smarter/templates/deployment-bastion.yaml, and repeated in the app, worker, beat and init job templates. The web server, both Celery workers, the scheduler, the init job, the backup job and the bastion all carry it. So do MariaDB’s and Redis’s own pods, through the Bitnami subcharts’ podLabels in values.yaml.

The corollary is the most important thing to understand before adopting this. A label is not authentication. Anyone who can create a pod in the namespace can put the label on it, and that pod can then reach the database. Network policies move the boundary from the network to Kubernetes RBAC, and the question “who can reach MariaDB?” becomes “who can create pods in this namespace?”. That’s a better question, because RBAC is audited and IAM-backed, but it has to be asked. A deploy pipeline or an operator with broad create pods rights has, in effect, database network access.

Egress limits ports, not destinations

The application group needs the outside world. It calls LLM providers and AWS over HTTPS, sends email through Amazon SES on 465 and 587, and lets the SQL plugin connect to a remote MySQL or MariaDB database on 3306. Its egress policy grants those ports to any address.

# Allow SMTP (AWS SES) - port 587 (Submission)
egress {
  to {
    ip_block {
      cidr = "0.0.0.0/0"
    }
  }
  ports {
    protocol = "TCP"
    port     = 587
  }
}

Port 80 is open as well, for ApiPlugins and the example weather function, which may call plain-HTTP APIs. Those rules make it hard for a compromised pod to speak an unexpected protocol, but they don’t stop it from reaching an unexpected host on 443. Code that fetches URLs supplied by users or by an LLM needs its own guard against that, such as an SSRF-hardened HTTP client, and an egress proxy with a destination allowlist is the next step for an organization that needs one. Network policies can’t name hosts, because they only see IP addresses, and the addresses behind an LLM provider’s API change.

Scope

The policies are written for one namespace, var.namespace, which is the stack’s production namespace. Another environment’s namespace in the same cluster doesn’t get them automatically. An installation that runs alpha, beta and production side by side should apply the same module to each namespace, or the non-production namespaces will be flat while production is fenced.

What it costs

I rate this a 7 out of 10 in importance. It covers lateral movement inside the cluster, the step after an attacker has a foothold in some pod, and it’s the only layer that separates Smarter’s data from other workloads on the same nodes. It doesn’t stop an attack through the app itself, which is a labelled pod and can reach everything a labelled pod can.

• Administration. The real cost. Every pod template, including those of third-party subcharts, has to carry the label. Every new outbound need, a new SMTP port or a plugin that calls a database on a non-standard port, is a Terraform change in another repository. Labels can’t be renamed without updating every reference, which the chart’s comments also warn about.
• Cost. Calico is open source, and there’s no license. The Tigera operator and the per-node Calico agents use a little CPU and memory on every node.
• User experience. Invisible to end users. Developers notice it the first time something times out, because a policy that’s too narrow doesn’t produce an error message, only a connection that never completes. kubectl get networkpolicy -n <namespace> and a look at the pod’s labels are the first two things to check.
• Stability. Mostly a risk to manage rather than a benefit. A missing label on a new Deployment makes it fail silently on its first database connection. The single enable_enhanced_security switch is the escape hatch.

The verdict is that it’s worth it for any cluster that runs more than one application, or keeps a database in a pod. Start with default-deny in both directions, grant by label, name the ingress controller’s namespace explicitly, and never write namespace_selector {} unless you mean every namespace.

Takeaways

• Deny ingress and egress by default, so that a new pod starts with no network.
• Use one label as the grant, document it in every pod template, and add it to subcharts through their podLabels.
• Name outside callers by the kubernetes.io/metadata.name label of their namespace, and limit them to the port they need.
• Remember that an empty namespace_selector matches every namespace, and a bare pod_selector matches only your own.
• Treat the right to create pods in the namespace as the right to reach its database, and grant it accordingly.
• Expect egress rules to limit ports, not hosts. Put an egress proxy or an SSRF-safe client in front of anything that fetches user-supplied URLs.
• Put the whole layer behind one switch, so that turning it off in an emergency is one reviewed change.

Closing thoughts

These policies aren’t complicated, and that’s their strength. About a dozen Terraform resources turn a flat network into one where the database answers only its own application, the application answers only its ingress controller, and everything else starts from silence. The work is in the labels, and in remembering that once the network trusts a label, the people who can set that label are the ones the network trusts.

• Smarter on GitHub
• smarter-infrastructure on GitHub
• The network policies
• The Smarter Helm chart
• Kubernetes NetworkPolicy
• Smarter on Kubernetes