Amazon VPC and EKS: Keeping Smarter's Pods Off the Internet

Smarter‘s production environment runs in an Amazon VPC where no pod, node, database or cache has a public IP address, and where web traffic has exactly one way in, through a Network Load Balancer on ports 80 and 443, in front of the Traefik ingress controller. This post describes that VPC, as smarter-infrastructure‘s Terraform builds it, and what it costs to run it that way.

The reason for the design is simple. Smarter’s pods hold everything worth stealing on the platform, including the Django SECRET_KEY, the LLM provider keys that pay for every prompt, AWS credentials that can change DNS records, and a MariaDB database of users, prompts and chat history. Most of the ways those things leak start with a listener that nobody meant to expose, such as a debugging port, a NodePort, a database bound to 0.0.0.0, a node launched with a public address. A network that has no route from the internet to the pods removes that whole class of mistake, whatever the Helm chart or a future pull request does.

Walking in from the internet

Layer	What runs there	Reachable from the internet?
Public subnets	The Network Load Balancer, the NAT gateway	The load balancer, on 80 and 443
Private subnets	EKS worker nodes, every pod, MariaDB and Redis (as pods)	No
EKS control plane	The Kubernetes API	Yes, with IAM authentication
Intra, database, elasticache subnets	Reserved tiers with no internet route	No

The rest of this post goes through those rows from the outside in.

Diagram of Smarter's AWS VPC, 192.168.0.0/20. Requests from the internet enter only through a Network Load Balancer on ports 80 and 443 in the public subnets, pass with proxy protocol to Traefik and then to the Smarter app on port 8000 in the private subnets, where the EKS worker nodes and the MariaDB and Redis pods have no public addresses. Outbound calls leave through one NAT gateway to LLM providers, SES and Route 53. The EKS control plane has a public, IAM-authenticated endpoint, and the intra, database and elasticache subnets are reserved with no internet route.

Five subnet tiers, two in use

The VPC is a /20 with five tiers of subnets, created by the community terraform-aws-modules/vpc module.

cidr = "192.168.0.0/20"
azs  = ["${local.aws_region}a", "${local.aws_region}b", "${local.aws_region}d"]

database_subnets    = ["192.168.1.0/24", "192.168.2.0/24"]
elasticache_subnets = ["192.168.3.0/24", "192.168.4.0/24"]
public_subnets      = ["192.168.5.0/24", "192.168.6.0/24", "192.168.7.0/24"]
private_subnets     = ["192.168.8.0/24", "192.168.9.0/24", "192.168.10.0/24"]
intra_subnets       = ["192.168.11.0/24", "192.168.12.0/24", "192.168.13.0/24"]

From aws/stack/vpc/terragrunt.hcl.

Only two tiers carry Smarter’s traffic today. The public subnets hold the load balancer and the NAT gateway. The private subnets hold everything else. The EKS module puts both the managed node groups and the control plane’s network interfaces there.

endpoint_private_access         = true
endpoint_public_access          = true
subnet_ids                      = var.private_subnets
control_plane_subnet_ids        = var.private_subnets

From aws/terraform/kubernetes/main.tf.

MariaDB and Redis are not RDS or ElastiCache. They are Bitnami subcharts of the Smarter Helm chart, so they run as pods on those same private nodes, behind a Kubernetes Service with no external address. The database and elasticache tiers exist so that a larger installation can move to the managed services without re-addressing the VPC. Subnet tiers are cheap to reserve and painful to add later.

A private subnet has a route to the NAT gateway for outbound traffic, and nothing else. An instance launched there has no public IP, so the internet has no address to send it a packet to, whatever its security group says. That is the property everything else in this post leans on.

The front door is one load balancer, proxy protocol and Traefik

Every request for the web console, the REST API or a deployed chatbot reaches the cluster the same way. Terraform installs Traefik with the Helm chart’s values in traefik-values.yaml.tpl, and its Service asks AWS for an internet-facing NLB.

ports:
  websecure:
    port: 443
    expose:
      default: true
      exposedPort: 443
    protocol: TCP
    forwardedHeaders:
      trustedIPs:
        - "192.168.0.0/20"
    proxyProtocol:
      trustedIPs:
        - "192.168.0.0/20"

service:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

  spec:
    externalTrafficPolicy: Local

Three settings in that file work together.

Proxy protocol. The NLB passes TCP through without terminating it, so the client’s address would otherwise be lost at the first hop. With the proxy-protocol annotation, the NLB prefixes each connection with a small header that carries the original source address, and Traefik reads it.
externalTrafficPolicy: Local. Kubernetes delivers a connection only to a Traefik pod on the node that received it, instead of forwarding it to another node and rewriting its source on the way. The address Traefik sees is the client’s.
trustedIPs. Traefik accepts the proxy-protocol header and X-Forwarded-* headers only from addresses inside the VPC’s range. The load balancer is inside it. A client on the internet is not, so the forwarded headers a client writes into its own request are not what Traefik takes as the client’s address.

The point of all three is that the client address Smarter logs and rate-limits on is the address that actually opened the connection. TLS is terminated by Traefik, with a certificate per host from cert-manager, and Traefik routes by Ingress to the Smarter app’s Service on port 8000. Neither the Terraform nor the Smarter Helm chart creates another Service of type LoadBalancer, and nothing in the chart uses hostPort, hostNetwork or a NodePort of its own.

Diagram of how the client's real address reaches Traefik, from client to Network Load Balancer to node to Traefik, with three settings working together. Proxy protocol has the NLB add a header carrying the original source address. externalTrafficPolicy Local keeps the connection on the node that received it without rewriting its source. trustedIPs 192.168.0.0/20 means forwarded headers are accepted only from inside the VPC. A forged X-Forwarded-For header from a client is not taken as its address.

Why the node security group allows NodePorts from anywhere

The one rule in this design that looks wrong at first is in the node security group.

# TCP port forwarding for NLB/LoadBalancer services
# for receiving external traffic from the Traefik Ingress Controller NLB
ingress_nlb_tcp_30283 = {
  description   = "${var.platform_name}: NLB client port 30283"
  protocol      = "tcp"
  from_port     = 30000
  to_port       = 32767
  type          = "ingress"
  cidr_blocks   = ["0.0.0.0/0"]
}

From aws/terraform/kubernetes/main.tf.

An NLB with instance targets preserves the client’s IP address all the way to the node. So the packets that arrive at Traefik’s NodePort come from every address on the internet, and the security group has to admit every address, or the load balancer’s traffic would be dropped. This doesn’t expose the nodes, because the nodes have no public addresses, so the only thing on the internet that can send a packet to a private node is the load balancer in front of it. The rule reads as “anyone”, and in practice means “anyone the NLB forwards”.

The same file is less strict inside the VPC. With enhanced security on, it replaces the EKS module’s recommended rules with Smarter’s own, and one of those admits all traffic from the VPC’s own address range. The VPC is a trust zone at the security-group layer. What sits inside it matters, and the pod-level network policies that Calico enforces in the application’s namespace carry the next layer of the work. That deserves a post of its own.

The control plane is public, and IAM is the lock

The EKS API endpoint is reachable from the internet (endpoint_public_access = true), as well as privately from inside the VPC. That surprises people who expect a private cluster to have a private API. It’s a deliberate trade. The public endpoint leads to the Kubernetes control plane that AWS runs, not into the VPC or the pod network. Every request to it has to authenticate as an IAM identity that the cluster maps to Kubernetes permissions, and the nodes use the private endpoint. Keeping it public means that the CI deploy workflow and an administrator’s laptop can run helm upgrade and kubectl without a VPN, which Smarter, as a community project with no corporate network, does not have.

An organization with a VPN or a private CI runner should restrict the endpoint. The module takes a list of allowed source ranges (endpoint_public_access_cidrs), whose default is every address, or the endpoint can be made private only.

What else is internet-facing

Two other things in smarter-infrastructure accept connections from the internet, and neither leads to a pod.

CloudFront serves the chat widget and static assets from S3, not from the cluster. And the VPC module still builds an EC2 bastion host in a public subnet that accepts SSH, from the time before the Helm chart had a bastion of its own. It sits outside the cluster, and it is the one exception to “no public IP addresses” in this VPC. The in-cluster bastion pod, which an administrator reaches with kubectl exec through the EKS API, does the same job without an open port, and it makes the EC2 host unnecessary. With that host gone, the NLB is the only listener in the VPC.

Outbound through one NAT gateway, and S3 off the meter

Private nodes still need the internet for LLM provider APIs, Docker Hub through ECR’s pull-through cache, Amazon SES, Route 53. That traffic leaves through a NAT gateway, and the VPC has exactly one.

enable_nat_gateway     = true
single_nat_gateway     = true
one_nat_gateway_per_az = false

That’s a cost decision with a stability price. One NAT gateway is a single point of failure for outbound traffic. If its availability zone has trouble, every node loses the internet, and Smarter can’t reach its LLM providers. A gateway per zone removes that, at about three times the hourly charge. For Smarter’s size, one is the right answer. For an organization with an availability target, it isn’t.

The other outbound decision is a free S3 gateway endpoint, added in commit a77f343.

# S3 gateway endpoint. Gateway endpoints are free, and keep S3 traffic
# (including ECR image layer pulls, which are served from S3) off of the
# NAT Gateway, avoiding its $0.05/GB data processing charge.
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = module.vpc.vpc_id
  service_name      = "com.amazonaws.${var.aws_region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids = concat(
    module.vpc.private_route_table_ids,
    module.vpc.intra_route_table_ids,
  )
}

Image layers are the largest thing a node downloads, and they come from S3. With the endpoint, they never touch the NAT gateway’s meter. The same commit removed the ECR and STS interface endpoints, which are billed per zone and per hour, and cost more than the NAT charges they saved. Security people sometimes want interface endpoints for their own sake, to keep AWS API calls off the internet. That’s a fair goal, but at Smarter’s scale it’s about $74 a month to move traffic that is already TLS-encrypted.

Diagram of the two outbound paths from the private EKS nodes. One goes through a NAT gateway, with an hourly charge plus 0.05 dollars per GB, to LLM provider APIs, Docker Hub via ECR's cache, Amazon SES and Route 53. The other goes through a free S3 gateway endpoint, added in commit a77f343, to Amazon S3, including ECR image layers. The same commit removed the ECR and STS interface endpoints, about 74 dollars a month.

One more default is worth knowing about. The EKS module’s managed node groups require IMDSv2 with a hop limit of 1, so a process inside a pod, one network hop away from the node, can’t get a token for the node’s instance metadata service and the node’s IAM role. Smarter doesn’t set this. It inherits it, and anyone overriding the node group’s metadata_options should keep it.

What it costs to run

On the threat side, I rate this design a 9 out of 10 in importance. It covers the largest attack surface a cluster has, every port on every node and pod, and it does it in the network rather than in configuration that has to stay correct on every deploy. It doesn’t stop an attack through the front door. A flaw in Django, in a plugin or in an LLM tool call is still reachable through Traefik. It makes sure that the front door is the only door.

Administration. Close to none, once it exists. The subnets, routes and NAT gateway don’t change. The work is in review, noticing a pull request that adds a LoadBalancer Service, a hostPort or a node group in a public subnet.
Cost. One NAT gateway and one NLB, each with an hourly charge and a per-gigabyte charge. The S3 gateway endpoint is free and pays for itself on image pulls. Private nodes cost nothing extra.
User experience. None for end users. For administrators, there is no SSH to a node and no public address to connect a database client to. Everything goes through the Kubernetes API, which takes some getting used to.
Stability. Mostly positive, because fewer listeners means fewer things to break. The single NAT gateway is the exception, and it’s a deliberate one.

The verdict is that it’s worth it, for almost anyone running Kubernetes on AWS. Private nodes are the EKS module’s ordinary configuration, not an advanced one, and most of the effort is in not undoing it later.

Takeaways

Put nodes, pods and stateful services in private subnets, and give the internet one load balancer to talk to.
If you need real client addresses, use proxy protocol, externalTrafficPolicy: Local and a trusted-IP list for forwarded headers together. Any one of them alone doesn’t do it.
Read a 0.0.0.0/0 rule on a private node’s security group for what it is. It can only ever match traffic that the load balancer forwards.
Treat the VPC as a trust zone, and keep track of everything inside it, especially anything with a public address.
A public EKS endpoint is a reasonable trade without a VPN, as long as IAM is the lock. Restrict its source ranges if you have a network to restrict it to.
Add an S3 gateway endpoint to every VPC. It’s free.

Closing thoughts

None of this is novel. It’s the network that AWS’s own reference designs describe, built with the community Terraform modules, and that’s the case for it: there’s nothing clever to get wrong. The pods have no address on the internet, the load balancer is the one way in, and the Kubernetes API is the one way to administer them. Everything else the platform does about security starts from there.