cert-manager and Route 53: DNS-01 Certificates, One Issuer per Delegated Zone

Smarter gets its TLS certificates from Let’s Encrypt through cert-manager, using the DNS-01 challenge against Amazon Route 53, with one cert-manager ClusterIssuer for each Route 53 zone it owns. This post explains that arrangement as it is built in smarter-infrastructure, what it protects against, what it asks of the people who run it, and when I’d recommend it.

Smarter needs a lot of certificates. The web console and the API each have one, and every deployed chatbot gets its own host name and its own certificate. Those hosts are created by the platform at run time, not by an operator, so certificate issuance has to be automatic, has to work for a host that has only just been created, and has to be scoped tightly enough that one environment can’t affect another’s names.

Zones as a tree

Each environment has two Route 53 hosted zones of its own, one for its platform domain and one for its API domain. Terraform creates them and writes the delegation into the parent, in route53.tf. The platform zone’s NS records go into the root domain’s zone.

resource "aws_route53_zone" "environment_platform_domain" {
  name = local.environment_platform_domain
  tags = local.tags
}

resource "aws_route53_record" "environment_platform_domain_ns" {
  zone_id = data.aws_route53_zone.root_domain.zone_id
  name    = local.environment_platform_domain
  type    = "NS"
  ttl     = "3600"
  records = aws_route53_zone.environment_platform_domain.name_servers
}

and the API zone’s NS records go into the platform zone.

resource "aws_route53_record" "environment_api_domain_ns_parent" {
  zone_id = aws_route53_zone.environment_platform_domain.zone_id
  name    = local.environment_api_domain
  type    = "NS"
  ttl     = "3600"
  records = aws_route53_zone.environment_api_domain.name_servers
}

The root zone is created by hand and only read by Terraform. Everything below it is an environment’s own. So alpha.api.platform.example.com and api.platform.example.com are different zones, with different zone IDs, and a change to one can’t touch a record in the other.

One issuer per zone

cert-manager’s ClusterIssuer is where the ACME account and the challenge solver are configured. Smarter renders one per zone from a single template, issuer.yml.tpl, and names each issuer after its zone.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ${target_domain}
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: no-reply@${root_domain}
    privateKeySecretRef:
      name: ${target_domain}
    solvers:
    - dns01:
        # hosted Zone ID for for the environment domain.
        route53:
          region: ${aws_region}
          hostedZoneID: ${hosted_zone_id}

The hostedZoneID is the important line. Without it, cert-manager’s Route 53 solver searches the account for the zone that best matches the host name. With it, the issuer writes challenge records into exactly one zone. kubernetes-cert-manager.tf fills the template twice per environment, once with the platform zone and once with the API zone.

issuer_api_manifest = yamldecode(templatefile("${path.module}/templates/issuer.yml.tpl", {
  app_name          = var.platform_name
  root_domain       = var.root_domain
  target_domain     = aws_route53_zone.environment_api_domain.name
  aws_region        = var.aws_region
  hosted_zone_id    = aws_route53_zone.environment_api_domain.zone_id
}))

Because the issuer has the zone’s name, an Ingress says which zone its certificate comes from just by naming it. Every chatbot Ingress that Smarter creates at run time carries the API domain’s issuer, from ingress.yaml.tpl.

annotations:
  kubernetes.io/tls-acme: "true"
  cert-manager.io/cluster-issuer: ${cluster_issuer}

where cluster_issuer is smarter_settings.environment_api_domain. The platform doesn’t decide anything about DNS when it requests a certificate. It names a zone, and the issuer for that zone does the rest.

Why DNS-01, not HTTP-01

ACME offers two common ways to prove control of a name. HTTP-01 asks you to serve a token at http://<host>/.well-known/acme-challenge/ on port 80. DNS-01 asks you to publish a TXT record at _acme-challenge.<host>. Smarter uses DNS-01 everywhere, for three reasons.

First, nothing has to be reachable to get a certificate. A new chatbot host has an A record and an Ingress, but HTTP-01 would also depend on the load balancer, Traefik’s routing and the HTTP-to-HTTPS redirect all being right before the very first certificate could be issued. DNS-01 only depends on the zone. A routing mistake shows up as a routing mistake, not as a certificate that never arrives.

Second, port 80 doesn’t have to serve anything for certificates. Smarter’s Traefik still listens on 80, to redirect browsers to HTTPS, but certificate issuance never depends on a plain-HTTP path into the cluster. Certificates keep working even if someone later tightens what port 80 is allowed to do.

Third, the proof lives where the name lives, in a zone that Terraform created and delegated. Whoever can’t write to that zone can’t get a certificate for its names, no matter where traffic happens to be routed.

Credentials without keys

DNS-01 has one real cost, which is that the issuer needs permission to write DNS records. A static AWS access key in a Kubernetes Secret would work, and would be a long-lived credential that can change DNS, sitting in the cluster. Smarter gives cert-manager its AWS permissions through IAM Roles for Service Accounts (IRSA) instead, in kubernetes_cert_manager/main.tf.

module "cert_manager_irsa" {
  source                        = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version                       = "~> 5.34"
  create_role                   = true
  role_name                     = "${var.namespace}-cert_manager-irsa"
  provider_url                  = replace(data.aws_eks_cluster.eks.identity[0].oidc[0].issuer, "https://", "")
  role_policy_arns              = [aws_iam_policy.cert_manager_policy.arn]
  oidc_fully_qualified_subjects = ["system:serviceaccount:${var.cert_manager_namespace}:cert-manager"]
}

Only the cert-manager service account in cert-manager’s namespace can assume that role, and it receives short-lived credentials from the cluster’s OIDC provider. There is no key to leak, rotate or forget.

The role’s policy is narrower than administrator access but wider than the issuers. It allows route53:ChangeResourceRecordSets on hostedzone/*, any zone in the AWS account. The issuers are pinned to their zones, but the role itself is not. If your account holds zones that cert-manager should never touch, list their ARNs in the policy instead of the wildcard. It is a small change, and it makes the IAM boundary match the issuer boundary.

Checking propagation the way the CA does

Before cert-manager tells Let’s Encrypt that a challenge is ready, it checks that the TXT record is visible. By default it asks the cluster’s resolver, which inside an AWS VPC can see a different picture of DNS than the public internet does. Smarter tells it to ask only public resolvers, in cert-manager-values.yaml.tpl.

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: ${role_arn}
  name: cert-manager
extraArgs:
  - --enable-certificate-owner-ref=true
  - --dns01-recursive-nameservers-only
  - --dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53

Google’s and Cloudflare’s resolvers see what Let’s Encrypt will see. That avoids the frustrating failure where cert-manager is satisfied, the CA is not, and the order fails. --enable-certificate-owner-ref makes a certificate’s Secret belong to its Certificate, so deleting a chatbot’s Ingress and Certificate also removes its TLS key from the cluster.

How much does it matter?

I’d put this at a 6 out of 10. It doesn’t stop an attacker on its own. What it does is remove three common causes of certificate trouble, namely shared DNS credentials, issuers that can write anywhere, and issuance that depends on the health of the HTTP path. A platform that issues certificates automatically, at run time, for names it creates, is exactly the place where those problems turn into either outages or a wider blast radius than anyone intended.

Cost, burden and stability

The cost is close to nothing. Let’s Encrypt certificates are free, cert-manager is open source, and the two hosted zones per environment cost fifty cents a month each, plus queries. IRSA has no charge.

The ongoing burden is mostly about limits rather than maintenance.

• Let’s Encrypt has rate limits, including a weekly cap on new certificates per registered domain. One certificate per chatbot host is the right security choice, and it also means a platform with many deploys and undeploys should watch that cap. The template’s comment points to the staging server for testing, which has much higher limits.
• cert-manager renews certificates on its own, well before they expire. The thing to watch is a failing renewal, which shows up in the Certificate’s status and cert-manager’s logs long before a browser notices.
• The cert-manager chart is pinned to a version in Terraform. Upgrades are deliberate, which is good, and also means someone has to make them.

For stability, DNS-01 has one more advantage, because it doesn’t need the cluster to be reachable at all. A certificate can be issued or renewed while the load balancer is being replaced, which is exactly when you’d least like to discover that HTTP-01 is failing.

The user doesn’t see any of this, which is the point. A new chatbot’s host name gets a valid certificate within minutes of being deployed, and the platform polls for it without blocking a worker.

Should you adopt it?

If you run Kubernetes on AWS and issue certificates for names you create at run time, yes. The pieces are all standard. They are Route 53 zones per environment with delegation from the parent, a ClusterIssuer per zone with hostedZoneID set, IRSA instead of keys, and public resolvers for the propagation check. None of it is Smarter-specific, and all of it lives in Terraform, so it is created and destroyed with the environment.

If you have only a handful of fixed host names and a working HTTP path, HTTP-01 is simpler and needs no DNS credentials at all. DNS-01 pays off when names are many, new, or not yet reachable.

Takeaways

• Give each environment its own DNS zone, delegated from its parent by NS records that infrastructure code writes.
• Create one cert-manager ClusterIssuer per zone, name it after the zone, and pin its solver with hostedZoneID.
• Use DNS-01 when the platform creates host names at run time, so issuance never depends on routing.
• Give cert-manager its cloud permissions through IRSA, and scope the IAM policy to the zones it serves.
• Check challenge propagation against public resolvers, the view your certificate authority has.
• Watch Let’s Encrypt’s rate limits if every host gets its own certificate.

Closing thoughts

The arrangement is a tree of zones and a matching set of issuers, each one able to vouch for its own branch and nothing else. It is mostly configuration, written once in Terraform, and it lets the platform request a certificate for a brand-new host by naming the zone it lives in.

• Smarter on GitHub
• smarter-infrastructure on GitHub
• Smarter on Kubernetes
• route53.tf
• issuer.yml.tpl
• kubernetes_cert_manager/main.tf
• Route53 DNS-01, in the cert-manager docs
• The Smarter blog