Route 53 and cert-manager: One DNS Record and One TLS Certificate per Deployed LLM

When Smarter deploys an LLM chatbot, it creates an Amazon Route 53 A record, a Kubernetes Ingress and a cert-manager TLS certificate for that chatbot’s hostname and nothing else. There is no wildcard DNS record behind the platform’s API domain, and no wildcard certificate in front of it. This post explains what that design protects against, what it costs to run, and whether it’s worth adopting in your own multi-tenant platform.

Smarter is an open-source Django platform for building and running LLM applications. A chatbot in Smarter is a resource called an LLMClient, declared in a YAML manifest. Every LLMClient that an account deploys gets a public URL of its own, and a platform with hundreds of tenants can have thousands of them. The usual shortcut for that many hosts is a wildcard, meaning one *.api.example.com DNS record pointing at the load balancer, one *.api.example.com certificate in the ingress controller, and an application that sorts out the Host header. It’s fast to set up and it works. It also means that every name anyone can think of resolves, connects and handshakes as if it were real, and that one private key speaks for every tenant at once.

What a deployed chatbot owns

Four objects per chatbot, each named after exactly one host. A Celery task creates them, a second Celery task watches them become ready, and a third removes them. Application code never calls the AWS SDK directly. It goes through Smarter’s infrastructure service (infrastructure.dns, infrastructure.kubernetes), which is also what the tests fake.

The hostname is the tenant boundary

An LLMClient’s default host is built from its name, its account number and the environment’s API domain.

@property
def default_host(self) -> str:
    domain = f"{self.rfc1034_compliant_name}.{self.base_default_host}"
    SmarterValidator.validate_domain(domain)
    return domain

base_default_host is <account-number>.<environment API domain>, so a chatbot called hr in account 1234-5678-9012 on the alpha environment is hr.1234-5678-9012.alpha.api.example.com. Two accounts can both have a chatbot called hr, and their hosts never collide. Because the account number is part of the name, a host can’t be claimed by a different tenant. Another account that creates hr gets a different hostname, a different record and a different certificate.

Why not a wildcard?

I think of the wildcard question as two separate questions, one for DNS and one for TLS, because they fail differently.

A wildcard DNS record answers for names that don’t exist

With *.api.example.com in the zone, anything.api.example.com resolves to the platform’s load balancer. Scanners, typo-squatters and anyone probing for a deleted tenant’s chatbot all reach the ingress controller, which then has to decide what to do with a host it has never heard of. Without the wildcard, a name that Smarter didn’t create returns NXDOMAIN, and the request never leaves the client.

In Smarter the record is a copy of the API domain’s own A record, which Terraform manages as an alias of the Traefik load balancer. The deploy task copies it, rather than pointing at an address of its own.

a_record = self.get_environment_a_record(api_host_domain)
if a_record is None:
    raise DNSZoneNotFound(f"{api_host_domain} has no A record to copy to {hostname}.")
return self.get_or_create_record(
    zone_id=zone_id,
    name=hostname,
    record_type="A",
    ttl=smarter_settings.llmclient_tasks_default_ttl,
    values=a_record.values,
    alias=a_record.alias,
)

The source is smarter/apps/infrastructure/services/dns.py. The chatbot’s record says nothing that the API domain’s record doesn’t already say. It adds exactly one name.

A wildcard certificate is one key for every tenant

A wildcard certificate’s private key authenticates every host under it. If that key leaks, from a misconfigured backup, a debug dump, or a compromised node, the holder can impersonate every chatbot on the platform until the certificate expires or is revoked, and revoking it takes every chatbot down at the same moment. A per-host certificate limits each of those to one host. Its key lives in one Kubernetes Secret named after that host, it can be revoked or reissued on its own, and it is deleted along with the chatbot.

The ingress template makes the exact-host rule plain. The rule’s host and the TLS block’s host are the same single name, and the Secret is named after it.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ${domain}
  namespace: ${environment_namespace}
  annotations:
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: ${cluster_issuer}
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  ingressClassName: traefik
  rules:
  - host: ${domain}
    http:
      paths:
      - backend:
          service:
            name: ${service_name}
            port:
              number: 8000
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - ${domain}
    secretName: ${domain}-tls

The source is smarter/apps/llmclient/k8s/ingress.yaml.tpl, trimmed. Traefik builds one router per Ingress rule, matched on the Host header. A request whose Host matches none of them gets Traefik’s own 404 and never reaches a Django pod.

The platform’s own certificates follow the same rule. The CloudFront certificate in smarter-infrastructure covers only the CDN host; its wildcard subject alternative name is commented out, with a note that the CDN doesn’t need it. Custom domains are the one place where a wildcard appears. When a customer delegates their own domain, Smarter asks AWS Certificate Manager for a certificate covering the domain and its subdomains, and uses its DNS validation as proof that the delegation is real. That certificate is a proof of control, not what Traefik presents. To serve a chatbot on its custom hostname, the deploy task applies the same single-host Ingress template, with its own cert-manager certificate.

How the certificate gets issued

Each environment’s API domain has its own Route 53 hosted zone and its own cert-manager ClusterIssuer, which solves the ACME challenge with DNS-01 in that zone.

spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: ${target_domain}
    solvers:
    - dns01:
        route53:
          region: ${aws_region}
          hostedZoneID: ${hosted_zone_id}

The source is aws/terraform/smarter/templates/issuer.yml.tpl, trimmed. DNS-01 means Let’s Encrypt validates by reading a TXT record that cert-manager writes into the zone, so nothing has to answer on port 80 for the certificate to be issued, and the challenge works the same way for the first certificate and every renewal.

Issuance takes minutes. The deploy task doesn’t wait for it, because a Celery worker blocked on a certificate is a worker that can’t deploy anything else. It applies the Ingress, then hands off to a continuation task that checks once and reschedules itself.

# continue_default_api_deployment checks the ingress resources and certificate 10 minutes after
# the ingress is applied, and then every minute, for up to half an hour, without blocking a worker.
CERTIFICATE_FIRST_CHECK_SECONDS = 600
CERTIFICATE_CHECK_INTERVAL = 60
CERTIFICATE_MAX_ATTEMPTS = 30

The source is smarter/apps/llmclient/tasks/deploy_default_api.py. Each check looks for all three of the Ingress, the Certificate and the TLS Secret. When they’re ready, the task checks that the hostname resolves, every five minutes for up to four hours. Only then does it mark the LLMClient’s DNS as verified and email the account’s primary contact. If the certificate never arrives, the LLMClient’s certificate status is set to failed and the deployment is reported as failed, rather than left half-finished.

Taking it down without racing yourself

Undeploying a chatbot deletes its A record, so the name stops resolving at once. Deleting the LLMClient deletes the A record, the Ingress, the Certificate and the TLS Secret. The deletion task is queued with transaction.on_commit(), so a deletion that is rolled back removes nothing.

Deploy and undeploy tasks for the same chatbot can run at the same moment on different Celery workers, for example when a manifest is reapplied. Each task locks the LLMClient’s database row while it reads its state and creates or destroys the record.

with transaction.atomic():
    try:
        llmclient = LLMClient.objects.select_for_update().get(id=llmclient_id)
    except LLMClient.DoesNotExist:
        ...
        return None
    domain_name = llmclient.default_host
    if smarter_settings.llmclient_tasks_create_dns_record:
        _, created = infrastructure.dns.create_domain_a_record(
            hostname=domain_name, api_host_domain=llmclient.base_api_domain
        )

Whichever task runs second waits, then reads what the first one saved. The undeploy task also does nothing if the LLMClient has been marked deployed again since it was queued, so a late undeploy can’t delete the record of a chatbot that is meant to be live. Only that short section holds the lock; the slow Ingress and certificate work happens outside it. Commit 5faaad9f4 added the lock, with a test that runs both tasks in two threads, each with its own MariaDB connection, the way two workers would.

Even with all of that, a record can outlive its chatbot if a deletion task fails for good. A daily sweep handles those, and a companion post on orphaned DNS records covers it.

How much does it protect?

The threat that per-host naming addresses is impersonation and confusion at the edge, such as a stolen key used against tenants who had nothing to do with the leak, a deleted chatbot’s name still answering, a scanner enumerating hosts for free because every guess resolves. Those are real, but they aren’t the most likely way an LLM platform gets hurt. Prompt injection, leaked provider keys and broken tenant isolation in the database are more likely. I’d rate per-host DNS and TLS at 6 out of 10 for an LLM platform. It removes a whole class of edge failures cleanly, at almost no runtime cost, but it does nothing for the threats that live inside the application.

The design also creates one exposure that a wildcard avoids. Every publicly trusted certificate is written to Certificate Transparency logs, so every deployed chatbot’s hostname, including its account number, becomes public the moment its certificate is issued. Smarter’s account numbers are identifiers, not secrets, and nothing in a hostname grants access. If your tenant identifiers are sensitive, though, don’t put them in hostnames that you certify one at a time.

What it costs to run

Money is the smallest cost. Let’s Encrypt certificates are free, and cert-manager renews them on its own. Route 53 bills per hosted zone and per query, not per record, so a thousand A records cost about the same as one. The limits that matter are quotas, not prices. Route 53 has a default limit on records per hosted zone, and Let’s Encrypt rate-limits new certificates per registered domain per week. A platform that deploys thousands of chatbots in a week, or a test suite that deploys for real in a loop, will hit the second one. Smarter’s tests fake every DNS and Kubernetes call, so they never request a certificate.

The admin burden is mostly watching. Renewal is automatic, and a failed issuance shows up as a failed deployment and a log line, not as an expired certificate three months later. What does need attention is drift between the database and the cluster, such as an Ingress or record that outlives its chatbot. The orphan sweep handles the DNS side. There are more moving parts than with one wildcard, with four Kubernetes objects and a DNS record per chatbot, and Traefik’s router table grows with every deployment.

What users notice

A new chatbot isn’t live immediately. The first certificate check runs ten minutes after the Ingress is applied, so a typical deployment takes ten minutes or more before the owner gets the “your chatbot has been deployed” email. A wildcard setup would answer in seconds. In exchange, each chatbot has a URL of its own that it can keep, put in a page’s embed code, and lose cleanly when it’s deleted. Redeploying an existing chatbot is fast, because its certificate is already issued.

Stability

Per-host certificates make failures local. A certificate that fails to issue blocks one deployment, not the platform. A Let’s Encrypt outage stops new deployments but doesn’t touch existing chatbots, whose certificates are valid for months. A mistake in one Ingress breaks one host. The wildcard’s failure modes are the opposite. There’s only one thing to break, but when it breaks, every tenant goes down together.

Takeaways

• Give each tenant-facing host its own DNS record, created when the host goes live and deleted when it goes away. NXDOMAIN is the cheapest rejection you’ll ever serve.
• Issue one certificate per host with cert-manager and DNS-01. A leaked key then speaks for one host, not for every tenant.
• Put the tenant identifier in the hostname, so that one tenant can never claim another’s name.
• Route on the exact host in the ingress controller, so an unknown Host header is turned away before it reaches the application.
• Never block a worker on a certificate. Check later, and reschedule.
• Lock the resource’s row while you create or destroy its DNS record, and sweep for orphans anyway.
• Remember that Certificate Transparency makes every certified hostname public.

Closing thoughts

A wildcard is a promise that every name under a domain is real. Smarter makes the opposite promise: a name resolves, presents a certificate and reaches a pod only while a chatbot owns it. It costs ten minutes per deployment and some bookkeeping, and in return a stolen key, a deleted chatbot or a guessed name stays a problem for one host at most. For a multi-tenant platform on Kubernetes, I think that’s a good trade.

• Smarter on GitHub
• Documentation
• Deploying an LLMClient on Kubernetes
• deploy_default_api.py and ingress.yaml.tpl
• The ClusterIssuer template in smarter-infrastructure
• ACME DNS-01 with Route 53, in the cert-manager docs
• The Smarter blog