Route 53 and Celery Beat: A Daily Sweep for Orphaned DNS Records
Smarter runs a Celery Beat task once a day that finds Amazon Route 53 A records whose chatbot no longer exists, and deletes them. This post is about that sweep, why dangling DNS records matter, why Smarter’s are less dangerous than most, and the rules that let an automated job delete production DNS records without becoming the next incident.
Smarter is an open-source Django platform for building and running LLM applications. Every chatbot it deploys, a resource called an LLMClient, gets its own hostname and its own A record, of the form <name>.<account-number>.<env>.api.<root>. Deleting the LLMClient queues a Celery task that deletes the record, along with the chatbot’s Kubernetes Ingress and TLS certificate. Most of the time that’s the end of it. But a task can fail after all its retries, a worker can be replaced while a task is queued, or a deletion can happen in a way that never queues the task at all. Each of those leaves a record in the zone that nothing in the database owns.
The problem with a record nobody owns
A dangling DNS record is the starting point of a subdomain takeover. The classic case is a CNAME that points at a third-party resource, such as a storage bucket, a hosted app or a released cloud IP address, after that resource has been deleted. Anyone who claims the same resource name inherits the subdomain, with its cookies, its reputation and its place in your users’ bookmarks.
Smarter’s records are not built that way, and it’s worth being precise about what that does and doesn’t protect. Each chatbot’s record is a copy of the environment API domain’s own A record, which Terraform manages as a Route 53 alias of the platform’s Traefik load balancer. An orphaned record therefore points back at the platform itself, not at a name that a stranger could register. That closes the classic takeover. It doesn’t make the record harmless.
- It keeps answering. A deleted chatbot’s hostname still resolves and still reaches the platform’s edge, where it’s turned away later and more expensively than an NXDOMAIN would be.
- It outlives the infrastructure it was copied from. If an environment’s load balancer is replaced, or an environment is torn down, every leftover record keeps the target it was copied with. A stale target is exactly the precondition that takeovers need, and nobody is watching those names any more.
- It’s an inventory leak. Each record names an account and a chatbot that used to exist.
- It accumulates. Route 53 hosted zones have a record quota, and records that nobody deletes count against it forever.
The rule I follow is that the DNS zone should be a function of the database, so every chatbot host that exists in one should exist in the other. The deletion task enforces that in the common case. The sweep enforces it in all the others.
How the sweep decides what’s orphaned
The task lists the A records in the environment’s API zone, computes the set of hosts that the database says are live, and takes the difference. Its interesting parts are the rules that keep it from deleting something it shouldn’t.
Only records with exactly the right shape
A chatbot’s default host has a precise shape. It is one RFC 1034 label, an account number, then the environment’s API domain. The sweep compiles that into an anchored regular expression.
HOST_LABEL = r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?"
ACCOUNT_NUMBER = r"\d{4}-\d{4}-\d{4}"
def host_pattern(domain: str) -> re.Pattern:
return re.compile(rf"^{HOST_LABEL}\.{ACCOUNT_NUMBER}\.{re.escape(domain)}$")
The code is from smarter/apps/llmclient/tasks/delete_orphan_dns_records.py, trimmed. Everything else in the zone is invisible to it, including the API domain’s own record, TXT records such as ACME challenges, wildcards (a * is not a valid label), and other environments’ hosts. That last one matters. A shared parent zone can hold example.1234-5678-9012.alpha.api.example.com for the alpha environment, and production’s sweep, looking for <label>.<account>.api.example.com, sees one label too many and leaves it alone.
Live means “exists”, not “deployed”
A host is live if any LLMClient uses it, whether or not that LLMClient is currently deployed. Hosts of LLMHosts, Smarter’s self-hosted open-weight models, have the same shape in the same zone, so they count as live too.
llmclients = list(LLMClient.objects.select_related("user_profile__account"))
if not llmclients:
raise OrphanDNSRecordsError("the database has no LLMClients.")
hosts: set[str] = set()
try:
for llmclient in llmclients:
hosts.add(normalize_name(llmclient.default_host))
for llmhost in LLMHost.objects.select_related("user_profile__account"):
hosts.add(normalize_name(LLMHostService.default_hostname(llmhost, LLMHostService.base_name(llmhost))))
except Exception as e:
raise OrphanDNSRecordsError(f"the host of a resource could not be computed: {e}") from e
The sweep is for records of things that are gone. Undeploying a chatbot already deletes its record; if that ever fails, the owner can redeploy or undeploy again, and the sweep stays out of a decision that belongs to them.
When in doubt, delete nothing
Two conditions make every record in the zone look orphaned. One is an empty LLMClient table, and the other is a host that can’t be computed. An empty table can mean a restored backup that hasn’t finished loading, a misconfigured database URL that points at a fresh schema, or a migration that went wrong. Rather than treat any of those as permission to empty the zone, the task raises OrphanDNSRecordsError, logs it, and returns without deleting anything. The same applies when the zone itself can’t be found.
Only in the environments that own their zone
In local development, the API domain is a proxy domain that every developer’s environment shares, and one developer’s database knows nothing about another’s chatbots. So the task returns immediately anywhere other than Smarter’s AWS environments.
if smarter_settings.environment not in SmarterEnvironments.aws_environments:
logger.info("%s skipped: the %s environment shares its DNS zone.", fn_name, smarter_settings.environment)
return []
A ceiling on every run
Even with all of that, a bug in the live-host computation could still produce a long list of false orphans. The task deletes at most 50 records per run, logs a warning when it finds more, and leaves the rest to the following days.
if len(orphans) > MAX_DELETIONS_PER_RUN:
logger.warning(
"%s found %s orphaned A records in %s. Deleting the first %s; the next runs delete the rest.",
fn_name, len(orphans), domain, MAX_DELETIONS_PER_RUN,
)
orphans = orphans[:MAX_DELETIONS_PER_RUN]
The worst case for a bad day is 50 records, each of which is recreated by redeploying its chatbot, and an alert-worthy warning in the logs.
Running it
Celery Beat schedules the task daily on the infrastructure queue, the same queue that deploys chatbots, so it never competes with prompts for a worker.
"delete-orphan-dns-records": {
"task": "smarter.apps.llmclient.tasks.delete_orphan_dns_records.delete_orphan_dns_records",
"schedule": timedelta(days=1),
"options": INFRASTRUCTURE,
},
The schedule is in smarter/workers/celerybeat.py. An operator can run it by hand, and should start with a dry run, which lists what it would delete and deletes nothing.
python manage.py delete_orphan_dns_records --dry-run
The sweep went in with commit 1f7ec8ac6, whose tests cover each of the guards above. Only orphans are deleted, dry runs delete nothing, the environment, readiness, zone, empty-database and uncomputable-host checks, the cap, failed and already-deleted records, the management command and the host pattern itself. The DNS service is mocked in every one of them, so no test ever touches the real Route 53.
Scope
The sweep covers DNS records in the environment’s API zone, and nothing else. It doesn’t remove Kubernetes Ingresses or certificates; the deletion task does that, and without a DNS record pointing at it, a leftover Ingress is not reachable by name. Records that a chatbot created in a customer’s own delegated zone, for a custom domain, are outside the API zone and outside the sweep.
Is it worth it?
On Smarter’s design, I’d rate the sweep at 4 out of 10 for attack-surface reduction. The alias records already close the classic takeover, so what remains is hygiene, with fewer names answering, less inventory exposed, nothing left pointing at infrastructure after it’s replaced. On a platform whose per-tenant records are CNAMEs to third-party services, the same sweep would be a 9, because there every orphan is a takeover waiting for someone to notice.
The rest of the ledger is short.
- Cost is too small to measure. One Route 53 list call and a handful of deletes a day, on a worker that’s already running.
- Admin burden is reading a warning when one appears. A run that hits the cap, or refuses to run because the database looks empty, is worth a look, because both mean something upstream is wrong.
- UX is unaffected. Users never see it. A chatbot whose LLMClient still exists is never touched.
- Stability comes from the guards, which make this safe to automate. Without them, a daily job with delete rights on production DNS is a liability; with them, the worst case is 50 recoverable records.
Should you adopt it? If your platform creates DNS records per tenant, yes. The pattern is small and portable. It needs an exact shape for the records you own, a live set computed from your database, a refusal to act when the live set looks wrong, a cap per run, a dry-run mode, and a schedule. Write the guards first.
Takeaways
- Treat your DNS zone as a function of your database, and reconcile the two on a schedule, not just at deletion time.
- Make records point at targets you own. An alias to your own load balancer can’t be claimed by a stranger, a CNAME to a deleted bucket can.
- Match only the exact shape of the records your application creates, with an anchored pattern.
- Refuse to delete anything when the live set is empty or incomplete.
- Cap deletions per run, and log loudly when the cap is hit.
- Ship a dry run, and use it before the first real run in every environment.
Closing thoughts
Subdomain takeover gets attention because the attack is cheap and the damage is public. The defense is cheaper still. Delete records when their owners go away, and check every day that you did. Smarter’s sweep is under two hundred lines, and most of them are about when not to delete.