Smarter’s Custom Django Middleware That Turns Away Vulnerability Scanners

Smarter runs two small pieces of Django middleware whose only job is to make automated vulnerability scanners go away. One refuses requests for files like .env, .git/config and id_rsa, and the other blocks anonymous clients that keep asking for pages that don’t exist. This post explains what they do, what they cost, what they don’t protect against, and whether you should run something like them in your own Django project.

Every host with a public IP address gets scanned. Soon after a new hostname appears in a certificate transparency log, requests start arriving for /.env, /wp-config.php, /phpmyadmin/, /backup.sql and /.aws/credentials. None of those files exist in a Smarter deployment, and Django doesn’t serve files from its source tree anyway. So most of these requests end in a 404, and that’s the point to keep in mind for the rest of this post. This middleware is not what keeps your secrets safe. Not putting them where a web server can reach them is what keeps them safe.

What the middleware does is narrower. It shortens the scanner’s visit, keeps the logs readable, and gives you one more chance in the unlucky case where something sensitive did become reachable, for example a misconfigured static files directory or a volume mounted in the wrong place.

Two middleware classes, two signals

The scanners give themselves away in two ways, and Smarter has one class for each.

Both sit in MIDDLEWARE in smarter/settings/base.py, next to each other.

"smarter.lib.django.middleware.sensitive_files.SmarterBlockSensitiveFilesMiddleware",
# custom throttling middleware to block excessive 404 errors originating
# from botnets probing for vulnerabilities.
"smarter.lib.django.middleware.excessive_404.SmarterBlockExcessive404Middleware",
"django.middleware.common.CommonMiddleware",

ADR-016 is the rule behind them. Smarter defers to Django’s own security features, and the rare exceptions live in smarter.lib.django.middleware, in one place, where they’re easy to review.

Normalize first, then match

A blocklist of file names is easy to write and easy to get around. A scanner that wants /.env can ask for /%2e%65%6e%76, /%252eenv, /static/..%2f.env, \.env, or a Unicode lookalike. A path check that compares the raw string catches none of those.

So the sensitive files middleware normalizes the path before it looks at it. This is the whole function, from sensitive_files.py.

@staticmethod
def normalize_path(path: str) -> str:
    """Normalize paths to reduce traversal and encoding bypasses."""

    for _ in range(2):
        path = urllib.parse.unquote(path)

    path = unicodedata.normalize("NFKC", path)

    path = path.replace("\\", "/")
    path = path.replace("\x00", "")

    path = re.sub(r"/+", "/", path)

    path = posixpath.normpath(path)

    if not path.startswith("/"):
        path = f"/{path}"

    return path.lower()

Each line answers one evasion trick. Decoding twice undoes double URL encoding. NFKC folds Unicode compatibility characters, such as fullwidth letters, into their plain forms. Backslashes become slashes, null bytes disappear, repeated slashes collapse, posixpath.normpath() resolves .. and . segments, and lowercasing makes /.ENV the same as /.env.

Then the normalized path is compared four ways, as a whole against exact names, as a whole against glob patterns, and segment by segment against both. The list is a frozenset, split once at import into exact names and globs.

SENSITIVE_FILES = frozenset(
    {
        ".env",
        "wp-config.php",
        ".git",
        "id_rsa",
        "phpmyadmin",
        "backup",
        "requirements.txt",
        "credentials.json",
        "kubeconfig",
        "*.pem",
        "*.key",
        "*.env.*",
        "*.bak",
        ".aws/credentials",
        # ...
    }
)

(This is trimmed, and the real list has 83 entries.) Matching by segment is what makes it catch /anything/deeper/.git/config without a pattern for every prefix. It is also what makes it aggressive. A segment named backup, db or sql anywhere in a path is enough. That’s deliberate, and the module’s docstring says it “intentionally favors defensive matching and may block requests resembling common attack patterns even if the target resource does not exist.”

Amnesty for paths the platform owns

Aggressive matching needs an escape hatch, or the platform blocks itself. Smarter’s own API, for example, has URLs that could contain a word on the list. The amnesty list is a setting, sensitive_files_amnesty_patterns, validated and compiled at startup by smarter_settings. Its defaults, in defaults.py.

SENSITIVE_FILES_AMNESTY_PATTERNS: List[Pattern] = [
    re.compile(r"^/$"),
    re.compile(r"^/static(/.*)?$"),
    re.compile(r"^/api/v\d+(\.\d+)?/.+"),
    re.compile(r"^/dashboard(/.*)?$"),
    re.compile(r"^/docs/manifest(/.*)?$"),
    re.compile(r"^/docs/json-schema(/.*)?$"),
    re.compile(r"^/config/?$"),
    re.compile(r"^/login/?$"),
    re.compile(r"^/logout/?$"),
    re.compile(r"^/admin/?$"),
]

The patterns are anchored, and they’re compared with the normalized path. The routes they cover have their own protection. The REST API, for example, requires authentication by default (Django REST Framework’s IsAuthenticated), and every API is rate limited.

Counting per client

A single blocked request is cheap. The cost comes from a scanner that tries five hundred paths. Both classes keep a counter per client IP address in the Django cache, which in Smarter is Redis, so every pod sees the same count.

The sensitive files middleware counts blocked requests. Once a client reaches THROTTLE_LIMIT (5), every request from it that isn’t covered by an amnesty pattern gets a 403, whatever it asks for, until the counter expires THROTTLE_TIMEOUT (600 seconds) after the last blocked request.

The 404 middleware works on the response instead of the request, so it catches probes that aren’t on any list.

def _process_response(self, request: HttpRequest, response: HttpResponseBase) -> HttpResponseBase:
    """Shared sync/async implementation."""
    if response.status_code != HTTPStatus.NOT_FOUND:
        return response
    if self.is_ignored_path(request.path):
        return response
    # Authenticated users are exempt
    if is_authenticated_request(request):
        return response
    client_ip = self.get_client_ip(request)
    ...
    if blocked_count >= self.THROTTLE_LIMIT:
        return HttpResponseForbidden(
            "Too many invalid requests detected from your IP address. "
            f"Contact {SMARTER_CUSTOMER_SUPPORT_EMAIL} for assistance."
        )
    self.increment_throttle(throttle_key)
    return response

From excessive_404.py. Three details make it livable. Logged-in users and API key holders never count, so a developer clicking stale links in the console is never blocked. JSON Schema lookups never count either, because the Smarter VS Code extension requests a schema for every manifest kind it sees, including obsolete ones, in bursts. And only every tenth 404 is logged, so a scan of five thousand paths doesn’t produce five thousand log lines.

Note what the 404 middleware does not do. A blocked client still gets the real response for a page that exists. It only stops the guessing.

Which IP address?

Per-IP counting is only as good as the IP address. Behind a load balancer and an ingress controller, Django’s REMOTE_ADDR is the proxy, not the client, so the shared helper in middleware_mixin.py reads the leftmost public address in X-Forwarded-For, then X-Real-IP, then REMOTE_ADDR.

A client can put anything it likes in X-Forwarded-For. The header is only trustworthy if the proxy in front of Django replaces what the client sent. In Smarter’s reference AWS deployment, Traefik sits behind a Network Load Balancer with proxy protocol, and accepts forwarded headers only from the VPC’s own address range. From traefik-values.yaml.tpl in smarter-infrastructure.

websecure:
  port: 443
  forwardedHeaders:
    trustedIPs:
      - "192.168.0.0/20"
  proxyProtocol:
    trustedIPs:
      - "192.168.0.0/20"

If you copy the middleware, copy this part of the deployment too. Without an edge that rewrites the header, per-IP limits are advisory.

The costs

Attack surface and importance

I’d rate this a 4 out of 10. It covers a wide surface, every path of every host the platform serves, but it defends against a narrow risk, a sensitive file that is reachable by mistake. The things that actually keep secrets out of reach are elsewhere. They are credentials that reach the process as environment variables rather than as files, application files that the image makes read-only outside local development, and Django serving only what its URL configuration names. The middleware is a second line, and a noise filter.

Admin burden

Close to none. The list of names is a frozenset in code, reviewed like any other change. The amnesty patterns are a setting, which only needs attention when a new top-level route collides with a word on the list. That shows up as a 403 in development, long before production. Both classes are behind waffle switches, on by default, which an administrator can turn off at run time from the Django admin. That’s useful if one of them misbehaves during an incident. It also means a security control can be switched off without a deploy, which is worth knowing about and worth auditing.

Cost

Small. Each request that isn’t covered by an amnesty pattern costs one Redis read for the throttle counter, and the result of the path check is cached, so the matching runs once per distinct path. The middleware runs after Django’s session and authentication middleware, so a scanner’s request has already paid for those by the time it’s turned away. A rule at the edge, in a web application firewall or the ingress, would be cheaper per request. The middleware’s advantage is that it ships with the application, and works the same on a laptop, in someone else’s cluster, or behind an ingress you don’t control.

UX

Mostly invisible. The false positives are paths with an unlucky word in them, and anonymous users behind one shared IP address, such as an office or campus network, who share one counter. Authenticated users are exempt from the 404 counter, which removes most of that risk for the people who use the console. The 403 messages name a support address rather than saying nothing.

Stability

Neutral. Neither class touches the database, and both use the Redis cache that every request already depends on for its session.

Takeaways

• Normalize a path before you match it, with decode, NFKC, slashes, null bytes, normpath, lowercase.
• Match by segment as well as by whole path, and keep an anchored amnesty list for routes you own.
• Count per client in a shared cache, not in process memory, or each pod forgives what the others saw.
• Throttle on 404s as well as on known names, and exempt authenticated users.
• Trust X-Forwarded-For only when your edge proxy overwrites it.
• Don’t treat any of this as the control that protects your files. Keep secrets out of the web root and the image, and treat this as the second line.

Closing thoughts

These two classes are a few hundred lines of Python, most of it docstrings, and they turn the background noise of the internet into a short list of 403s. They don’t make a deployment secure on their own, and other controls do most of that work. What they do is make reconnaissance slower and logs quieter, at the cost of one Redis read per request.

• Smarter on GitHub
• sensitive_files.py
• excessive_404.py
• ADR-016: Security
• Smarter security documentation
• The Smarter blog