LLM Guardrails in YAML: How Smarter Declares, Orders and Audits Them

Smarter declares its LLM guardrails as YAML manifests. A Guardrail is a resource like any other, applied with the same apply command as an LLMClient or a Plugin, and attached to an LLMClient by name. This post describes what a guardrail can check, what it can do about what it finds, the order in which several of them run, what they record, and exactly which text they look at. That last part matters more than the rest, because a guardrail that you think covers something it doesn’t is worse than none.

A guardrail is the code that sits between a person and an LLM and decides whether a message may pass, and in what form. Most teams start with a sentence in the system prompt (“never reveal a customer’s card number”) and discover that the model follows it most of the time. A guardrail is the same rule written as software. It runs on every prompt, whether the model is in a cooperative mood or not, and it leaves a record when it acts.

A guardrail is a manifest

Here is one of the built-in guardrails that every Smarter platform loads, trimmed. It redacts personal data from the user’s message before the message leaves for a third-party LLM provider.

apiVersion: smarter.sh/v1
kind: Guardrail
metadata:
  name: pii_redaction_input
spec:
  config:
    isActive: true
    mode: enforce
    stage: input
    category: pii
    strategy: detector
    detectors:
    - credit_card
    - us_ssn
    - email
    - phone_number
    - iban
    action: redact
    replacement: '[REDACTED]'
    severity: 4
    priority: 10

An LLMClient opts in by listing guardrails in its manifest, under spec.guardrails. That is the whole integration. The application code that calls the LLM doesn’t change, and the guardrail can be reviewed, versioned and diffed like any other file in a pull request. For a team where security review is a separate step from development, that is the main attraction, because the reviewer reads twenty lines of YAML, not the chat pipeline.

Each guardrail is a combination of two choices, a strategy, which decides whether some text triggers it, and an action, which decides what happens next.

The actions are log, flag, redact, transform, block and escalate. Redact and transform rewrite the matched spans, so they need a strategy that knows where the match is. The manifest refuses redact with llm_judge, for example, because a judge says “yes, this is PII” without saying which characters.

The detector strategy deserves a word, because it is the one most readers will want. Its built-in detectors are credit_card, us_ssn, email, phone_number, ip_address, iban, aws_access_key, private_key, api_key, jwt and password_assignment. Where the data has a checksum, the detector checks it. A card number must pass the Luhn check, and an IBAN the mod-97 check. They favor precision over recall, so a 16-digit order number is not taken for a card.

Order, and the most severe outcome

An LLMClient usually has several guardrails, and they interact. Smarter runs them one at a time, sorted by priority and then by id, and each one sees the payload as the guardrails before it left it. This is from pipeline.py.

_DISPOSITION_PRECEDENCE = [
    PipelineDisposition.ALLOWED,
    PipelineDisposition.FLAGGED,
    PipelineDisposition.REDACTED,
    PipelineDisposition.TRANSFORMED,
    PipelineDisposition.ESCALATED,
    PipelineDisposition.BLOCKED,
]

The pipeline’s result is the most severe disposition of any guardrail that triggered. Two consequences follow, and both are deliberate.

First, ordering is a privacy control. A redaction at priority 10 runs before an llm_judge at priority 90, so the judge, which is itself a call to a third-party model, never sees the card number that the redaction removed.

Second, ordering is a cost control. The first guardrail that blocks stops the pipeline, so the cheap local checks should have the low priority numbers, and the guardrails that call a model should come last. A message that a keyword list rejects in microseconds never pays for a judge.

When an input guardrail blocks, the LLM is not called at all. The user gets the guardrail’s message as an ordinary assistant reply with a finish_reason of content_filter, which is what the OpenAI API returns when its own filter steps in, so clients that already handle that case need nothing new. The blocked message is also replaced in the conversation’s stored history with a placeholder, so it isn’t sent to the model with the next prompt either.

Shadow mode, and failing open

Two settings decide how a guardrail behaves when you are not yet sure of it, and when it breaks.

mode: monitor evaluates the guardrail and records what it would have done, but never changes or blocks anything. New guardrails should start there. A regex that looks precise on a whiteboard will match something you didn’t expect in real traffic, and monitor mode lets you find out from the event log rather than from a user.

failClosed decides what happens when the guardrail itself fails, for example because its moderation provider timed out. The engine catches every exception a strategy raises, records it as an error event, and moves on. By default the prompt continues, so a broken guardrail never breaks a chat. With failClosed: true, the failure blocks instead. The rule in the pipeline is short.

blocks = guardrail.fail_closed and guardrail.mode == GuardrailMode.ENFORCE

Fail-open is the right default for most guardrails, which exist to reduce risk rather than to enforce a legal obligation. For the few that do enforce one, failing closed is a one-line change in the manifest, and the cost of that choice (an outage of the moderation provider becomes an outage of your chatbot) is visible in the same file.

An audit trail that doesn’t repeat the secret

Every time a guardrail triggers or fails, Smarter writes a GuardrailEvent row that records which guardrail, which LLMClient and session, the stage, the action, the disposition, the confidence and a short excerpt of what it matched. An audit trail that stored the credit card number it had just redacted would be a new place for the card number to leak from. So the excerpt is masked by category, in events.py.

def mask(text: str, category: str) -> str:
    if category == GuardrailCategory.SECRETS:
        return f"[{len(text)} characters]"
    if category == GuardrailCategory.PII:
        return "*" * max(len(text) - 4, 4) + (text[-4:] if len(text) >= 8 else "")
    return text

A secret is recorded only by its length. Personal data keeps its last four characters, enough for a support person to say “the card ending in 4242” and no more. The pipeline also sends Django signals when a guardrail triggers, blocks, escalates or fails, which is where an escalate action would page a human.

What a guardrail looks at

This is the section to read twice. The guardrails scan two pieces of text, and nothing else.

• Input guardrails scan the user’s latest message. Not the system prompt, which would make a prompt-injection guardrail trigger on the operator’s own instructions. Not the earlier turns of the conversation.
• Output guardrails scan the LLM’s final reply, meaning the content and refusal of each choice, after any tool calls have been answered.

The code is explicit about it, in text_extraction.py.

def extract_pre_segments(payload: SmarterDictType) -> list[TextSegment]:
    """Return the segments of the latest user message of a chat completion request."""
    messages = payload.get("messages") or []
    index = latest_user_message_index(messages)
    if index is None:
        return []
    return content_segments(messages[index].get("content"), f"messages[{index}].content")

So the guardrails don’t see the results that plugins, MCP servers and extensions return to the model, or the arguments of the tool calls the model makes along the way. A web page that the web search plugin fetched, and that contains instructions aimed at the model, reaches the model without passing an input guardrail. That kind of attack, indirect prompt injection, is real and is not what these guardrails defend against. The output guardrails still see the final reply, so a secrets or PII guardrail on the output stage will catch a leak in what the user is shown, whatever caused it. They won’t catch a tool call that sends data somewhere else before the reply is written.

The guardrails also run only in Smarter’s chat pipeline. The provider proxy at /api/v1/proxy/, which forwards an SDK’s requests to OpenAI or Anthropic unchanged, doesn’t run them.

None of this makes the guardrails less useful for what they do cover. It means the defense against a hostile tool result has to be somewhere else, in plugins with least privilege, read-only connections, tools that can’t send data to an address the model chooses, and output guardrails as the last check. Smarter’s own documentation says the same thing in its security notes. Guardrails reduce risk, they don’t remove it.

How much does it cover?

On a scale of one to ten, I put guardrails at about a 6 for a typical Smarter deployment. The surface they cover is the most visible one, a person typing into a chat box and reading the answer, and the most common incidents there (personal data sent to a provider, a reply that repeats a secret, a user trying to talk the assistant out of its instructions) are exactly what they handle. They lose points because the less visible surface, everything a tool brings into the conversation, is outside their scope, and because pattern guardrails are evaded by anyone willing to spell a word differently. The model-based strategies are harder to evade, and are themselves models that can be argued with. Smarter tells the judge in its system message that the text under judgement is untrusted data whose instructions it must not follow, which helps, and is not a proof.

What it costs to run

Cost. The local strategies are free. Each semantic, moderation or llm_judge guardrail adds one provider call to every prompt it runs on, so an LLMClient with two judges on input and one on output makes four model calls where it used to make one. Reference embeddings for the semantic strategy are cached, so each prompt embeds only the new message.

User experience. A block arrives as a polite assistant message, not an error. Redaction is visible, because the conversation’s saved history holds [REDACTED] where the user typed a card number, which is honest, and occasionally confusing when they come back to it. False positives are the real UX cost, and monitor mode is how you keep them out of production.

Stability. The engine catches every exception, the provider clients time out after 15 seconds and retry once, and fail-open is the default. A guardrail is unlikely to take a chatbot down, unless you set failClosed and its provider goes down.

Admin burden. Moderate and ongoing. Someone has to read the event log, tune thresholds, move guardrails from monitor to enforce, and revisit the keyword lists when users find new phrasings. This is the part teams underestimate.

Takeaways

• Put guardrails in configuration that a reviewer can read, not in the code that calls the model.
• Run them in a defined order, cheap and local first, and let the most severe outcome win.
• Redact before any guardrail that sends text to another model.
• Start every guardrail in a monitor mode, and decide per guardrail whether it fails open or closed.
• Mask what the audit trail records, by category.
• Write down exactly which text your guardrails scan, and defend everything else (tool results above all) by other means.

Closing thoughts

Is it worth it? Yes, with the scope in view. A YAML guardrail that redacts card numbers before they reach a provider, and blocks a reply that contains an API key, is a small amount of configuration for a real reduction in risk, and its audit trail is the evidence a compliance review will ask for. What it isn’t is a boundary around everything the model reads. Treat the guardrails as the controls on the front door, and keep the tools behind them on a short leash.

• Smarter on GitHub
• Guardrails in the Smarter docs
• The guardrail services README
• The guardrail strategies README
• pipeline.py, text_extraction.py and events.py
• The built-in guardrail manifests