Smarter’s MCP Server Gives Sysadmins Live Platform Diagnostics in a Chat

Smarter’s MCP server has a set of tools for the people who run a Smarter platform. A sysadmin opens Smarter Sysadmin, a chatbot in the Prompt Engineer Workbench, and asks “Is anything wrong right now?” or “Why can’t this user sign in?” The chatbot answers from live data. It reads the platform’s Kubernetes pods and nodes, the AWS EKS cluster they run on, the database, cache and Celery workers, the last 72 hours of pod logs, and the state of one user’s account. The same tools work from Claude Code, or any other MCP client, with a staff member’s API key.

The MCP server is a Django app inside the platform that speaks the Model Context Protocol. Its public tools, which answer questions about Smarter’s code, documentation and blog, are the subject of Introducing the Smarter MCP Server and How Smarter’s MCP Server Searches Its Own Code, Documentation and Blog. This post introduces its staff-only tools, what they do, how to work with them, and how they work. Claude Code designed and wrote them, with me as co-author, and every code sample comes from the mcpserver app of the platform’s repository.

What the staff-only tools do

There are seven of them, and each answers one kind of question.

Every one of them only reads. Nothing they do changes the platform, and nothing they find goes anywhere but back to the staff member who asked. Each call takes a ticket, the help desk ticket or the reason for the call, and each call is recorded in an audit trail.

That covers the questions an on-call sysadmin and a help desk ask most. Is the platform healthy? What broke, when, and how often? What happened to this one person? And what can I tell them?

Working with them

In Smarter Sysadmin

Smarter Sysadmin is a built-in LLMClient named smarter_sysadmin. Every platform applies it to the Smarter account, the account that runs the platform, and it’s staff-only, so nobody else can see it, superusers of other accounts included. It’s never deployed. Open it in the Prompt Engineer Workbench and ask in plain English.

apiVersion: smarter.sh/v1
kind: LLMClient
metadata:
  name: smarter_sysadmin
spec:
  config:
    prompt:
      staffOnly: true
      appName: Smarter Sysadmin
  mcpClients:
    - smarter_operations
    - smarter_project
  guardrails:
    - prompt_injection_keyword_input
    - secrets_redaction_input
    - secrets_leak_output
    - script_injection_output

Trimmed from its manifest in the example-manifests repository. It reaches the MCP server through two built-in MCPClients. smarter_operations connects to the staff-only tools, and smarter_project to the public ones. With both, it can find an error in the logs and then explain it from the code that raised it.

Some questions to start with.

• “Is Smarter down, or slow, right now?”
• “What has the Celery worker been logging errors about in the last hour?”
• “Is the EKS cluster ready for the next Kubernetes version?”
• “Why can’t the user at this address sign in? Ticket 4211, verified because they replied from it.”
• “Draft a reply to that ticket, and check it before I send it.”

A help desk ticket, end to end

Here’s how the tools work together on a typical ticket.

• A user writes “My login isn’t working” from their own address. A staff member opens Smarter Sysadmin, pastes the ticket, and says the address was verified because the user replied from it.
• The agent calls platform_status with the ticket number. Nothing is wrong with the platform.
• It calls user_support_snapshot with the address and how it was verified. The user signed up with Google and has no password, and their four failed sign-ins today all failed for that reason.
• It drafts a reply that suggests signing in with Google, and checks it with check_reply, which finds nothing.
• The staff member reads the draft and sends it. The audit trail has two rows for the ticket, both under the staff member’s name.

The user gets an answer in minutes. Nobody read a log by hand, and nothing about any other user went near the reply. 😎

In Claude Code

A staff member creates an API key for their own user in the web console, and adds the staff-only tools to Claude Code.

claude mcp add --transport http smarter-operations \
  https://<your-smarter>/api/v1/mcp/smarter-operations/ \
  --header "Authorization: Token $SMARTER_API_KEY"

Keep the key in an environment variable, not in a file that’s committed. Every call is recorded under the key’s user.

How it works

The platform’s health, read live

platform_status reports the platform’s health at the moment it’s called, and starts with what looks wrong, in plain sentences. Then come the details.

• Pods, with their readiness, restarts, CPU and memory, and problems such as CrashLoopBackOff, a restart after OOMKilled, or Pending.
• Nodes, with their readiness, memory or disk pressure, cordons, CPU and memory, and whether each is a spot or an on-demand server.
• Kubernetes warning events of the last few minutes, in the platform’s namespace.
• Backing services. The web console dashboard’s own health checks of the database, cache, broker and workers, run fresh, plus each Celery queue’s backlog and the database’s connections, running threads and slow queries.
• Upstream status. The public status pages of OpenAI, Anthropic and Mistral, and the AWS status feeds for EKS and EC2 in the platform’s region.

eks_status reports the cluster as AWS EKS sees it. That’s the control plane’s status, version and health issues, each node group’s size, instance types and issues, each add-on’s version and whether a newer one exists, and the EKS insights that aren’t passing, such as readiness for the next Kubernetes version.

Both read only what’s already there. That’s Kubernetes and metrics-server (the numbers kubectl top shows), the EKS API, the platform’s own database and broker, and public status pages, which need no AWS account or paid support plan. No cloud monitoring is added, and so no cost. When one part can’t be read, the result names it and reports the rest. A status page that can’t be reached is reported as unknown, never as healthy, because an agent that says “OpenAI looks fine” when it simply couldn’t ask is worse than one that says nothing.

Three days of pod logs

kubectl logs shows only what a container wrote since it last started, and nothing at all of a pod that’s gone, such as one on a spot node that was replaced an hour ago. So the platform’s Helm chart keeps its own copy of the logs, inside the platform’s Kubernetes namespace.

Grafana Alloy, a log collector, runs as a single pod and tails the logs of the app, both Celery workers, the scheduler and MariaDB through the Kubernetes API. Its Role lets it read only the pods of its own namespace, so several environments can share one cluster. It labels each line with its component, pod and container, and replaces email addresses, IP addresses, bearer tokens, API keys, JWTs and passwords with redaction markers before the line is stored. Loki, a log store, keeps 72 hours of lines on its own 10 GB volume. Nothing goes to S3 or CloudWatch.

Three tools read the logs.

• summarize_log_errors is where to start. It groups a period’s errors, and optionally its warnings, by component, level and signature, which is the message with its timestamp, numbers, IDs and quoted values taken out. The same error from 200 requests becomes one row, with its count, its pods, when it was first and last seen, and its latest line.
• search_logs finds the lines around a problem, by component, pod, text, pattern and level, newest first. With no filters, it works like tail.
• list_log_sources lists the components and pods that logged recently. A component that’s missing has written nothing, often because its pods are down.

Loki has its own query language, LogQL. An LLM could write LogQL, but anything an LLM writes can be steered by what it read, and anyone can put text into a log line. So the tools take structured filters instead, and the log service builds the LogQL itself and escapes every value. A tool call can’t inject a query, and can’t ask for more than the 72 hours the store keeps.

For the same reason, every result that quotes log lines starts with a notice that what follows is data, not instructions, and the server’s instructions tell the agent never to follow a request found in a log line, whoever appears to have written it.

The same logs feed the daily pod log review in the smarter-sh/automation repository, which reads every pod’s log of the day from Loki, including pods that are gone.

One person’s problem

A failed sign-in to the web console is logged once, at WARNING, with its reason. That’s a wrong password, an unknown account, an inactive account, an account with no password (one that signs in with Google or GitHub), an invalid form, or an error. The email address is never logged, because people sometimes type their password into the email field. Each line carries a keyed hash of the address instead.

def login_identity(email: Optional[str]) -> str:
    normalized = (email or "").strip().lower()
    if not normalized:
        return "none"
    return salted_hmac(LOGIN_IDENTITY_SALT, normalized, algorithm="sha256").hexdigest()[:LOGIN_IDENTITY_LENGTH]

From account/login_audit.py. The hash is keyed with the platform’s secret key, so nobody without it can turn a list of addresses into a list of hashes. The platform can, which is how user_support_snapshot finds one person’s attempts.

user_support_snapshot takes an email address and reports what the platform knows about that person’s problem. It covers whether a user has the address and is active, how they can sign in, their Smarter accounts, their failed sign-ins of the last 72 hours counted by reason, and their failed requests and prompts of the last few days, from the platform’s journal, each with its error. Then it adds hints for staff, such as resetting a password, signing in with Google instead, or calling platform_status for a server error. It searches for failed sign-ins even when no user has the address, because the person may be typing a wrong one.

It also requires verified_by, which says how the staff member confirmed that the person who asked owns the address, and that goes into the audit trail too. Anyone can email a help desk and claim to be someone else, so the server’s instructions say never to take an address from the text of a request as proof of who sent it.

A reply that a person sends

The server’s instructions say what a reply to a user may contain. It may say whether the platform has a problem, whose problem it was, its cause in general terms, what the user can do, and when it started or was resolved. It never says anything about another user, never quotes the platform’s internals, and never says whether an email address has an account, unless the person who asked is verified as its owner. “There is no account with that address” tells a stranger who uses the platform.

check_reply enforces that on the agent’s draft. It finds other people’s email addresses, IP addresses, phone and card numbers and secrets with the same detectors as Smarter’s Guardrails, allowing the requester’s own address. It also has rules of its own for pod, node and host names, internal URLs, log lines, stack traces, exception names, version numbers and wording that gives away whether an account exists. Each finding says why a reply must not contain it.

It’s a safety net, not a guarantee. The agent drafts, check_reply checks, and a staff member reads the reply and sends it. The server never sends anything.

Who may use them

The staff-only tools have an endpoint of their own, /api/v1/mcp/smarter-operations/, with their own instructions and no cache, rather than being a hidden tier of the public tools. A mistake in deciding who may see a tool can’t expose a private one, because the public endpoint doesn’t offer any.

Django’s User.is_staff flag means a user may sign in to the Django admin. In Smarter, which hosts many accounts on one platform, it means a user administers their own account, so every customer account’s admin has it. Neither it nor is_superuser says that someone runs the platform, so the server asks a narrower question.

def is_platform_staff(user: Optional[object]) -> bool:
    if not isinstance(user, User):
        return False
    if not user.is_active:
        return False
    if not user.is_staff:
        return False
    smarter_account = smarter_cached_objects.smarter_account
    in_smarter_account = UserProfile.objects.filter(user=user, account=smarter_account).exists()
    return in_smarter_account

Trimmed from account/utils.py. The platform’s staff are the staff users of the Smarter account. The check is never cached, so a person who leaves the staff loses access on their very next call.

From Claude Code, the API key says who is calling. From Smarter Sysadmin it’s harder, because an MCPClient sends one credential, from a Secret, the same for everyone who chats, and a prompt runs as its LLMClient’s owner. A staff API key in a Secret would record every call under one service user rather than a person. So the platform vouches for the person itself.

• The prompt API marks the person who sent the prompt as the acting user, for the length of the prompt.
• When an MCPClient calls this platform’s own staff-only endpoint, and no other server, it adds an X-Smarter-Acting-User header. Its value is the user’s id, signed with the platform’s secret key, and it expires after five minutes.
• The server verifies the signature, then checks that the user is platform staff.

def acting_staff_user(value: str) -> tuple[Optional[User], HTTPStatus]:
    verified = verify_acting_user(value)
    if verified is None:
        return None, HTTPStatus.UNAUTHORIZED
    if verified.user_id is None:
        return None, HTTPStatus.OK
    user = User.objects.filter(pk=verified.user_id).first()
    if not is_platform_staff(user):
        return None, HTTPStatus.FORBIDDEN
    return user, HTTPStatus.OK

Trimmed from protocol/auth.py. A manifest can’t set the header, and the MCPClient compares the scheme, host, port and path of its endpoint with the platform’s own, so a server elsewhere whose path looks the same never learns who is chatting. The user_id is None branch is for requests the platform makes for nobody in particular, such as the background refresh of an MCPClient’s tool list. Such a request may list the tools, so the MCPClient shows as connected, but it can’t call one.

Every call, including one that fails, becomes a row of the audit trail, which superusers read in the Django admin as Operations Tool Calls. A row holds when, the staff member, the tool, the ticket and the other arguments, and whether it failed. The rows are read-only, and keep the staff member’s username after the user is deleted. Results are never cached, by the server or by the MCPClient, because an answer depends on who asked and when, and every call has to reach the audit trail.

Limits

• Log lines don’t carry a request ID yet, so following one request through the platform takes a few searches.
• The logs reach back 72 hours, and no further.
• check_reply finds the mistakes it was taught to find, which is why a person still sends the reply.

Takeaways

• Give the people who run a platform a chat that reads its live state. Most help desk and on-call questions are answered from data the platform already has.
• Put private tools at an endpoint of their own, with their own login and instructions, rather than a hidden tier of public ones.
• Define “staff” yourself. When a framework’s flag means something slightly different in your product, write a check whose name says what it means.
• Let the platform vouch for the person chatting, with a short-lived signature sent only to your own server, and record every call under a person, with a reason.
• Redact logs before you store them, and don’t let an LLM write a query language.
• Report unknown as unknown, and end at a draft that a person sends.

Closing thoughts

A sysadmin’s first questions are almost always the same. Is it down? What broke? What happened to this one person? Smarter’s MCP server now answers them in a chat, from the platform’s own pods, cluster, services, logs and accounts, for the staff and no one else, with a person’s name and a reason on every call.

• The documentation of the MCPServer’s staff-only tools
• The smarter_sysadmin manifest
• The smarter_operations MCPClient manifest
• The mcpserver app’s source code
• v0.41 Release: A Help Desk MCP Server for the People Who Run Smarter
• Introducing the Smarter MCP Server
• How Smarter’s MCP Server Searches Its Own Code, Documentation and Blog