Leveraging Automations to Keep Smarter Healthy

Smarter stays healthy because of four kinds of automation. GitHub Actions workflows test it, Dependabot keeps its dependencies current, pre-commit hooks check every commit, and coverage thresholds fail any test run that leaves too much code untested. The first three are set up separately in each of The Smarter Project’s 29 active repositories. The fourth kind is a repository of its own, smarter-sh/automation, which holds the scheduled GitHub Actions workflows and the Python scripts that look after all of the other repositories at once.

None of these is new on its own. What’s worth writing about is how they fit together. Each one does a specific job for reliability, and most of them do more than one. This article breaks the automations out by the five jobs they do.

• Eliminating bugs before they’re merged.
• Surfacing unintended consequences that no single repository’s tests can see.
• Nudging about things that have a shelf life, like passwords and old pull requests.
• Checking and re-checking that things are where they’re supposed to be, in the state they’re supposed to be in.
• Verifying the developer experience of installing Smarter from PyPI, npm and the Go module proxy.

Then it estimates how much manual work each automation replaces, because the answer is very uneven, and the unevenness is instructive.

Two layers of automation

The automations sit in two layers, and the difference between them matters.

The per-repository layer lives inside each repository and acts on that repository’s own changes. Its pre-commit hooks run on every commit, its tests and coverage checks run on every pull request, and Dependabot opens pull requests against its dependencies. This layer is good at catching what a change breaks in the repository that changed.

The organization layer is smarter-sh/automation. It has no application code. Each of its workflows checks out one or more of the other repositories, tests or inspects them, sometimes opens a pull request or an issue, and posts one message to Slack. This layer is good at what the first one can’t see, which is how the repositories relate to each other, and how they drift when nobody is changing them.

The scheduled jobs are spread across the week, so that no single day’s Slack channel is a wall of reports.

Eliminating bugs

The cheapest bug is the one that never reaches a pull request, and the next cheapest is the one that never reaches main.

pre-commit hooks

Most of the organization’s repositories have a pre-commit configuration, with between 17 and 30 hooks each. They run on the developer’s machine at git commit, and in most repositories again in CI, so a commit that skipped them locally still gets checked. The platform repository’s list, trimmed, is typical.

repos:
  - repo: https://github.com/psf/black
    hooks:
      - id: black
  - repo: https://github.com/PyCQA/flake8
    hooks:
      - id: flake8
  - repo: https://github.com/PyCQA/isort
    hooks:
      - id: isort
  - repo: https://github.com/PyCQA/bandit
    hooks:
      - id: bandit
  - repo: https://github.com/pre-commit/pre-commit-hooks
    hooks:
      - id: detect-private-key
      - id: detect-aws-credentials
      - id: check-merge-conflict
      - id: check-yaml

Most of these hooks are about formatting, and formatting isn’t reliability by itself. What it does is remove noise. When black, isort and Prettier decide the layout of every file, a diff shows only what changed in meaning, and a reviewer, human or AI, spends their attention on that. The other hooks catch real bugs. flake8 finds unused variables and names that don’t exist, bandit finds risky calls such as shell=True, and detect-private-key stops a key at the commit instead of in the public history of a public repository.

Tests, with coverage thresholds

Every pull request runs the repository’s tests in GitHub Actions, and in most repositories the run fails when coverage drops below a threshold. The threshold is set where the tools are configured, so it can’t be quietly forgotten. The Python SDK sets it in pyproject.toml.

[tool.coverage.report]
show_missing = true
fail_under = 95

The Node.js SDK sets it per measure in its Vitest configuration, thresholds: { statements: 90, branches: 90, functions: 90, lines: 90 }, and the command-line tool’s Makefile fails when any one Go package falls below 90%. The platform asks for at least 90% per Django app, and 95% where that takes moderate effort.

Coverage doesn’t prove that code is correct. What it does is guarantee that every line was run at least once by a test that someone wrote on purpose, which rules out the whole class of bugs that live in code nobody has ever executed, like the except branch with a typo in it.

Pod Log Review

Some bugs only show up in production. Every morning, the Pod Log Review job downloads the logs of the production platform’s pods, removes credentials and personal data from them, and has Claude Code read them in a checkout of the platform’s code. Claude opens an issue for each new problem, and may propose a small fix as a draft pull request. How the daily pod log review works covers it in detail.

Surfacing unintended consequences

A repository’s own tests answer one question, which is whether this change broke this repository. They can’t tell you that a change somewhere else broke it, or that nothing changed and it broke anyway.

Dependabot, run through the tests

Dependabot opens a pull request for each new version of a dependency, against each repository’s alpha branch. The platform’s configuration covers its npm packages, its GitHub Actions and its Helm chart.

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "monthly"
    target-branch: "alpha"
  - package-ecosystem: "github-actions"
    directory: "/.github"
    schedule:
      interval: "monthly"
    target-branch: "alpha"
  - package-ecosystem: "helm"
    directory: "/helm/charts/smarter"
    schedule:
      interval: "quarterly"
    target-branch: "alpha"

The value of each Dependabot pull request is less the update itself than the test run it starts. A new major version that changes behavior shows up as a failing check on one small pull request, with one cause, instead of as a mystery after a dozen updates have been bundled together.

Repository Tests, every night

A repository nobody has touched in a month can still break, because a package it installs, a registry it downloads from, or a GitHub runner image changed underneath it. Every night, the Repository Tests job starts the test workflow of every active repository on its default branch, waits for the runs, and posts one message listing which passed and which failed. A failure on a quiet repository is an unintended consequence of something outside it, and finding it overnight beats finding it on release day.

API Contract Sync

The platform publishes its API as a contract, a set of JSON Schemas plus an OpenAPI document. The Python, Node.js and Go SDKs, the command-line tool and the VS Code extension each carry a copy, and generate code from it. Every Monday, API Contract Sync updates each client’s copy to the platform’s latest release, regenerates the code, and runs the client’s tests. When the copy was behind, it opens a pull request. When the tests fail after the update, the pull request is a draft.

That draft is the point. It means a change to the platform broke a client, and it says so before anyone tries to release the client. 📖 The API contract explains what’s in it.

Nudges about things with a shelf life

Some things are fine today and wrong six months from now, without anyone changing them. A password ages. A pull request goes stale. A dependency falls behind. None of these produces an error, so nothing surfaces them except a reminder.

Secrets Age Review

Every Friday, the Secrets Age Review job lists every GitHub Actions secret, at the organization, repository and environment levels, that hasn’t been updated in six months. It reads only each secret’s name and last-updated date, never its value. The core of secret_age_review.py is a date comparison.

for item in items:
    name = item["name"]
    updated_at = item["updated_at"]
    if updated_datetime(updated_at) < cutoff:
        finding = {"scope": scope, "location": location, "name": name, "updated_at": updated_at}
        results["findings"].append(finding)

The check is simple, and so is rotating a key once you’re reminded to. The value is in the reminder, because “rotate the tokens every six months” is the kind of rule nobody remembers until the day a token leaks.

Stale Items Report

Every Thursday evening, the Stale Items Report lists the pull requests and issues that have been open for more than a month, across every repository. It never closes or comments on anything. An old pull request is usually either a good change that got forgotten, or a dead one that should be closed, and either way a person should decide.

Dependabot Merge

Dependabot’s pull requests have a shelf life too. Left open, they pile up, fall into conflict with each other, and stop being merged at all. Every Wednesday, the Dependabot Merge job goes through every open Dependabot pull request in the organization, oldest first, and decides what to do with each one from its state. The decision is one small function in dependabot_merge.py.

def decide(state: dict, conflicted_at_start: bool, auto_merge: bool) -> str:
    mergeable, merge_state = state.get("mergeable"), state.get("mergeStateStatus")
    if state.get("isDraft") or merge_state == "DRAFT":
        return "draft"
    if mergeable == "CONFLICTING" or merge_state == "DIRTY":
        return "closed" if conflicted_at_start else "rebasing"
    if mergeable != "MERGEABLE":
        return "unknown"
    if merge_state in ("CLEAN", "HAS_HOOKS"):
        return "merged"
    if merge_state == "UNSTABLE":
        return "failing"
    if merge_state in ("BLOCKED", "BEHIND"):
        return "auto_merge" if auto_merge else "blocked"
    return "unknown"

A clean pull request is merged, one still running its checks is set to auto-merge, and one that was already in conflict is closed with a comment that says how to get it back. A failing one is left open and listed in Slack, which is the nudge. It never merges with admin rights, so branch protection and the required checks still decide.

Checking and re-checking the state of things

Settings drift. Someone turns off a branch protection rule to get a hotfix in and forgets to turn it back on. A new repository is created without secret scanning. A workflow is copied from a blog post with an action pinned to @main. None of these breaks a test, and each one is easy to miss for a year.

Repository Security Review

Every Monday evening, the Repository Security Review job compares each repository’s settings with what they should be, and reports the ones that differ. It checks the default token permissions of GitHub Actions, the protection of the default branch, secret scanning, and Dependabot’s security updates. It changes nothing. A setting that was turned off on purpose stays off, and a person decides.

Workflow Security Review

Every Tuesday evening, the Workflow Security Review job reads every workflow file in every repository. It flags third-party actions that aren’t pinned to a full 40-character commit SHA, workflows without explicit token permissions, workflows that ask for write permissions, and jobs without a timeout. A tag like @v4 can be moved to different code by whoever owns the action, while a commit SHA can’t. A job without a timeout can run for six hours on a hung process.

Embed Version Sync and API Contract Sync

Two of the jobs check that each copy of something is the version it should be. API Contract Sync checks the clients’ copies of the API contract, as described above. Embed Version Sync does the same for Smarter Chat, the chat widget published to npm as @smarter.sh/ui-chat. The WordPress, Drupal, Shopify, Google Tag Manager and docs-site integrations each pin an exact version of it. Every Tuesday the job updates each pin to the latest version on npm, and opens a pull request when it changed. It also reports a repository whose alpha branch is current but whose latest release isn’t, as waiting for a release.

Checking the checker

The organization layer needs checking too. GitHub turns off a public repository’s scheduled workflows after 60 days without a commit, and only sends an email. A repository of cron jobs can go that quiet easily. In smarter-sh/automation, Dependabot Merge merging Dependabot’s own monthly updates to that repository keeps it active, and the README gives the one-line check for when every job stops at once.

gh api repos/smarter-sh/automation/actions/workflows \
  --jq '.workflows[] | "\(.state)\t\(.path)"'

A state of disabled_inactivity is the tell.

Verifying the developer experience

The last job is the one closest to the people who use Smarter. A release isn’t done when CI passes. It’s done when a developer can type pip install, npm install or go get and get the version that the release notes describe.

Published Package Health Check

Every Wednesday evening, the Published Package Health Check job finds every package the organization publishes, by reading each repository’s package manifests and release workflows. For each one it does three things.

• Compares versions. The latest stable version on the registry must match the latest stable GitHub release. A mismatch means a release whose publish step failed, or a package published by hand that never got a release.
• Installs it. It installs each npm and PyPI package at that exact version in an empty temporary directory, and downloads each Go module, the way a new user would. npm packages are installed with their install scripts turned off.
• Reports what failed to #general in Slack.

This catches the failures that every other check misses, because they happen after the code is right. Examples are a release that never reached the registry, a version on the registry that was never released on GitHub, or a Go module tag that the proxy can’t download.

The SDKs and plugins, from the user’s side

API Contract Sync and Embed Version Sync serve the developer experience too. A developer who uses the Python SDK gets a client that knows about the platform’s newest resources. Someone who installs the WordPress plugin gets the newest chat widget. Neither of them has to know that those are separate repositories with separate release schedules.

How much work each automation replaces

The automations don’t all save the same amount of work, and the differences are large. The chart estimates the hours a month that it would take to do each one by hand, as often as the automation does it.

The inputs are measured, and the minutes per task are my estimates.

That comes to about 116 hours a month, or nearly three working weeks.

Three things stand out. First, Dependabot and the pre-commit hooks dwarf everything else. They act on every change, and there are a lot of changes. Dependabot in particular does a job that nobody would do well by hand. Without it, dependency updates happen in a big, painful batch once a year, which is exactly when they break the most. 😬

Second, the weekly audits look expensive by hand because nobody would actually do them by hand. Reading 96 workflow files every Tuesday isn’t a chore that gets done slowly. It’s a chore that doesn’t get done at all. For these jobs, the honest comparison isn’t three hours against zero. It’s checking every week against never checking.

Third, the short bars aren’t the unimportant ones. The Secrets Age Review saves about two hours a month, which is the smallest saving on the chart apart from the Stale Items Report and the fact of the day. But the consequence of the thing it reminds you about, a leaked token that never expired, is among the worst on the list. Labor saved and risk reduced are different measures, and a job can be worth running for either one.

The chart leaves out coverage and the test suites, because they don’t replace a chore. They prevent bug hunts, and a bug hunt that never happened can’t be timed.

What keeps the automations safe

Automation that writes to other repositories is itself a risk, so the jobs in smarter-sh/automation follow a few rules.

• Pull requests, not pushes. A job never pushes to main, never force-pushes a branch it doesn’t own, and never merges with admin rights. Its changes go through pull requests into alpha, and run that repository’s own tests.
• Reports, not changes. The audit jobs read and report. They use a separate read-only token, and their public run summaries on GitHub contain counts only, because the repository is public.
• Dry runs. Every job has a dry-run input that checks and reports but writes nothing to GitHub and posts nothing to Slack.
• Tested scripts. The logic is in Python scripts that use only the standard library, with 180 unit tests, none of which reaches GitHub, Slack, npm or AWS. Every pull request to the repository runs them, along with actionlint, which type-checks the workflows and runs ShellCheck on their shell steps.

Each job also writes one Slack message per run, with a link back to the run. A job that fails silently is worse than no job at all, because it looks like coverage.

Takeaways

• Turn on Dependabot and pre-commit first. They replace the most manual work, they cost almost nothing to set up, and everything else builds on them.
• Put coverage thresholds in the tool’s configuration, so a test run fails below them. A threshold that lives only in a wiki is a suggestion.
• Test what nobody changed. A nightly run of every repository’s tests on its default branch finds the breakage that comes from outside.
• Give scheduled jobs their own repository once there are more than a few. They act on many repositories, and they don’t belong in any one of them.
• Report before you change. Start an audit as a report. Make it act only once you trust what it finds.
• Judge a job by risk as well as hours. Some of the cheapest checks guard against the most expensive failures.

Closing thoughts

Each automation here does a small, specific thing, and none of them is clever. Together they make sure that a bug is caught at the commit, a broken dependency at its pull request, a quiet breakage overnight, a drifted setting within a week, and an expiring password before it matters. That’s what keeps 29 repositories healthy with a team that is mostly one person and a few AI agents. Each job’s header comment says what it does and which secrets it needs, so they are easy to copy into another organization.

• smarter-sh/automation on GitHub
• Introducing smarter-sh/automation
• Smarter on GitHub
• Smarter documentation