Seven CI/CD Practices That Let Smarter Delegate to AI Agents

Every Smarter repository ships automatically through GitHub Actions and semantic-release, including multiple packages published on PyPI and npmjs. A merged pull request becomes a version number, a CHANGELOG, a GitHub tag, a set of built and signed artifacts, a deployment and an updated docs site, and nobody types any of it. That is a lot of automation for a community project that a handful of people maintain, and it’s part of what makes it possible to hand real work to AI agents like Claude Code and its peers.

Smarter’s CI/CD assumes that the software it ships is a twelve-factor app, and it is designed to keep it that way. The Smarter platform, a Django application for building and running LLM applications, adopted the twelve-factor methodology in its fourth architecture decision record, ADR-004, which requires every pull request to respect all twelve factors. The twelve factors make an application predictable enough to verify automatically, and the pipeline is what verifies it, on every change.

Importantly, the amount of work you can safely delegate, to a pipeline or to an AI coding agent, is capped by how much your pipeline can prove on its own. Each of the seven practices in this post lets a machine do something that a maintainer used to do by hand and check by eye. The time saved is the obvious payoff. The quieter one is consistency. Every contributor, human or agent, benefits from the same consistent developer experience, because the same pipeline enforces the same rules on every change.

The repositories

The languages and the package registries differ. The practices don’t.

1. Set the test coverage bar high enough to delegate against

Google’s testing blog describes 60% code coverage as acceptable, 75% as commendable and 90% as exemplary. In every Smarter repository, 90% is the minimum. The three Python packages fail their CI runs when line and branch coverage falls below 95%. The Node.js SDK fails below 90%, and the command-line tool fails when any one of its Go packages falls below 90%. The platform’s test suite covers about 97% of its code.

My view of how much testing is enough has changed since AI coding agents arrived. High coverage used to be a point of pride. Now it’s what makes delegation possible, because a test suite is the only definition of correct behavior that an agent can actually run. When an agent changes a well-tested module, the tests tell it within minutes whether it broke something, and it fixes the mistake before anyone else sees it. When an agent changes code that no test exercises, nothing tells it whether the change is right, and the change passes CI anyway.

Contract tests

Line coverage shows that code ran. It doesn’t show that the code still agrees with the other programs that depend on it. The platform publishes its REST API as an OpenAPI document, and each client repository keeps a copy of that document. Contract tests check each side against it.

• In the platform, a test fails when the OpenAPI document committed to the repository no longer matches the API, or when an API route is missing from it.
• In the SDKs, a test fails when the platform offers an API operation that the SDK doesn’t call, unless the operation is listed as deliberately unused, with a reason.
• In the command-line tool, a test runs every command against a mock server that answers only the routes in the OpenAPI document.

These tests catch the changes that unit tests miss, such as a route that the platform renamed but a client still calls by its old name.

High coverage turns “the agent says it works” into “the test suite says it works”.

2. Write the guardrails where the agent will read them

High coverage tells an agent when it has made a mistake. It doesn’t tell the agent what it must never do. For that, every Smarter repository has a CLAUDE.md file at the root of its source code, and a directory of Agent Skills, which are short Markdown files of instructions that an agent loads for a particular kind of task. Several of the most important rules in these files are about CI/CD.

• Agents never run git commands that change the repository. When asked to “write a git commit”, an agent prints the command for a person to review and run. The commit message skill adds that frustrated or emphatic wording doesn’t change the rule, because agents have broken it in exactly that situation.
• Agents never start a release. The command-line tool’s CLAUDE.md forbids make release, gh workflow run and the version script by name, and the release skills forbid editing a changelog or a version file by hand.
• Tests never touch real cloud infrastructure. The Docker containers in which developers run the platform hold working credentials for a real Kubernetes cluster and real DNS records. The infrastructure safety skill requires every test to replace those calls with fakes, and gives the reason in one sentence. “An unguarded test once created two real EKS node groups, one of them a GPU instance.”
• Nothing private goes into a repository. The repositories are public, so their code, tests, documentation and skills must not contain keys, tokens, account IDs, private hostnames or customer data.

The third factor, from both sides

The third twelve-factor principle is to store configuration in the environment, never in the code. The platform reads every setting from environment variables, through a typed settings class that validates them. The CI workflows hold up their end of that bargain. The platform’s test workflow gives the tests the LLM provider API keys that they need and no AWS credentials, so a test that gets past its fakes still can’t reach the cloud from CI. The deployment workflow reads the production database, email and Django secrets from Kubernetes secrets at deployment time, and passes them to the application as environment variables. None of them is ever written into the repository.

Guardrails are source code

The README of the command-line tool’s skills directory says “Review a change to this folder as carefully as a change to CI.” A skill that tells an agent to skip a check, or to run a release command, does as much harm as a workflow that does the same thing. So the skills live in the same repository as the code they govern, and they are reviewed and versioned like that code. Every new agent session reads the current copy.

Agents write the code, the tests, the documentation, the commit messages and often the workflows themselves. People keep the actions that make a change official, which are committing it, merging it and, where publishing is manual, publishing it.

A rule that isn’t written down will eventually be broken by an agent that has never heard of it.

3. Make the agent lint and test its own work first

CI is the slowest and most expensive place to discover a missing import. The platform’s Python test suite builds the whole platform in Docker and takes about 13 minutes. An agent that pushes a typo and waits for CI to find it wastes those 13 minutes and a reviewer’s attention, and then does it again with the next typo. So every Smarter repository tells its agents to find these mistakes on the developer’s own machine first, where checking takes seconds.

The pre-commit hooks define clean code

Each repository has a pre-commit configuration, a list of formatters, linters and security checks that run before every commit. They include black, isort, flake8 and bandit for Python, Prettier and ESLint for TypeScript, gofmt and golangci-lint for Go, and checks for leaked credentials and private keys. The code quality skill makes these hooks the only standard that counts. If the hooks pass, the code meets the standard, and if they fail, the agent fixes the code. It never skips a hook to get a change through. Before an agent reports its work as finished, it runs the hooks on every file it changed.

git diff --name-only | xargs pre-commit run --files

Run the tests and read the result

The platform’s CLAUDE.md requires an agent to run the tests itself after every change, inside the Docker container, and to report failures honestly, including failures that were there before it started. Running the tests in the container is the tenth twelve-factor principle, dev/prod parity, applied to testing. The developer’s container, the CI test machines and the production pods are all built from the same Dockerfile, so a test that passes on the developer’s machine is testing the software that will run in production.

The testing skill adds that a coverage report is not a test result. An agent must read the counts of passed and failed tests, because a run in which half the tests crashed can still print a coverage table.

Every repository defines “done” the same way. The change follows an existing pattern in the code, tests cover it, the agent ran those tests, the pre-commit hooks pass on the changed files, and the agent’s summary says what it changed, what it verified and what it skipped.

CI checks everything again

None of this replaces CI. The Python packages’ CI runs every pre-commit hook on every file in the repository, so a skipped hook still fails the pull request. Another hook runs commitlint on every commit message, which matters for the next practice.

The platform’s test workflow also avoids repeating work. It computes a fingerprint of the source files that each test suite depends on, and skips a suite that has already passed with exactly those files.

- name: Fingerprint the tested files
  id: keys
  shell: bash
  run: |
    fingerprint() {
      git ls-files -s -- "$@" | git hash-object --stdin
    }
    PYTHON=$(fingerprint smarter/smarter smarter/manage.py smarter/requirements ...)
    REACT=$(fingerprint smarter/react .github/workflows/test.yml)

A pull request that changes only the platform’s React web console doesn’t wait for the Python test suite, and the reverse is also true.

The cheapest CI run is one that never reports a mistake the agent could have caught itself.

4. Let the commit messages choose the version

Nobody chooses the version number of a Smarter release. Every repository uses Conventional Commits, a convention in which each commit message begins with its type, and semantic-release reads those messages to work out the next version. A fix commit produces a patch release, a feat commit produces a minor release, and a commit marked BREAKING CHANGE produces a major release. Commits that change only documentation, tests or CI configuration produce no release at all. semantic-release then writes the changelog, tags the commit and publishes a GitHub release.

Work goes to the alpha branch first, where each releasable commit produces a prerelease such as v0.36.27-alpha.1. A pull request from alpha to main produces the release itself, v0.36.27. The platform’s build workflow then merges main back into alpha, so that the branches don’t drift apart. This is the first twelve-factor principle, one codebase with many deploys. The alpha, beta and production environments all run builds of the same repository, distinguished only by their version tags and their configuration.

Only the pipeline writes the version

Each repository has a single place where the version number enters the source code, and only semantic-release writes to it. In the Python packages it’s a small __version__.py module that the package build reads. The Node.js SDK and the command-line tool each have a version file of their own. In the platform repository, every version number that is committed is a placeholder, and the build replaces it with the release tag, so there’s never a stale version number for someone to forget to update.

Why this matters when agents write the commits

Agents write most of the commit messages in these repositories, so the commit message skill teaches them the rules that decide the version. A feat commit causes a minor release, so the skill says that feat must be rare, and that a commit that could be either is a fix. The skill also forbids agents from writing the release commits that belong to semantic-release. The commitlint hook then rejects any message that doesn’t follow the convention, before the commit is created.

The version history that results means something to users. A user who sees 0.36.26 become 0.36.27 knows that the release contains fixes, and a user who sees 0.36 become 0.37 knows to read the release notes.

5. Build each release once, from its tag, and verify what was published

The fifth twelve-factor principle is to separate the build, release and run stages strictly. The build stage turns a version of the code into an executable artifact. The release stage combines that artifact with an environment’s configuration. The run stage starts it. Smarter’s pipelines treat each release as a GitHub tag, and build everything that is published for a release from that tag, exactly once.

The platform’s build workflow checks out the release tag rather than the commit that started the workflow. semantic-release adds its own commit after that one, so building the earlier commit would produce the previous version. The workflow builds one Docker image, pushes it to Amazon ECR, and copies that same image to Docker Hub instead of building it a second time. It packages the Helm chart from the same tag, signs it with cosign, and verifies the signature before it attaches the chart to the GitHub release. The image never changes after it’s built. Each environment’s configuration is added later, at deployment.

Rehearse, publish and install it back

The three Python packages share one release workflow, and it’s the most thorough in the project.

The workflow runs the full CI workflow first, and then semantic-release. It rebuilds the package from the new tag and checks that the version inside the package matches the release. It uploads the package to TestPyPI, a separate test instance of the Python Package Index, installs it from there, and checks the version again. Only then, and only for releases from main, does it publish to PyPI and install the package back from PyPI as a final check. When it installs the package back, it reads the package’s dependencies from the package’s own metadata rather than from a list in the workflow. That keeps the second twelve-factor principle, explicitly declared dependencies, true of the published package and not just of the repository.

Releasing and publishing happen in one workflow, so that a version can’t be tagged on GitHub without also reaching PyPI.

No stored publishing tokens

The Python packages publish with PyPI’s trusted publishing, in which GitHub proves to PyPI which workflow is uploading, so there’s no PyPI token to leak. The platform’s two npm packages publish the same way, and only when their source code folder has changed since the last published version. The documentation sites sign in to AWS by the same mechanism, as a role that can write to one storage bucket and nothing else.

Two repositories keep a person in charge of publishing. The Node.js SDK is published with make release, which runs the tests and the build first and refuses to run on any branch except main. The command-line tool is published by a workflow that a maintainer starts by hand, which builds the binaries and publishes them to GitHub, Homebrew, an Ubuntu package archive, Chocolatey and Docker Hub. In both repositories, the version number, the changelog and the checks before publishing are still automatic, and the skills forbid agents from starting the publication.

A build that can’t be reproduced from its tag isn’t a release. It’s a snapshot of somebody’s afternoon.

6. Chain the deployment to the release, and make each step a gate

The platform reaches production through four GitHub Actions workflows. Each one starts only after the one before it succeeds.

• Test runs on every pull request to main. The main branch requires its checks to pass before a pull request can be merged.
• Build runs when the pull request is merged. It runs semantic-release, builds the Docker image and the signed Helm chart, and passes the release tag to the next workflow.
• Deploy runs when Build succeeds. It confirms that the release’s Docker image exists, then installs the release’s Helm chart on the production Kubernetes cluster in Amazon EKS, with the production configuration.
• Publish runs when Deploy succeeds. It builds the documentation for the version that is now running, and uploads it to docs.smarter.sh.

Deploy is the release stage of the fifth factor. It combines the image that Build made with the production environment’s configuration, and Kubernetes runs the result. After each deployment, Helm starts a one-off Kubernetes job that runs the platform’s manage.py initialize_platform command, which loads the example resources and seed data. That’s the twelfth factor, admin tasks run as one-off processes. The platform’s CLAUDE.md forbids putting any initialization step anywhere else, whether in the Helm chart, the Makefile or a CI workflow, so the deployment pipeline never grows its own private copy of the platform’s setup logic.

If the merged commits don’t call for a release, for example because they only change documentation or tests, Build creates no release and nothing is deployed. Production changes only when its software does.

The chain behaves the same way every time. There’s no deployment checklist for a maintainer to follow and possibly get wrong, and a release at midnight goes through exactly the same steps as one on a Tuesday afternoon.

When a deployment is a chain of gates, a failure stops it early and in one place.

7. Publish the documentation with the release

Documentation falls out of date when it’s published on a different schedule from the code that it describes. In every Smarter repository, the documentation is published as part of the release, and built from the release tag.

• The platform’s Publish workflow builds the Sphinx documentation after each deployment, and uploads it to docs.smarter.sh.
• Each Python package publishes its own documentation site as the last step of its release workflow, after PyPI has the new version, and then checks that the site shows the new version number.
• The Node.js SDK generates its API reference into the platform’s documentation, so that readers have one site to search instead of two.

The documentation builds treat every Sphinx warning, such as a broken cross-reference or a missing page, as an error that fails the build. The hosting for each Python package’s documentation site is defined in Terraform in that package’s own repository, and CI validates the Terraform on every pull request.

When the documentation is part of the release, out-of-date documentation stops being a chore and becomes a failed build.

One pattern in every repository

Each practice is useful on its own. They pay off most because every repository applies them the same way. smarter-python’s CLAUDE.md names the cache-results repository as the model for every Smarter Python package, and the three packages’ release workflows are the same file. The pre-commit hooks, the commitlint configuration, and the names and structure of the skills directories are nearly identical across the repositories. The Python packages’ release skills share one rule, “Every make target has a CI counterpart, and the other way round.”

This consistency matters to two kinds of contributor.

For people, it means that someone who has fixed a bug in cache-results already knows how to fix one in smarter-types. The test command, the pre-commit hooks, the commit message format and the release process are the same, and no repository depends on knowledge that only one maintainer has.

For agents, it matters even more, because agents copy the patterns that they find. An agent working in a repository where every workflow, skill and Makefile follows one pattern produces work that follows it too. When the platform’s API changes, the contract sync skill gives the same steps in each of the five client repositories, and tells the agent not to update one repository and leave the others behind. A monthly maintenance skill does the same for dependency updates.

How much time it saves

The platform has made 421 releases since its first GitHub tag in February 2024, not counting prereleases. Suppose that a release done by hand takes half an hour of a maintainer’s time, to update the version number, write the CHANGELOG, tag the release, build and push the Docker image, package and sign the Helm chart, deploy it, and build and publish the documentation. That estimate is low, since building the documentation alone takes about 20 minutes.

421 releases × 0.5 hours ≈ 210 hours, or a little more than five 40-hour weeks, for one repository.

The estimate leaves out the other repositories, each with its own releases and its own documentation site. More importantly, it leaves out the releases that wouldn’t have happened at all. If the world were manual then we’d batch the fixes into a few big releases, and users would wait for them, like the way things worked in the 20th century.

Takeaways

These are in rough order of payoff.

• Adopt the twelve factors first, and design the pipeline to enforce them. Configuration in the environment, one codebase with many deploys, and separate build, release and run stages are what make an application predictable enough for a pipeline to verify.
• Raise test coverage until it can carry delegation. Treat 90% as the minimum, make CI fail below it, and add contract tests for anything that other programs depend on.
• Write the guardrails where the agent reads them, in CLAUDE.md or a skill, in the repository, and review them as carefully as CI configuration. Keep committing, merging and publishing for people, and say so explicitly.
• Make the agent lint and test its own work with the same pre-commit hooks that CI runs, in the same container image that production runs, and require it to report the counts of passed and failed tests.
• Let Conventional Commits and semantic-release choose the version, write it in exactly one generated file, and check the commit message format with a hook.
• Build each release once, from its GitHub tag, rehearse the upload, install the package back to check it, and publish with short-lived OIDC credentials instead of stored tokens.
• Chain the deployment to the release, so that each workflow starts only when the one before it succeeds, and no release means no deployment.
• Publish the documentation with the release, with warnings treated as errors.
• Enforce a consistent pattern in every repository. Each repository that follows it costs a fraction of the first, and every contributor, person or agent, gets the same experience.

Closing thoughts

None of these practices is new. The twelve-factor methodology, semantic versioning, high test coverage, signed artifacts and continuous deployment were all good ideas long before AI coding agents existed. What has changed is how much they’re worth. An agent can produce a month’s worth of changes in an afternoon, and a project can absorb that only if every change is checked, versioned, built, deployed and documented by a pipeline that doesn’t get tired and doesn’t skip steps. The automation that once saved a maintainer half an hour per release now decides how much an agent can be trusted with.

The other half of the payoff is consistency. When every repository follows the same pattern, and every rule is written down, contributing to Smarter is the same experience in every repository. That holds for a student opening a first pull request, for a maintainer returning after a month away, and for an agent starting a new session with nothing but the repository to read. ✨

Until next time, happy Smartering!

• Smarter on GitHub
• The Smarter documentation
• How Smarter applies the twelve factors, and ADR-004
• The platform’s Agent Skills
• cache-results’ release workflow, the model for Smarter’s Python packages
• Part 1: Maintaining Long-Term Code Quality With Agentic AI
• The Smarter blog