IMPLEMENTATION COMPANION

The Coding Harness

The phase-by-phase build order — L1-0 through L3-6.

By Charlcye Mitchell — Director of Engineering Operations, agentic systems architect, field notes writer.

THE WHOLE PAPER IN FOUR LINES

  1. This is the three-level build from Context, Loops, Graphs end-to-end: a single source of truth (Level 1), at least one production loop (Level 2), and a self-improving coding harness on top of them (Level 3). Each level pays for itself alone; every phase inside a level produces value alone.
  2. Build in phases L1-0 → L3-6. Skipping produces confident garbage — in the coding harness that means confident garbage in production code, which costs more than confident garbage in a draft email.
  3. You need no new infrastructure. GitHub, git, CI, and Claude Code carry the whole system. The load-bearing decisions are the ones you’d think are boring.
  4. Autonomy is earned per ticket-class, monitored continuously, and revoked on failure. The humans aren’t in the loop as a courtesy. They’re the reason it works.

Who this is for

You’ve read Context, Loops, Graphs and you want the build order for all three levels — the context store, the first production loop, and the coding harness that sits on top of them. You’re an engineer — or a founder with the keys — at a five-to-fifty-person company. Claude Code is available to the team.

The levels build on each other. Do not skip. Building the coding harness before you have a knowledge base is how you build an agent that confidently opens PRs that contradict your own ADRs.

How to read this

There’s an engineer’s read and an agent’s read. Both work.

Engineer: you’ll implement it yourself. Skim “The infrastructure choices that actually matter” below first — it explains why the boring parts stay boring across all three levels. Then read the Level 1 build carefully; Levels 2 and 3 unlock from there.

Agent: hand this document to Claude Code with access to your repos. There’s a script at the end — inventory the gaps against your codebase, present them to a human, then implement phase-by-phase from L1-0 to L3-6. Every phase has a readiness gate; stop at each one and get explicit sign-off.

The infrastructure choices that actually matter

Every decision in this build has a shiny alternative someone is being paid to sell. Here’s why we picked the ones we did. Disagree specifically if you want — but understand what these choices are doing before swapping them out. Every choice below applies to all three levels; the coding harness inherits every decision the context store and the first loop already made.

GitHub, not Obsidian (or Notion, or Confluence)

The context store is a git repo. Not a wiki, not a knowledge platform, not a Notion workspace, not an Obsidian vault. This is the single most consequential choice in the system, and it looks the most boring.

What git gives you that a wiki doesn’t:

  • CODEOWNERS. A one-line file that says positioning changes need the head of marketing, invariants need the eng lead — enforced by CI, not by hope. In Obsidian, ownership is a convention people forget within a quarter.
  • Onboarding and offboarding through org membership. A new hire gets access with one gh command. A departure loses access with another. You don’t discover six months later that the former head of sales still has edit rights to the pricing doc.
  • Pull requests as the accuracy gate. Every change to shared truth passes through review. Crucially, this is the same mechanism agents use to write to the same store — so human freshness enforcement and agent-write safety are one system, not two.
  • CI as the enforcement surface. Frontmatter checks, index checks, staleness detection all run as required checks. In Obsidian, checks are a plugin you asked one person to install.
  • Diffs, branches, and history by default. You can see what the doc said the morning of an incident. You can revert. You can blame. Wiki edit histories are a courtesy; git history is a system.

The one thing wikis are genuinely better at is search UI. If your team keeps hitting that pain, you don’t have a wiki problem — you have an index problem. Fix the folder briefs before you migrate.

GitHub Actions, not LangGraph (or Temporal, or Airflow)

Every stage in every level of this build is a workflow. Every handoff is a commit. Every approval gate is a pull request. The orchestrator is CI.

You will be told this is naive. It isn’t. It’s what CLG’s Level 3 argued for, and it holds up in production because:

  • Your team already knows it. Debugging a LangGraph run means learning LangGraph’s state model. Debugging a workflow means reading YAML and clicking into the run — every engineer on the team can do it on day one.
  • State is inspectable. Every stage’s output is a file, diffable and versioned. LangGraph’s state is in a database you’ll build a UI for.
  • Approval gates are already a review UI. A PR is a shared, familiar, comment-able surface. Rebuilding one inside an orchestration framework is its own quarterly project.
  • Access control is free. GitHub already knows who can approve what. Adding orchestration adds a second permission model to keep in sync.
  • Audit is free. Every action is a workflow-run URL plus a commit SHA. At 2 a.m. you’re not paging into someone else’s dashboard.

You’ll want a real orchestrator when state files contend so often that rebase-retries are your bottleneck, when queries span too many hops, or when inter-stage latency actually matters. Adopt one then, with data. Not because a diagram suggested you would.

Frontmatter + a compiled graph.json, not a graph database

The knowledge graph is compiled from YAML frontmatter on documents that already exist. It ships as a graph.json file in the repo. No Neo4j, no separate node store, no vector DB — yet.

Why not just spin up Neo4j and get proper graph queries?

  • Two sources of truth is one too many. If the graph lives outside the docs, they drift. They always drift.
  • Frontmatter is human-editable in the same PR as the decision. An ADR author declares depends_on: [D-042] while writing the decision. In a graph DB, the author has to remember to update a second system. Half the time they won’t.
  • CI compiles it on push. scripts/build-graph.py regenerates and validates on every merge; broken edges fail the build. That is the entire graph platform.
  • You can grep it. When something is off, jq on graph.json finds it in a minute. Debugging a graph DB query starts with connecting to the graph DB.

Graduate to a graph store when you have thousands of nodes and traversal queries actually take too long. That will be later than you think.

git as the state store, not Redis (or DynamoDB, or a queue service)

Pipeline state — the outbound loop’s leads/ directories, the coding harness’s queue claims, autonomy ledger, and release findings — lives in files in the repo that owns them. JSON. Writes go through a composite action (or a rebase-and-retry loop) that does read-hash → write → rebase-on-conflict.

  • Every state change is versioned and diffable. You can look at last Tuesday’s queue and see who claimed what.
  • Rollback is git revert. Not a runbook.
  • Concurrent writes resolve mechanically. The hash-check + rebase loop is thirty lines of shell. A database gives you transactions; this gives you audit history for free.
  • No new secret, no failover story. The state store is the same repo the workflows already push to.

Migrate to a real datastore when write contention on a state file becomes your bottleneck. Instrument the retry counter first. If it’s under three most days, you don’t have that problem.

Claude Code skills in markdown, not a prompt-management platform

Every reusable agent behavior — QA runners, review checklists, the senior-reviewer perspective, the spec drafter, the qualifier critic, the copy evaluator — is a markdown file under .claude/agents/ or agents/. They live in the same repo humans read. They pass through the same PR review. The dream loop proposes edits to them via PR; humans approve.

Prompt-management platforms pull prompt authoring out of the docs repo and into a UI where fewer people see them and no one owns them. The whole thesis of Context, Loops, Graphs is that expertise belongs in artifacts your team already reviews. A prompt platform undoes that.

JSON Schemas + a fixtures folder, not an eval platform

Every inter-stage contract is a JSON Schema in schemas/. Every judge is regression-tested against fixtures in evals/ — historical items with known verdicts, one JSON file per fixture, with date placeholders expanded at run time so freshness-sensitive fixtures don’t rot. Both live in the repo. The runner is pytest (or a light Python harness).

Eval platforms are useful eventually. They give you dashboards, dataset versioning, comparison views. At sub-fifty-person scale they mostly give you another SaaS bill and one more system to onboard people to. Migrate when the golden sets are big enough that regenerating them locally takes minutes, or when you have enough eval jobs that a dashboard is actually shorter than reading run logs.

A bot account with narrow PATs, not a GitHub App

Fine-grained PATs, one per capability, each scoped to the minimum permission it needs. The dream loop’s write token is the only token with internal-docs write access. The implementer never sees it. The release-intelligence miner reads and cannot write.

GitHub Apps are the “right” long-term answer — installation-scoped tokens, better rate limits, cleaner audit trail. They also require an app registration flow, a JWT signer, and a webhook server if you want events. Pre-Series-B, narrow PATs with rotation are enough. Migrate when you’re managing more than a handful.

Claude -p in a workflow, not a hosted agent service

Every agent run is claude -p --allowedTools <whitelist> inside a GitHub Actions job. The tool whitelist is the security boundary. The workflow is the retry, timeout, and log capture.

A hosted agent service adds a control plane, a second billing surface, a new failure mode when it’s degraded, and a place your prompts and traces live outside your repo. claude -p in a workflow does the job with your existing runners and one API key. Adopt a hosted service when you need cross-team agent coordination or long-running background jobs a workflow’s six-hour ceiling can’t hold.

Slack for ops, not PagerDuty (yet)

One ops/alerts channel absorbs pipeline noise. One leadership channel receives nothing automated. Ever. The heartbeat lives in ops; leadership hears from a human.

PagerDuty when you’re running production infrastructure with an on-call rotation. Slack until then. The failure mode you’re trying to avoid is a heartbeat alert waking your CEO; the fix is channel routing, not another tool.

The build order

Level 1 is a slow-cook, most of it convention-setting and organizational buy-in. Level 2 goes fast — everything Level 1 installed makes the safeguards, the grounding, and the enforcement almost free. Level 3, the coding harness itself, is infrastructure plus the earned-autonomy runway on top. You do not skip levels. Every phase’s readiness gate is the entrance ticket to the next.

The Level 1 build · The context store

L1-0 · Domain taxonomy, Mintlify shell, CODEOWNERS

Repo: internal-docs (new). Mostly writing.

One repo. Not a wiki, not a Notion workspace, not a shared drive with a folder structure someone drew on a whiteboard. Mintlify from day one — docs.json, astro.config.mjs, src/ — so the SSOT could also serve the public docs site with no separate publishing pipeline.

Numbered domain folders at the root, in the order a reader would traverse them:

  • 01-market-intelligence/ — ICP, buyer personas, positioning, competitive landscape, threat intel, regulatory signals
  • 02-brand/ — brand kit, voice-eval rubric
  • 03-commercial-revenue/ — accounts, sales playbook, sequences, rubrics/
  • 04-marketing/ — content ops, decks, claims-vetted table
  • 05-product-technical/ — cohorts, docs-ops, market updates
  • 06-operational/ — decision log, processes, quarterly reviews, provenance-ledger.ndjson, rhythm-config.yaml
  • 07-engineering-docs/ — ADRs, backing-stores, service/core/SDK docs, postmortems, specs
  • 08-team-culture/ — code review, mode-tag framework, QA philosophy
  • Grounding folders alongside: getting-started/, team/, decisions/, docs-audit/, images/

CODEOWNERS routes every folder to a single accountable person. Brand and product to the CEO plus the engineering lead. Market intelligence and marketing to the PM. Commercial and partnerships to sales. This is one file. Enforced by required reviews on every PR. Ownership stops being a convention that decays; it starts being a check that fails.

A CLAUDE.md at the root, loaded on every session, points to the folder briefs. A CLAUDE.md per folder, capped at ~500 lines, is the doc-index brief for that domain — short enough to load into a fresh session, structured enough that an agent can route from the root to the right document in three hops or fewer. If it can’t, the brief is broken, not the query.

Contents were drafted by mining what already existed. Meeting transcripts held the positioning debates and their resolutions. PR-comment history held the senior engineers’ actual review standards, stated one correction at a time. Sales calls, existing ADRs, and support threads held the rest. The AI drafted; humans reviewed and approved on every PR. Curation was human. Transcription of expertise mostly wasn’t.

You’d think you sit your experts down to write docs. You don’t. You mine what they’ve already produced. Every one of the domain folders above filled from existing corpus — transcripts, comments, decisions — before any expert was asked to write anything from scratch.

Readiness gate for L1-1: A fresh Claude session with no prior context can answer a substantive positioning or product question from the SSOT alone and cite the documents it read.

L1-1 · Frontmatter, hooks, and the .claude/ harness

Repo: internal-docs. Most of the work is convention-setting.

Frontmatter schema on every document that participates in anything downstream:

  • title, description — one line each
  • owner — single accountable person, mandatory
  • status — draft | review | approved | archived
  • last_reviewed — ISO date; used by ownership pings

Nothing else about the schema is negotiable. Everything downstream — freshness alerts, routing, the graph in L3-1 of the coding harness — reads these fields.

The .claude/ harness lives in the same repo as the docs it enforces. Top-level layout:

  • agents/ — sub-agent prompts. The load-bearing one is a persona reviewer that codifies the CEO’s PR-review voice (see below).
  • commands/ — a dozen slash commands including capture-call, capture-meeting, competitive-intel, review, sync-sdks.
  • hooks/ — six PreToolUse/PostToolUse hooks, listed below.
  • indexes/ — five YAML lookup tables (accounts.yaml, competitors.yaml, customer-environments.yaml, decisions.yaml, features.yaml) that other repos load for cross-repo reference without a graph DB.
  • rules/ — the frontmatter schema, the doc-index rule, the provenance rule, the feature-bucket-mapping rule.
  • skills/ — fourteen engineering skills including build-core, cpp-qa, openapi-sdk-regen, release-kickoff, pr-review.
  • team-memory/ — the shared corpus, gated (see below).
  • Config: meta-conventions.md, slack-channels.md, settings.json.

The hooks that made the store safe for agents:

Hook Type What it does
enforce-kebab-case.sh PreToolUse Filename enforcement; dotfiles and CLAUDE.md exempt. Naming drift kills grep, which kills every downstream lookup.
block-push-main.sh PreToolUse Every change to shared truth flows through a PR. Direct pushes are blocked at the client, not just at the branch protection.
require-team-frontmatter.sh PreToolUse Writes to team-memory/ must carry team_relevant: true plus a why_team: sentence. If you can’t articulate why the memory is shared, it goes to personal instead.
block-personal-memory.sh PreToolUse The inverse: writes to personal auto-memory paths are blocked in the shared repo.
require-workflow-provenance.sh PreToolUse New GitHub Actions workflows that don’t call the shared agent-provenance action are blocked. Opt-out marker claude-no-provenance for rare exceptions.
lasso/post-tool-defender.py PostToolUse Prompt-injection scanner on Read/WebFetch/Bash/Grep/Task. Warns, never blocks. External content is data, not instruction.
log-skill-usage.sh UserPromptSubmit Slash-command usage log written outside the repo for later prompt-caching audits.

The persona PR reviewer — a sub-agent under .claude/agents/ — codifies the CEO’s PR-review voice. The reason it exists: on this team the CEO averaged seven blunt PR comments per review while the rest of the team gave the CEO’s PRs almost none. The bot fills the asymmetry. Every PR gets a persona review before any human touches it. When the team disagrees, they push back on the bot, which is a much cheaper argument than pushing back on the CEO.

Feature-index YAMLs in .claude/indexes/ are the mechanism for cross-repo reference. When engineering docs describe a feature, its graph_id is recorded in features.yaml alongside the home file. Agents in other repos (sales-ops, agent-ops later) load these indexes for grounded lookup without any graph database.

The retired-claim guard and the blocked-name grep are enforced here, not later:

  • 04-marketing/content-ops/reference/claims-vetted.md holds the vetted phrase list plus a “Retired claims” table.
  • A grep-based CI check on the public site fails any build that reintroduces a retired phrase.
  • schemas/blocked-names.json (referenced from CI on every external-facing repo) is a name list — target accounts, confidential relationships — that can’t appear in anything the outside world sees.

You’d think a rule saying “declare owners” is enough. It isn’t. The hook that fails the tool call when a document lacks an owner is. Every one of the rules above is enforced at the moment of writing, not documented in a wiki no one rereads.

Readiness gate for L1-2: every document carries frontmatter with an owner; the seven hooks are wired and blocking on real PRs; the persona PR reviewer runs on every SSOT PR and the team is comfortable being reviewed by a bot before a human sees the diff.

L1-2 · Read-only ops loops, provenance, drift review

Repo: internal-docs (workflows + scripts).

Sixteen read-only workflows shipped as a single autonomous-GTM batch. Every one grounds against the SSOT. Every one posts to a dedicated ops channel — never the leadership channel. Every one has a heartbeat that alerts on absence-of-output, not just on error.

Workflow Cadence What it does
cve-monitor.yml Weekly Scans NVD for CVEs matching the product’s stack
aeo-monitoring.yml + aeo-gap-remediation.yml Weekly AI answer-engine scans; auto-drafts FAQ PRs on gap
regulatory-monitor.yml Weekly Regulatory + government signal scan
conference-tracker.yml Weekly Speaking / sponsorship / CFP signal scan
rfp-monitor.yml Weekly SAM.gov + private RFP signals
weekly-competitive-signals.yml Weekly Competitor moves consolidated into a digest
weekly-engineering-signals.yml Weekly Public repo activity + hiring signals in the category
external-activity-digest.yml Daily Product/company mentions consolidated
product-market-updates.yml Weekly ICP-facing category shifts
weekly-digest.yml Weekly Meta-digest that summarizes the digests
health-map.yml Weekly Topic-cluster refresh across the docs
pr-review-digest.yml Weekly Summary of PR feedback patterns across the org
claude-updates-digest.yml Fri noon Anthropic + Claude release digest
cwc-playlist-monitor.yml Monthly Category YouTube playlist watch
docs-drift-review.yml Weekly Compares public docs against source; opens PR on drift
blog-auto-draft.yml + blog-publisher.yml + blog-hero-image-autogen.yml Ad hoc Draft → schedule → hero image for public-site posts

Long-running-agent state lives on bot/* branches, force-pushed to handle non-fast-forward divergence from concurrent runs. Not elegant. But git log on a bot/* branch is a complete audit of every run, and the branch pattern keeps every stateful mutation out of main until a human reviews the resulting PR.

Attribution and provenance. Every agent commit in this repo carries a provenance trailer: which agent, which workflow run, triggered by whom. Interactive and headless runs use split conventions so the trailers stay parseable. A shared reusable action agent-provenance re-authors the commit to the bot identity and appends the trailer. The require-workflow-provenance.sh hook blocks any new workflow that doesn’t call this action. For public-facing repos, a footprint-scan job strips Claude attribution from commit messages, PR bodies, and code comments before the PR opens.

The rhythm dashboard (scripts/generate-rhythm-dashboard.py, PEP-723 inline deps + uv run) is the weekly PR × OKR tracker. Every open PR across the org gets mapped to its motivating OKR from 06-operational/, rendered to a static rhythm-dashboard.html, deployed to Cloudflare Pages, and Slack-linked. It’s the single-page view of “is the work matching the plan,” and the answer often surprises everyone.

Meeting-notes ingest (granola-meeting-sync.yml, daily 12:00 UTC) pulls yesterday’s meeting notes via the official API, summarizes each into decisions and action items, and files them with frontmatter into 06-operational/. attio-call-sync.yml pushes the same transcripts into the CRM so account timelines stay complete.

Deploy layer. deploy.yml publishes the SSOT / public docs to Cloudflare Pages. Provisioning note: the Cloudflare account is a bus-factor risk if it lives under a founder’s personal login rather than an org — worth resolving before it becomes an incident.

You’d think you can start a coding harness without any read-only loops first. You can. But every safeguard the coding harness needs — provenance trailers, heartbeat on absence, ops-channel routing, prompt-injection scanning, retired-claim greps — is dramatically cheaper to install once, on a low-stakes weekly digest, than to bolt onto a workflow that’s already opening PRs against production code. Every read-only loop above is a rehearsal for a safeguard that later becomes load-bearing.

Readiness gate for the Level 2 build: all sixteen loops are green on schedule; failure and staleness alerts land in ops (never leadership); the retired-claim + blocked-name grep gates block on every merge; provenance trailers are on every agent commit; the rhythm dashboard is deployed and updating weekly.

The Level 2 build · The outbound loop

Level 2 goes fast. Not because the pattern was easy — because everything Level 1 installed made the safeguards, the grounding, and the enforcement almost free. What follows is the outbound sales pipeline. Adapt it to whichever generation task you’re using as your first non-read-only loop; the shape transfers.

L2-0 · The pipeline as a git commit chain

Repo: sales-ops (new).

Twelve agents, each a folder under agents/ with a single CLAUDE.md prompt. In pipeline order:

crawler → event-ingest → dedup → pre-filter → enrichment → stack-profile → qualifier-critic → sequence-enrollment → copy-evaluator → review-queue → evergreen-nurture → feedback-loop

The state machine is directories under leads/, one per stage, one JSON file per batch keyed by date:

leads/raw/ → leads/deduped/ → leads/pre-filtered/ (with paired -rejects.json) → leads/enriched/ → leads/stack-profiles/ → leads/critic/ → leads/approved/ → sends/queue/ → sends/log/

Every stage commits its output JSON back to main. The commit chain is the pipeline transport. git log is literally the audit trail — no orchestrator dashboard, no queue introspection UI, no second permission model. Re-runs are idempotent because each agent selects its input by “does this input file lack a same-date output file.”

Race handling — multiple stages can push concurrently on the same day’s batch — is a rebase-and-retry loop on git push. Thirty lines of shell. That replaces a queue service at this scale. Instrument the retry counter; you’ll migrate to a real queue when it’s over three most days, and not before.

Suppression is a file. suppression/list.jsonl — one JSON per line, checked into git — is the send-time blocklist. Keyed on email, domain, or company name with reason codes: bounced_hard, unsubscribed, complaint, competitor, internal, gdpr_request, prior_outreach. It’s checked twice: a cheap domain match at dedup, and a full email/domain/company match after email lookup but before signal research. The second check exists because signal research burns paid API calls.

Cross-repo grounding is sparse checkout. Every judge and enrollment workflow sparse-checks-out the SSOT via INTERNAL_DOCS_PAT to pull the current rubrics (03-commercial-revenue/rubrics/) plus the grounding docs (ICP, buyer personas, positioning architecture, brand voice-tone, claims-vetted.md, format-rules.md). Judges never carry a hardcoded rubric; the rubric lives once, in the SSOT, and the pipeline reads the current version at run-time. This is what makes rubric edits a single-source change.

Attio schema setup (attio-schema-setup.yml) is idempotent. It runs on every workflow that could hit the CRM, but only creates missing fields. First-run bootstrap and steady-state parity are the same code path.

You’d think you need a queue service to run a multi-stage pipeline. You don’t. A folder per stage, a JSON per batch, and a rebase-and-retry loop on git push carries the pipeline for at least the first year. Migrate to a queue when the retry counter tells you to, not because a diagram suggested you would.

Readiness gate for L2-1: twelve agent folders exist; the state machine runs end-to-end on a manually-loaded batch (event CSV → raw → deduped → pre-filtered → enriched → stack-profile → critic → approved); the commit chain is clean; the suppression file is loaded from historical outbound data and both checks are wired.

L2-1 · Judges, rubrics, and golden sets

Repos: sales-ops (judges + evals) + internal-docs (rubrics).

Two blind evaluators, each a rubric in the SSOT — not in the pipeline repo:

  • 03-commercial-revenue/rubrics/qualifier-critic.md — 23 criteria across five categories. PASS at ≥70, with zero Category-E fails.
  • 03-commercial-revenue/rubrics/copy-evaluator.md — 24 criteria across five categories. PASS at ≥80, with zero Category-E hard-blocks. Category E includes hallucinated personalization; a hallucinated personal detail fails the draft outright.

The judges are checkers, not makers. Their agent prompts (agents/qualifier-critic/CLAUDE.md, agents/copy-evaluator/CLAUDE.md) say so explicitly at the top. Each emits a schema-validated verdict JSON with per-criterion scores and rationales. A PASS verdict moves the lead forward to leads/approved/ or sends/queue/; a FAIL verdict routes the lead to leads/critic/ or review-queue/. Failed verdicts do not silently retry.

Golden sets live under evals/ in the sales-ops repo:

  • evals/qualifier-critic/golden-leads.json — ~150 lines of hand-labeled leads with expected.json verdicts
  • evals/copy-evaluator/golden-leads.json + evals/copy-evaluator/golden-drafts/ — four hand-labeled touch-1 drafts covering one pass and three distinct fail modes: banned-opener, hallucinated personalization, unvetted-claims

The harness (evals/run_eval.py, ~150 lines) does two non-obvious things:

  1. Sandbox the SSOT. Its setup command copies internal-docs into a temp directory so the eval runs deterministically against a snapshot, not the live tree. A rubric edit landing mid-eval can’t corrupt the run.
  2. Materialize date placeholders. {{TODAY}} and {{DAYS_AGO_N}} in fixtures get expanded to real dates at run time, so freshness-sensitive criteria — “hiring signal within the last 30 days” — never rot the fixture. Fixtures without this pattern silently become wrong when they age.

judge-evals.yml runs Mon 11:00 UTC on a schedule and on any PR touching agents/qualifier-critic/, agents/copy-evaluator/, or evals/. A verdict flip on the golden set fails the workflow and freezes rubric merges until the eval is green again. Because the rubric edits live in a different repo (SSOT) from the judge (sales-ops), the merge gate has to be triggered from both sides — the SSOT PR check calls back into sales-ops eval, and the sales-ops eval fails loud if the pulled rubric produces a verdict change.

Monthly feedback-loop (feedback-loop.yml, 1st of month, 13:00 UTC) mines the human edits made to drafts before approval, clusters them into patterns, and — if it can cite three or more independent examples — files an evidence-backed rubric-proposal PR against the SSOT. The rubric improves from observed corrections; nobody has to notice the drift themselves. Humans approve the criteria change; they no longer have to be the ones who detect that the criteria need changing.

You’d think rubric maintenance is a standing meeting. It’s a scheduled loop. The feedback loop mines what humans corrected before approval and files evidence-backed proposals for review. Humans approve changes to the criteria; they no longer have to detect that the criteria are drifting.

Readiness gate for L2-2: both judges are wired as pipeline gates; both golden sets exist with expected.json; judge-evals.yml is green on schedule and a required check on rubric-path PRs in both repos; at least one monthly feedback-loop run has produced an evidence-backed rubric-proposal PR (approved or rejected — both count as success).

L2-2 · Send-side, deliverability, and the kill switch

Repo: sales-ops (workflows + Cloudflare Workers).

The send-side is where the safeguards get load-bearing. Every safeguard here is code, not prompt — because anything that must never happen shouldn’t depend on a model’s mood.

smtp-send.yml runs hourly M–F 13-23 UTC. Every run checks, in order:

  1. SEQUENCES_PAUSED — a repo Actions variable. Global kill switch. If true, the workflow exits silently with a heartbeat note. Nothing sends.
  2. The suppression file for every candidate send.
  3. sends/daily-count.json — a hard cap (100/day is the initial setting). The counter increments after each successful send.
  4. The EU router (lib/eu_router.py) — EU/EEA/UK contacts never SMTP-sent. Instead flagged for LinkedIn-only routing plus a Slack action item.

Sends go through Resend with ATTIO_BCC_ADDRESS on every outgoing message, so every send lands on the CRM contact timeline. The BCC is the audit trail — reviewers can reconstruct the full conversation from the CRM without touching sales-ops at all.

Reply intake (reply-monitor.yml, hourly M–F 13-23 UTC) does exactly two things: pauses the sequence for the replying account, and DMs a reviewer on Slack. Reply detection is the highest-blast-radius safeguard in the loop — a stale reply-monitor means Hard Rule 5 (pause on reply) is unenforceable, which is why the heartbeat below elevates stale-reply-monitor to a critical alert.

Deliverability monitor (deliverability-monitor.yml, daily 12:00 UTC) reads bounce and complaint rates from Resend. On a spike, it sets SEQUENCES_PAUSED=true autonomously. An automated agent can globally pause outbound based on delivery health without a human in the loop. The reasoning: sender-reputation decay has a fast curve — every extra minute of sending after a bounce spike makes the recovery cost higher. Waiting for a human is more expensive than pausing incorrectly.

Pipeline heartbeat (pipeline-heartbeat.yml) runs Tue–Sat 12:30 UTC plus Mon–Fri 17:00 UTC. It doesn’t check that workflows succeeded — it checks that they ran and produced output. Absence of output is the alert. Cross-workflow invariants it evaluates:

Invariant Alert level
Enrichment ran today given a crawl ran today Warning
Every scheduled workflow’s latest run is not failure Warning
Sends succeeded in the last 24h and reply-monitor is stale Critical — page ops
Crawl produced zero raw leads today Warning
Deliverability monitor hasn’t run in 48h Warning

HEARTBEAT_SKIP is a repo variable that silences a specific monitored workflow (like reply-monitor before DNS is live for a new sending domain) without deleting the invariant. Cheaper than commenting out checks and forgetting to restore them.

Cloudflare Workers carry the ambient services: inbox-agent/ handles inbound routing and slack-webhook/ handles Slack interactivity. Deployed via cloudflare-worker-deploy.yml + inbox-agent-deploy.yml on push to the relevant paths, using wrangler.

Headless execution contract — the last piece. Every headless agent’s CLAUDE.md carries an “Execution contract (READ FIRST)” preamble that forbids “want me to…”, “should I…”, “let me know if…”. Those messages get eaten by the workflow log, the next stage never runs, and the batch is silently stuck. Even with the preamble, sequence-enrollment.yml prepends PROMPT=$'START NOW. …' before the model’s actual instructions — the pattern was added after the model kept offering to check status first before drafting.

You’d think the kill switch is a red button someone pushes. It’s also a repo variable that a scheduled workflow flips autonomously on delivery-health spikes. The manual switch is the backup, not the primary. The primary is code, because sender-reputation decay moves faster than a human on-call rotation.

You’d think the heartbeat’s job is to notice failed workflows. Its job is to notice absent workflows. A workflow that errored loudly is annoying; one that silently didn’t run is dangerous. Absence of output is the alarm.

Readiness gate to build the coding harness: the loop is running steadily with (a) at least one automatic pause-and-recover triggered by deliverability-monitor.yml, (b) at least one heartbeat-caught staleness incident routed to ops, and (c) zero incidents involving a hallucinated personalization slipping past the copy-evaluator — or, if one did slip past, a fixture was added to the golden set from that incident and judge-evals.yml re-caught it.

The signal to build the harness

Two loops had been running steadily. Judges were regression-tested. Golden sets were catching drift on rubric edits. Ops-channel heartbeats had shaken out the silent-failure modes. The kill switch had fired at least once autonomously and recovered. The pain that made this paper’s phases feel obvious was small and specific: we were manually ferrying tickets into a coding session’s context, and manually ferrying the diff and reviews back out. That’s the exact pain Context, Loops, Graphs names as the Level 3 signal. It isn’t ambition. It’s a wrist-cramp.

You’d think you can start the coding harness before the loops are running. You can. It just costs more later. Every safeguard above — provenance trailers, heartbeat on absence, ops-channel routing, prompt-injection scanning, retired-claim greps, an autonomous kill switch, judge regression tests, sparse-checkout grounding — is dramatically cheaper to install on a low-stakes read-only loop or an outbound draft than to bolt onto a workflow that’s already opening PRs against production code. The habit is the load-bearing artifact, not the diagram.

The repo split for Level 3

By the time you start Level 3, you already have two repos from the levels below — the SSOT and the loop. Level 3 adds a third dedicated repo and touches a fourth. Fighting this split is the most common early mistake.

Repo Purpose
internal-docs (from L1) Your SSOT. Graph nodes live as frontmatter. Skills, rules, team memory, terminus docs. The dream loop writes here — nowhere else.
sales-ops / loop-ops (from L2) Your first production loop. Judges pull rubrics from internal-docs via sparse checkout; state lives here.
agent-ops (new, L3-0) New private repo. Workflows, agents, schemas, state/, evals, condensed session transcripts. Never touches production code directly.
product-repo(s) Your existing code. Agents open PRs against these. Guards are installed as required checks on agent/* branches.

The reason for the split is not neatness. It’s blast radius. The pipeline’s runtime commits (queue claims, transcripts, ledger updates) must not pollute your knowledge base’s history; your knowledge base’s edits must not require production repo access; your production repos’ merges must never require agent-ops credentials to review; and the loop repos must not carry write tokens for the SSOT.

Five concepts to align on before you write any code

These recur throughout the Level 3 phases below. If your team hasn’t agreed on them, agree now.

Terminus documents

The small set of load-bearing, high-fan-in artifacts everything else depends on. Operational guardrails (cost ceiling, WIP caps, merge gates), the spec template, system invariants, expertise paths, the mode taxonomy. These are the engineering equivalent of the sales rubrics in CLG: senior judgment, compiled. They live in terminus/ in internal-docs and change only through deliberate architectural PRs.

Mode taxonomy

Every ticket is labeled mode:claude-led, mode:mixed, or mode:requires-expertise. The intake workflow enforces this — it is not advisory. claude-led runs the full pipeline. mixed is refused. requires-expertise is pinned at L0 forever, and the expertise-path guard blocks agent PRs against those code areas regardless of the ladder.

Write down what kinds of tickets belong in each mode for your product. This doc is the highest-leverage thing to get right before L3-2 ships.

Skills

Markdown files under .claude/agents/ describing a repeatable task. Each one names itself, describes when to invoke it, and lists the steps. QA skills are the most important — the implementer’s self-verification runs them all in parallel as an Actions matrix before opening a PR. A failed QA skill abandons the run and alerts ops; the PR never opens.

Build at minimum: one QA skill per language in your stack, a pr-review skill, a senior-reviewer skill (Opus), and a spec-drafter skill. That’s the floor for L3-2.

The hash-check state pattern

Every state write goes through a composite action — never directly from an agent job. Read the file, record its blob SHA, compute the update, attempt to commit; if the SHA moved, rebase and retry up to three times, then fail loudly. This is your concurrency control. It is also the entire reason git can substitute for a queue service at this scale.

Session transcripts, condensed

Every implementer run produces a transcript. Raw transcripts are large and may contain secrets — they are never written to disk raw. A condensing step runs as the final job of the implement stage: it strips tool call bodies, keeps model text output, redacts anything matching your secret patterns, and stores the result under sessions/YYYY-WW/<issue>-<slug>.json.

The dream loop’s miners read these condensed summaries. They must not see raw content, ever.

L3-0 · Plumbing and safety floor

Repo: agent-ops + org settings.

Repo skeleton: agents/, evals/, state/, sessions/, schemas/, .github/workflows/. An attribution-strip composite action that re-authors commits to the bot and strips Claude trailers, plus a footprint-scan check for public-facing repos. A pipeline-heartbeat.yml that alerts on staleness, zero-output runs, and failed-latest runs — into the ops channel, never leadership.

Wire the cost ceiling now, before anything else runs. A weekly spend digest via gh api; alert at 70% of budget; hard-stop non-exempt workflows at 100%. Document the ceiling in terminus/operational-guardrails.md. Do not skip this step. A runaway agent that discovers your API key has no ceiling is the single most expensive failure mode in the whole system.

Wire the input scanner. It’s a PostToolUse hook that runs on every tool call that reads external content — issue bodies, PR comments, commit messages, scraped pages. It checks for prompt-injection patterns, secret patterns, and any name on schemas/blocked-names.json. On match: halt the tool call, alert ops, comment on the source flagging the pattern. External contributors are your highest-exposure surface; the scanner is your defense, not the model’s judgment.

Provision the PAT matrix. One token per capability, each scoped narrowly:

Token Scopes Used by
INTERNAL_DOCS_READ contents:read on internal-docs Every task agent
DREAM_WRITE contents:write + pull-requests:write on internal-docs Dream loop only
PRODUCT_READ contents:read on product-repo(s) Implementer
PRODUCT_WRITE contents:write + pull-requests:write on product-repo(s) Implementer
SPRINT_WRITE issues:write on sprint repo Ticket-drafter
ACTIONS_READ actions:read on agent-ops Cost digest, heartbeat

Never give a task agent a write token scoped beyond one PR. The dream loop’s token is the only one that touches internal-docs; it never leaves the dream workflow.

You’d think a small team can share a single powerful token. That’s how a compromised implementer session ends up rewriting your positioning docs. Narrow PATs are the cheapest boundary you’ll ever install.

Readiness gate: the bot opens a footprint-clean PR to a public-facing repo from CI; the cost digest and heartbeat both post to Slack.

L3-1 · Knowledge graph bootstrap

Repo: internal-docs. Mostly review work.

Extend your frontmatter schema. Every doc that participates in the graph declares:

  • graph_id — stable unique identifier
  • graph_type — one of architectural-decision | positioning-claim | component | customer-requirement | open-question
  • depends_on, satisfies, supersedes — edges to other graph_ids
  • last_changed_pr — the PR number of the last substantive change
  • last_verified — ISO date of last human or agent verification
  • confidence — one of verified | observed | assumed

Node homes map to folders you already have: ADRs to your ADR folder, components to per-repo doc folders, positioning claims to your messaging house, customer requirements to your account folders, open questions to your engineering signals folder.

Four mining agents run in parallel, each producing a batched PR of about ten nodes at a time:

  • ADR miner infers depends_on and supersedes from cross-references and rejected-alternatives sections; backfills last_changed_pr via git log --follow.
  • Invariants miner seeds the high-fan-in nodes from terminus/invariants.md.
  • Postmortem miner writes confidence downgrades on components implicated in incidents.
  • Positioning miner walks docs that reference meeting notes and extracts positioning-claim and customer-requirement nodes.

scripts/build-graph.py compiles graph.json with a reverse index (dependents_of). graph-build.yml runs on push to main: regenerate, validate (dangling refs, cycles, superseded-but-still-depended-on all fail), commit.

Wire staleness propagation. When a node changes, walk the reverse index, flag every direct dependent’s confidence for re-verification, and upsert a pinned Staleness Digest issue. Task agents read this digest before starting any work. Only the dream loop and humans can resolve entries.

You’d think mining agents produce ten thousand-line PRs. They produce batched PRs of about ten nodes each. Small enough to review in fifteen minutes; large enough to actually make progress. The batch size is the whole trick.

Readiness gate: graph.json regenerates on every merge; ≥90% of ADRs, all invariants docs, and all postmortems are nodes; validation is a required check; the staleness digest is live and non-empty.

L3-2 · Coding loop v1, humans review everything

Repos: agent-ops (workflows), internal-docs (specs + skills), product-repo(s) (PRs + guards).

Before this phase ships, the core skill set must exist in internal-docs: one QA skill per language in your stack, pr-review, senior-reviewer, spec-drafter. If any is missing, stop.

The pipeline is six workflow jobs.

Intake runs on a two-hour cron plus manual dispatch. It scans your sprint board for issues carrying mode:claude-led and agent:queued with no open agent branch, refuses mode:mixed and mode:requires-expertise, claims the ticket in state/queue.json via the hash-check composite action, comments on the issue. WIP cap: two concurrent tickets.

Spec runs the spec-drafter skill, pre-loading the relevant graph nodes for the affected area. Output is a PR to your specs folder in internal-docs. A human approves before anything gets coded. There’s a fast path — bug fixes restoring documented behavior, contract-preserving refactors, docs-only changes — that skips this step with a one-paragraph intent note attached to the ticket instead.

The spec template is seven required sections: Problem, Constraints, Relevant decisions, Success criteria, Out of scope, Implementation notes, Verification plan. Every field is scoreable — no vague language survives the judge in L3-3, and the earlier you enforce that, the less rework later.

Implement checks out agent/<issue>-<slug> (the branch is the checkpoint; re-runs resume from it automatically). It loads the approved spec, the graph nodes for touched components, invariants, and shared team memory. It runs all applicable QA skills in parallel — one Actions job per language or layer. It produces a condensed session transcript. On timeout, max-turns, or QA failure, it abandons, comments on the issue, and alerts ops. The PR does not open.

Self-review runs fresh context. Two separate claude -p invocations — separate jobs, separate logs. The first is pr-review. The second is senior-reviewer (Opus), fed only the diff, the spec, and the relevant graph nodes. The implementer must address or explicitly rebut both reviews’ findings before the PR opens; both transcripts are attached verbatim to the PR body.

You’d think the implementer can review its own output if the prompt is careful. It rubber-stamps itself. Every review is a separate process with no access to the implementer’s transcript. This is not a preference — it’s the reason the reviews catch anything at all.

PR opens. The bot creates the PR; attribution is stripped on public repos. Body includes the spec link, both review transcripts, the graph nodes consulted, and the QA skill results. A Slack notification lands in ops.

Guards are required CI checks on agent/* branches in product repos. The expertise-path-guard reads terminus/expertise-paths.md and fails any agent PR touching your highest-risk paths — for a SaaS product, think auth and session handling, billing and entitlements, data-layer internals, anything security-sensitive. This is deterministic. Never self-policed. The footprint-scan check confirms no Claude attribution markers landed in commit messages, PR bodies, or code comments.

Readiness gate: five tickets flow end-to-end. Median time-to-PR under 24 hours from label. Zero expertise-path violations.

L3-3 · Quality gates and judges

Repos: agent-ops (judge + evals), internal-docs (rubric), product-repo(s) (required checks).

Add a code-judge — an Opus process that scores each agent PR against the spec’s seven sections, your stability policy, and a senior-reviewer rubric mined from your team’s actual review history. It’s a required check. Must pass ≥80% to merge.

The rubric skeleton — adapt to your codebase — scores each category 0/1/2. Default threshold: ≥12/16 to pass.

Category What to assess
Correctness Does the implementation match every success criterion in the spec?
Invariant compliance Are all invariants in terminus/invariants.md preserved?
Scope discipline Does the PR do exactly what the spec says — no more, no fewer?
Test coverage Are new behaviors tested? Failure modes and edge cases?
Error handling Failure modes handled explicitly, not swallowed?
Readability Would a new team member understand this code in six months?
Performance Any obvious algorithmic issues or N+1 patterns introduced?
Security Input validation gaps, privilege escalation, insecure defaults?

Seed the initial rubric weights from your historical PR review comments. Look at the last twenty to thirty merged PRs, list the most common change-request categories, and weight the rubric accordingly. Those patterns are your senior judgment. Compile them.

Build the golden set. Twenty to thirty historical PRs with known human verdicts (approved as-is vs. required changes), pulled via gh api, stored in evals/fixtures/. Store the fixtures as JSON with the diff, the spec (or a reconstruction), and the verdict. This is the ground truth the judge is measured against.

judge-evals.yml runs matrix over code-judge and senior-reviewer. Triggers: any change to the rubric or fixtures, weekly cron for model drift, manual dispatch. A verdict flip on the golden set triggers a global auto-merge freeze until eval is green again.

Layer on the mechanical checks CI doesn’t already cover: coverage delta ≤ 2pp per file, invariant/contract suite inclusion on guarded paths.

Stub the release-kickoff skill — LLM-driven integration tests as a release gate, cheap path-filtered surfaces per PR (checking error message actionability when error strings change, checking docs voice on docs-only changes). Fully wire in L3-4.

Readiness gate: the judge is required on all agent PRs. Weekly eval is green. Judge/human verdict agreement ≥85% over ≥15 PRs.

L3-4 · Release intelligence

Repos: agent-ops (workflow + agent), internal-docs (rubric), sprint repo (proposed-ticket label).

The coding harness produces PRs. Without an outer loop, nothing tells you which of those PRs the market cares about — the highest-value work in your backlog stays invisible until a customer asks for it, and by then you’re behind. L3-4 closes that loop. Releases score themselves; the market-worthy ones arrive as proposed tickets for a human to promote or dismiss.

This is the phase most teams treat as a nice-to-have and then discover, six months in, is doing half the work. Build it now.

Prerequisites, all first-class graph nodes in internal-docs:

  • An ICP definition doc that’s current and treated as authoritative.
  • A positioning/messaging house doc.
  • A competitive intelligence system producing periodic digests accessible from Actions.

If any of the three is missing or stale, fix it before shipping this phase. The scoring rubric is only as good as the docs it anchors to.

The rubric. Same pattern as the code judge: numeric thresholds, editable in the docs repo, regression-tested on change. Five dimensions, each scored 0–2. Sum ≥6/10 is market-worthy; ≥8/10 is a high-priority template or docs candidate.

Dimension 0 1 2
ICP problem match No connection to a named ICP problem Adjacent to an ICP problem Directly addresses a problem named in the ICP doc
Messaging house alignment Contradicts or is silent Supports an existing claim Closes a documented gap or materially strengthens a claim
Market signal resonance Not in your intel digest Mentioned as emerging Active problem for the ICP cohort in the latest digest
Differentiation delta Parity with alternatives Incremental advantage Meaningfully harder to replace vs. the nearest named alternative
Template/docs impact No prospect-facing value Useful but not demonstrable A prospect would understand the product materially better from a template showing this

Score against your target ICP, not your current customer list. An early-stage customer base often reflects founder relationships more than market fit; the customers who look nothing like your ICP are a signal to understand, not to optimize for. The written ICP is the reference. The anecdote isn’t.

The workflow. release-intelligence.yml triggers on release: types: [published]. Its jobs, each a separate stage:

  1. Fetch merged PRs since the last release tag via gh api.
  2. Load context: ICP doc, positioning-claim graph nodes from graph.json, latest competitive intelligence digest.
  3. Score. release-intelligence-miner reads the PR list + context and outputs a schema-validated release-findings.json — one entry per PR with the five dimension scores, total, verdict, and evidence links.
  4. Validate against schemas/release-findings-schema.json; fail the workflow on schema error.
  5. Write state to state/releases/<tag>/findings.json through the hash-check action; idempotent on rerun.
  6. Draft tickets. For every finding scored market-worthy, the ticket-drafter agent opens one issue in your sprint repo. Body format is strict: Problem, What to create, Acceptance criteria, plus release tag, PR number, score, and evidence. Labels: agent:proposed and release-intelligence. Assigned to the sprint board but not to a sprint iteration.

The prospect-name guard validates every drafted body against schemas/blocked-names.json before the issue opens. Confidential target names never appear in tracker bodies that external contributors might see.

Human promotion is a hard gate. A PM reviews the proposed issues, applies agent:queued + mode:claude-led, and removes agent:proposed. The ticket enters the L3-2 pipeline on the next intake cron — no other action needed. The ticket body format is designed to be L3-2-compatible on day one, so the delivery layer requires no new plumbing.

Heartbeat guard. If a new tag exists with no corresponding state/releases/<tag>/ entry after 24 hours, alert ops. Prevents silent failure when someone pushes a tag without publishing a GitHub release.

You’d think a PM reads every release to find the marketable bits. The release scores itself. Each shipped change is evaluated against your ICP and messaging house, and the market-worthy ones arrive as proposed tickets with the scoring attached — for a human to promote or dismiss.

You’d think this only works once you have real customers. It works better before you do. When your customer base is small and unrepresentative, anchoring to the written ICP is the only way to keep the backlog pointed at the market you’re building for instead of the accidents of who bought early.

Readiness gate: a test release triggers the workflow; release-findings.json validates against schema; at least one finding carries evidence links; a proposed issue opens with correct labels, body format, and sprint-board assignment; the Slack notification lands in ops; the issue does not enter the coding pipeline until a human applies agent:queued.

L3-5 · The dreaming loop

Repos: agent-ops (dream.yml), internal-docs (output PRs). Carries an ongoing human-review commitment on every run.

The dream loop is the memory-refresh engine. It runs weekly on a Sunday cron plus manual dispatch. It has its own budget line and is not cost-exempt — it’s the first workflow to shed at the ceiling.

It reads: merged PR diffs and review comments across your org (human post-merge edits to agent PRs are the richest implicit correction signal), condensed session transcripts, postmortems, meeting notes if your notes tool syncs to internal-docs, the staleness digest, and judge-versus-human verdict disagreements.

One orchestrator spawns three parallel Task-tool subagents:

  • Transcript-failure miner reads condensed sessions; finds dead-ends, wrong assumptions, and missing context that caused sessions to fail.
  • Review-delta miner reads post-merge diffs between the agent’s PR and the final merged state; what a human changed after the agent finished is the strongest signal you have.
  • Doc-drift miner reads the staleness digest and graph nodes; downgrades confidence on nodes contradicted by merged code.

All findings are schema-validated JSON. The orchestrator synthesizes with a 3x evidence bar — a proposed change to shared memory requires ≥3 independent evidence links before it ships in the PR.

Output is one PR per run to internal-docs, touching: team-memory/ (shared context all agents load; corrections to wrong assumptions go here), graph frontmatter (confidence bumps, last_verified updates, new supersedes edges), and rules/ (guardrails that have proved necessary). From L3-6 onward it can also propose edits to skills and prompts.

Every proposal in the PR body carries ≥3 evidence links, a prevalence count across sessions, and a one-line “risk if this memory is wrong.”

Judge disagreements go into a second small PR to evals/ — proposed golden-set fixture additions.

Human approval is required indefinitely. This gate is the memory-poisoning firewall. No dream PR merges without a human reviewing it. Not at L3. Not at L4. Not ever.

You’d think you have to sit down and update the wiki after every incident. The dream loop mines the incident, the transcripts, and the post-merge edits, and files the update with evidence attached. You approve or reject. You never author from scratch.

Readiness gate: three consecutive weekly runs with reviewable PRs; ≥60% of proposals merged; ≥1 supersedes edge and ≥1 team-memory entry originated from a dream run.

L3-6 · Earned autonomy

Repo: agent-ops.

The autonomy ledger — state/autonomy-ledger.json — tracks per-work-class level and rolling metrics, updated by CI on PR close or merge, hash-checked like every other state file.

The ladder:

Level What changes
L0 Human-authored only
L1 Agent opens PR; human reviews spec + code
L2 Spec fast-path auto-approved; human reviews code
L3 Auto-merge on all-green + 24h revert window + Slack notify
L4 Auto-merge + auto-selected tickets from sprint column

Promotion criteria — all must hold, per class:

  • ≥10 consecutive merged PRs with zero change-requests and zero post-merge human edits
  • Judge first-pass ≥90%
  • Zero postmortems attributing an escaped bug to the class in the trailing 30 days
  • Coverage rule never tripped
  • Zero guardrail overrides

Demotion, automatic:

  • Postmortem with caused_by_class in frontmatter → −1 level plus auto-merge freeze for that class
  • Two consecutive change-requested PRs → −1 level
  • Judge verdict flip on the golden set → global freeze until eval green

The dream loop earns a self-improvement channel here: it may propose skill, prompt, or rubric edits. Any such PR must include the golden-set eval run on the changed content in its body before it can merge, and merging triggers judge-evals automatically.

First L3 candidates are the lowest-blast-radius classes in your backlog — codegen from OpenAPI specs, SDK regeneration, docs scaffolding, test scaffolding. Not your core engine.

You’d think a work-class earns autonomy and keeps it. It doesn’t. Every merged PR either raises or lowers the class’s standing. A regression demotes the class, not the harness. Autonomy is a rolling privilege.

Readiness gate: ≥1 class sustained at L3 with no demotion across a stable run. The weekly digest reports tickets per digest cycle, human-edit distance, judge agreement, dream acceptance rate, token spend per merged PR, staleness backlog.

What breaks and why

Risk Mitigation
Memory poisoning or prompt injection Every memory write is a human-reviewed PR. Input scanner on all external content. Evidence links make poisoned claims falsifiable at review.
Judge drift Weekly and on-change golden-set evals; verdict flip triggers global freeze; rubric changes require an attached eval run.
Graph rot Reverse-index staleness propagation on every main merge; 60-day last_verified decay flags; doc-drift miner as active re-verifier; CI fails on dangling or contradictory edges.
Runaway token spend Cost ceiling in L3-0 (70% alert / 100% hard-stop); per-workflow --max-turns; WIP cap 2; dream loop first to shed.
Agent touches high-risk code Deterministic path guard, seeded from terminus/expertise-paths.md. Never self-policed.
Bot footprint on public repos Re-authoring plus trailer strip plus footprint-scan required check; private provenance ledger.
Reviewer fatigue Judge verdict and self-reviews at the top of every PR body; WIP cap; spec fast-path; Slack digests instead of per-event pings; the ladder shrinks load over time.
Silent pipeline death Heartbeat workflow from L3-0 — absence of output is itself an alert.
State write race condition Hash-check composite action; every state write goes through it.
Transcript secrets leaking into the dream loop Condensing script strips secrets and tool bodies before writing to sessions/; raw transcripts never persist.
Ticket spam from release intelligence Threshold tuning in the rubric doc; WIP on agent:proposed visible on the board; human promotion is a hard gate.
ICP doc goes stale, corrupting market scoring ICP is a graph node; staleness propagates when positioning-claim nodes change without it being updated.
Release tag pushed with no GitHub release Heartbeat: if a tag exists with no state/releases/<tag>/ entry after 24 hours, alert ops.
Expertise path list goes stale as the codebase evolves The list lives in terminus; reviewing it is part of every postmortem that touches a guarded component.

What the team actually does

Action Who
Apply agent:queued to a ticket Any team member
Approve a spec PR Any team member (or fast-path skips it)
Review + merge agent PRs at L1/L2 Any team member
Review + merge dream loop PRs (memory updates) Any team member
Maintain expertise paths in terminus Engineering lead
Review proposed release intelligence tickets PM
Review the weekly metrics digest PM + engineering lead
Update the blocked-names list PM

The things that always require a human, regardless of ladder level: spec approval, memory writes (dream PRs), release intelligence ticket promotion, and — until a class reaches L3 — final merge.

Adoption sequence

The order is fixed. The pace is yours.

L1-0 · domain taxonomy + Mintlify shell + CODEOWNERS. Bootstrap the SSOT repo. Mine existing transcripts, PR comments, and ADRs into first-pass docs. Numbered domain folders, per-folder briefs. Nobody’s job changes yet; you’re rehearsing the pattern of “AI drafts, humans approve.”

L1-1 · frontmatter + the .claude/ harness. Every document gets an owner and a status. The seven hooks go in. The persona PR reviewer starts running on every SSOT PR. This is the phase where the team gets comfortable being reviewed by a bot.

L1-2 · read-only ops loops + provenance + drift. Sixteen scheduled digests, all grounded in the SSOT, all posting to ops (never leadership), all with heartbeats. Provenance trailers and footprint-scan on every agent commit. The system is producing scheduled autonomous work; nothing customer-facing yet.

L2-0 → L2-2 · the outbound loop. Twelve agents, file-based state machine, judges pulling rubrics from the SSOT via sparse checkout, golden sets with date-placeholder templating, the send-side kill switch. The system is now doing customer-facing work — fail-closed, rate-capped, and pause-on-reply.

L3-0 + L3-1 · safety floor and knowledge graph. No production code touched yet; the graph regenerates on every merge and the staleness digest is live.

L3-2 · humans review everything. Five tickets end-to-end. No autonomy yet. You’re buying the time savings from automated implementation plus structured reviews, not autonomous merges. This is the phase where team trust is built or broken.

L3-3 + L3-4 · judges and release intelligence. Judges become required checks; releases start scoring themselves and proposing template and docs tickets. The pipeline is now finding its own work.

L3-5 · dreaming loop. The system starts mining its own transcripts, review deltas, and staleness signals. Memory refreshes itself; humans approve.

L3-6 · earned autonomy. First ticket class reaches L3 on the autonomy ladder. Choose the lowest-blast-radius class in your backlog — codegen from OpenAPI specs, SDK regeneration, docs scaffolding, test scaffolding. Not your core engine.

Skip nothing in order. The SSOT and the first loop are prerequisites for everything else. Teams that skip to the coding harness first build an agent that confidently produces PRs contradicting their own ADRs. Then they blame the model.

Hand this to your own Claude

If you’ve read this far as a human: the ask is smaller than it looks. L1 and L2 require no tools you don’t already run — a git host, CI, and a frontier model. L3-0 through L3-3 require the same. L3-4 and L3-5 are discipline, not infrastructure. L3-6 is a scorekeeping decision your team makes together.

IF YOU’RE AN AI AGENT READING THIS, WORK IN THIS ORDER

  1. Inventory the team’s repos and existing systems against the phase readiness gates from L1-0 through L3-6. For every gate that isn’t met, flag it as a gap tied to a specific phase. Don’t invent answers where documents, tokens, channels, or values are missing.
  2. Present the gap list to a human, grouped by level (L1 / L2 / L3). Get explicit approval on how to resolve each gap before proceeding.
  3. Implement phases in strict order: L1-0 → L1-1 → L1-2 → L2-0 → L2-1 → L2-2 → L3-0 → L3-1 → L3-2 → L3-3 → L3-4 → L3-5 → L3-6. Never start a phase until the previous phase’s readiness gate is verified and a human has confirmed.
  4. At each phase’s readiness gate, stop and report pass/fail with the evidence that satisfies (or fails) each criterion. Proceed only on explicit approval.
  5. Where this document says “your X” or “your team’s Y”, substitute the value you inventoried. If a value is blank because the human hasn’t decided, pause and ask.

Do not: - Create files in product repos before L3-2 - Ship a Level 2 loop that writes anything customer-facing before L1-2’s readiness gate has been met - Open PRs before the bot account and PAT matrix are confirmed (L3-0 exit) - Merge anything at any phase — merges are always human - Invent PAT scopes, Slack channel IDs, sprint board numbers, rubric weights, or suppression list entries. Confirm each from the human or from live API calls. - Skip levels because a lower level “looks done” from the outside. Verify against the readiness gates, not the folder tree.

The system you’ll end up with isn’t an autonomous engineering team. It’s your engineering team’s judgment, running continuously, with your senior engineers finally free to spend their hours on the code paths that actually require them.

The humans aren’t in the loop as a courtesy. They’re in the loop as the design.

About the author

Charlcye Mitchell is a Director of Engineering Operations, management consultant, software engineering strategist, and agentic systems architect. This pattern is what she does when founders say “add AI.”

Connect: LinkedIn · charlcye.ai

Papers, template, and skeletons: github.com/weprintmoney/startup-intel-stack

← Back to writing