All posts

article

The Onboarding Test

A thought experiment that reveals more than most audits: if you had to write an AGENTS.md thorough enough to onboard a new senior engineer without a single tribal-knowledge conversation, what could you write — and what would you realize you don't actually know?

§ 01 — The Exercise: Write the Document You Wish Had Existed

Here is a thought experiment worth running with your engineering team. Imagine you have just hired a senior software engineer. Excellent technical skills, zero familiarity with your codebase. They are starting Monday. Your goal is for them to be genuinely productive by Friday — contributing real work, not just reading documentation.

The constraint: you cannot speak to them. No onboarding call, no pairing session, no quick Slack message to “ask me if you’re confused.” Everything they need to get up to speed must exist as written artifacts in your repository.

Now imagine that engineer is not a human — it’s an AI agent. Same constraint applies. What would you write?

This is the Onboarding Test. And here’s what makes it diagnostic: the gaps between what you can write and what you’d need an experienced engineer to explain are your context debt. The sections where you can write comprehensive guidance are your strengths. The sections where you realize you don’t know the answer clearly enough to write it down — those are your bus-factor risks, your institutional knowledge silos, your documentation debt made visible.

“You don’t need to run a documentation audit to find your context gaps. Just try to write the AGENTS.md that would make a new person genuinely effective in their first week. The blank sections are the audit.”

§ 02 — The Diagnostic: What You’ll Discover by Section

A comprehensive AGENTS.md has six core sections. Each section, when you try to write it honestly, reveals something specific about your team’s documentation health.

AGENTS.md Section Diagnostics — What Your Gaps Reveal

Section 1: Architecture Overview
”What is this system, in two paragraphs?”

Can’t write it clearly → Your team doesn’t share a consistent mental model. Different engineers would describe the system differently. High onboarding cost for both humans and AI agents.

Takes more than 500 words → Your system has accumulated complexity without corresponding simplification. The architecture has outgrown its description.

Clean two-paragraph summary → You have a shared architectural vocabulary. Low onboarding friction.

Section 2: Forbidden Zones
”What should never be touched without a second review?”

No list comes to mind → Your team relies on institutional memory for safety boundaries. One bad day — a new contractor, an AI agent, a tired engineer — and something catastrophic gets modified.

List is very long (>10 items) → Too many critical zones scattered through the codebase. Complexity debt or insufficient modularization.

3–7 specific, well-described zones → Healthy boundary definition. Risks are known and bounded.

Section 3: Tech Stack and Version Constraints
”What are we using and why are specific versions pinned?”

“Pinned for reasons I’d need to dig through git for” → Version decisions are undocumented. AI agents will try to upgrade these. New engineers will unknowingly introduce incompatibilities.

Can list versions but not rationale → Partial documentation. The version is preserved; the reason isn’t. Next person inheriting this has no basis for evaluating change.

Section 4: Code Conventions
”How do we write code here? What patterns are expected?”

“Varies by module” or “you’ll pick it up” → No consistent conventions. AI agents will average across the inconsistency and produce code that follows none of the patterns well.

Specific, written conventions with examples → Codified practice. AI agents generate consistent code. New engineers ramp quickly.

Section 5: Testing Requirements
”What constitutes ‘done’ from a testing perspective?”

“We try to maintain reasonable coverage” → No defined standard. AI agents have no basis for knowing whether their generated tests are complete. New engineers will undertest.

Specific requirements with coverage thresholds, test types, and how to run → Testability is a first-class value. AI agents can generate tests that meet the standard.

Section 6: Known Landmines
”What has bitten us before?”

More than 3 stories that start “well, back in 2023…” → Incident knowledge lives in oral history only. It will be re-discovered the hard way by AI agents and new engineers alike.

Written, referenced, annotated in code → Institutional knowledge is codified. The next person has the benefit of prior experience without needing to experience it themselves.

§ 03 — The Practice: Running the Onboarding Test With Your Team

The most effective way to run this exercise is as a workshop, not as an individual writing task. Gather your senior engineers — ideally the three or four people who carry the most institutional knowledge about different parts of the system — and have each person independently draft the AGENTS.md for their primary area of ownership.

The divergences in their drafts are more valuable than the drafts themselves. Where two engineers describe the same system differently, that’s a documentation gap and possibly an architectural ambiguity. Where one engineer writes five paragraphs and another writes two sentences for the same section, that’s uneven knowledge distribution. Where someone writes “this is complex to explain” — that section is your highest-priority documentation task.

The Length Signal

There is a specific signal worth paying attention to: AGENTS.md sections that are very long. A forbidden zones section with 15 items is not comprehensive documentation — it is evidence that your codebase has 15 critical areas that could cause incidents if touched carelessly. That’s a system design problem, not a documentation success. The AGENTS.md that is too long to be useful is telling you something important about the system it describes.

Length as signal: An AGENTS.md that exceeds 3,000 tokens is probably describing a system with too many critical zones, too many exceptional conventions, or too much accumulated complexity. The documentation problem is actually an architecture or process problem in disguise.

The “Ask Me” Sections

Pay special attention to sections where the instinct is to write “ask [specific person]” rather than actual content. Every “ask [name]” in your documentation is a bus-factor risk and a context gap. That person’s knowledge is not in your system — it’s in a human who may leave, may be unavailable during an incident, and cannot be loaded into an AI agent’s context window.

The goal is not to eliminate the expert — it’s to supplement them with written context that preserves their knowledge in a form that survives their absence and can be consumed by both AI agents and future engineers.

§ 04 — The Payoff: More Than AI Context

The AGENTS.md written through this exercise is not primarily an AI tool. It is a first-principles engineering document that would have been valuable before AI existed. Teams that have done this exercise consistently report three benefits beyond improved AI output quality:

Faster human onboarding. New engineers who start with a comprehensive AGENTS.md reach productive contribution in days rather than weeks. The tribal knowledge that previously required a month of pairing to absorb is available on day one.

Reduced incident frequency. When the forbidden zones are written down rather than residing in tribal knowledge, they get respected by everyone — including contractors, new hires, and yes, AI agents. The onboarding test surfaces the constraints that were only being enforced through convention.

Visible complexity debt. The sections you can’t write clearly are the sections you should be simplifying. The AGENTS.md serves as a complexity meter: the harder it is to explain, the more debt it’s pointing to.

A useful heuristic: If a senior engineer joined your team today and, in their second week, made a change that caused a production incident — could you point to written guidance they should have followed? If the answer is “not really,” your AGENTS.md is incomplete in the section that governs whatever they broke.

The teams that build great context infrastructure are not teams that love documentation. They are teams that have taken the Onboarding Test, seen the gaps, and understood that closing those gaps is engineering work — not a separate documentation project. The AGENTS.md is the artifact. The thinking that produces it is the practice. And the practice, done consistently, is what makes a codebase an asset rather than a liability — regardless of whether an AI agent is touching it or not.

“If you can’t write a CLAUDE.md that would make a new senior engineer effective in their first week, you haven’t lost documentation. You’ve lost knowledge. Documentation is the recovery.”

6

AGENTS.md sections that each diagnose a different class of context debt

What Each Section Reveals

  • Architecture overview — shared mental model health
  • Forbidden zones — boundary definition maturity
  • Version constraints — decision rationale preservation
  • Conventions — pattern consistency
  • Testing requirements — quality standard clarity
  • Landmines — incident knowledge codification

Warning Signals

  • “Ask [name]” instead of content
  • Architecture overview > 500 words
  • Forbidden zones list > 10 items
  • AGENTS.md total > 3,000 tokens
  • Conventions that vary by module

Benefits Beyond AI

Teams doing this exercise report:

Faster human onboarding — days not weeks

Reduced incident frequency — constraints written, not assumed

Visible complexity debt — hard-to-explain sections = system problems