Key takeaways

  • Problem: AI-assisted coding moves so fast that people don't even know what the software does, and whether it's adequately verified.
  • Idea: use LLMs to generate retro-specs from code and risk-based traceability matrices. Consider verification methods beyond tests.
  • Result: six months in, the system has proved reliable and useful. The routine runs in roughly 20 minutes, cost is negligible, and PRs take 5-10 minutes to review.
  • What it unlocks: know the product, find coverage gaps, use the specs to feed dev LLMs, generate help content.

The problem

I've been writing software for more than 2 decades. Every team hits the same wall eventually: a pile of production code that's too hard to grasp. Everybody becomes obsessed with the 2 fundamental questions of legacy software:

  • What is the system supposed to do?
  • Is it actually doing it?
The two questions every team ends up asking about its own production code.

Why does this matter?

  • Not knowing your product in depth is a huge business risk. You need a reliable source of truth that doesn't depend on people. Otherwise, you'll lose credibility with your customers, make poor decisions about what to build next, and be blind when issues arise.
  • Reliable products come from verification of software behavior. But verification has an unbounded cost. You need a method to find coverage gaps based on risk to allocate resources wisely.

I've worked on high-stakes systems in health, aerospace, finance and energy. The traditional answer in these industries is:

  • Write detailed specifications.
  • Link specs to tests.
  • Generate traceability matrices to find blind spots.
  • Create processes to guarantee correctness and pass audits.

It works great. But it's costly and boring as hell. In non-regulated environments, nobody wants to do it. In startups, even less. And now that we have AI-assisted engineering and code is flying? Oh boy…

But the new tools offer new opportunities. What if we could use AI to capture most of the benefits of traceability for a fraction of the human effort? Can we get 80% of the value for 5% of the effort?

Idea of a solution

My main idea, inspired by Karpathy's LLM wiki, is to create a process where LLMs generate retro-specs from code, and trace them back to tests.

The loop: a weekly spec agent reads what changed in the code repos and proposes updates to the specs repo, which humans, coding agents and a help routine all read from.

Note that it's very different from spec-first development, where specs are carefully thought through before implementation happens. This is great in some regulated environments, or where the culture and available resources make it part of the daily work. I'm trying to capture part of that value in other environments, where specs-first simply doesn't happen, for whatever reason, good or bad.

Important concepts to add to this core idea:

  • Requirements are graded by risk. We don't chase blind coverage; we scale verification effort to what's at stake.
  • Verification methods are expanded beyond tests to adapt to the agentic era. Notable additions are vetting agents, evals, and monitoring in prod.

Implementation

The implementation of this system comes in 2 parts:

  • Spec system. Obsidian configuration and front matter definition to easily navigate specs, attach metadata, and run queries.
  • Prompts. A general prompt for writing specs that can be used in manual or auto mode. And an automated update routine.

Note: this section is technical and targeted at developers. Executive readers may want to skip to the conclusion.

Spec system in Obsidian

The generated wiki lives in Obsidian. This great tool offers:

  • Metadata (or front matter) on top of markdown files.
  • A UI to navigate through the data.
  • plugin system that extends it.

Configuration

Front matter

The metadata is defined in a schema. Here's what it looks like:

The front matter of one spec: identity, parent, repositories, risk level, and links to the exact test cases that verify it.

Fields of special importance are:

  • Risk level: an estimate of how bad it would be if the software failed the requirement. Used to guide the verification effort.
  • Verification methods:
    • Post-deployment tests (one service in isolation).
    • UI tests (front-end with mocked back-ends).
    • e2e tests (front+back, for the app).
    • Monitoring (alerts, availability tests, dashboards…).
    • Vetting agents (continuous verification of an agent's behavior by another agent).
    • Evals (statistical assessment of agent output quality based on several executions on a golden dataset with mocked tools).
    • Unit tests (only when no post-deployment method applies).
Plugins

The Obsidian plugins we currently use are the following: Linter, Style settings, Breadcrumbs, TreeView, DataView.

DataView is the most important one, because it enables writing queries on the front matter of specs. For example:

  • Display specs without verification, by risk level.
  • Display the number of verifications for each spec, by verification type.
  • Specs by repository.
PR validation gate

As always at Omniscient, we do gated commits (automated checks in PRs) to protect the quality of the main branch. The checks we scripted are the following:

  • Validate front matter schema.
  • Correctness of spec ids (uniqueness, cross-references).
  • Correctness of file hierarchy vs parent id and spec names.
  • Unit tests of scripts themselves.

These guardrails enforce the mechanical quality of the specs. Agents are great at using this feedback to self-correct.

What the end result looks like

The graph view displays how specs connect to each other via hyperlinks. The colors correspond to how we have split our system (user-facing app, data ingestion pipeline…).

Filtering works as expected. For example, here's the subset of specs that reference signals (our core concept to make sense out of the noise of the web):

The whole spec wiki as a graph, coloured by subsystem, filtered to display only specs that references signals.

Clicking on a node opens the spec. Voilà. Information is now structured and browsable.

Traceability matrix

Combining verification traceability with risk assessment makes it easy to spot specs that have no coverage.

A DataView query over the front matter: every medium-risk spec that had no verification attached when the screenshot was taken.

This helps us gain confidence in our QA system and make informed decisions on where to allocate our testing effort.

Prompts

Now that the scaffolding is in place, we need a prompt to have LLMs write and update the specs and their frontmatter.

Agents.md

Our agents.md (about 300 lines) describes all the rules to write and update the specs.

Here's a breakdown of what it says:

  • Purpose of the repo.
  • Naming convention for directories and files in the specs folder.
  • The front matter schema.
  • What to put in the body of a spec (context, behavior, constraints, open questions).
  • Guidelines on spec lifecycle: when to create a spec, how to split one into sub-specs.
  • How to find the source of truth for verification methods in the other repos (code and infra) and link them to specs. Guidelines for the granularity (we want only the important stuff, not completeness).
  • How to assess the risk level of each spec depending on user experience and NFRs (security, performance…).
  • Example of a well-written spec.

The agents.md instructions work for both manual session (active prompting) and auto session (scheduled weekly executions).

Spec content

Here's an example of a spec body on the email alert system (sentinels are bots users create to alert them when something relevant to their interests happens on the web):

A generated spec body: context, a trigger/behavior table, and the constraints that matter.

Across all the changes I've checked, the text is concise and accurate — good enough to be useful.

Auto update routine

A script compares each in-scope repo's current commit SHA against the SHA recorded at the last run — so that the update routine runs only on what has changed.

The state file: one commit SHA per repo, so each run only looks at what moved since the last one.

The repo contains a set of instructions to update specs. Key points:

  • Follow agents.md.
  • Scan git changes and update the state file.
  • Update traceability (verif-* front matter, lines of test cases in test files).
  • Update specs that correspond to changes in implementation.
  • Create new specs.
  • Validate the scripts that check the rules.
  • Open a PR for spec updates and a PR for new specs (both in draft state, as always, so that a human reviews the changes before publication).

The self-update is done with a Claude Code Routine that invokes the spec refresh playbook. Key points:

  • Get the code of all repos.
  • Apply the spec refresh playbook.

We end up with 3 levels of nested LLM prompts:

  • Agents.md: general instructions for manual prompting or automated update.
  • Spec refresh playbook, following agents.md. Can also be invoked manually.
  • Claude Routine instructions, following the spec refresh playbook.

Results

Excerpts from a PR created for new specs:

The PR description the routine writes: each new spec with the change that motivated it, plus coverage notes and the candidates it deliberately declined to spec.
The same PR's diff: new spec files nested by subsystem, each opening with the front matter and its verification links.

We review these PRs thoroughly and they are accurate (we rarely have fixes to request). They definitely bring value.

Conclusion

We've been using the wiki specs system for roughly 6 months and the auto-update routine for 3 months. The routine runs in roughly 20 minutes, cost is negligible compared to our global Claude Code consumption, and PRs take 5-10 minutes to review.

What stands out is that the resulting specs and the test traceability are accurate. The system helps the whole team know our product in depth, and guides where we spend verification effort. Without automation, it would certainly not exist.

Once operational, the specs wiki serves various purposes:

  • Find blind spots in test coverage.
  • Having reasonably good source of truth for software behavior is useful for:
    • Onboarding newcomers.
    • Answering product questions.
    • Imagining next features.
  • It can be fed back to coding agents to give them a summarized and up-to-date summary of what the platform is doing.
  • Generate help content. I used the specs to generate a first version of the knowledge files that our conversational agent reads to guide users when they need assistance. I am now working on another routine to automate this process and run it weekly.

I encourage others to experiment with this technique. It provides the benefits of retro-specs and traceability matrices to teams that don't have the time, budget or culture to perform these activities in the traditional way.

Note: I didn't provide any source code for confidentiality and security reasons. But you can use this page as input in a prompt for your favorite coding agent to frame a solution that works in your environment.