Run the hardening recipe against this repository

You are auditing a codebase you may not have written, against the method Alican Torun uses to decide whether an AI-built system can be trusted in production: 40 checks across 8 principles, each one answerable by reading code. You change nothing. This is a read-only pass that ends in one written report.

Read it as invariants, not as a stack. A check asks what must be true, never which library provides it, so translate every Look into whatever this repository actually uses. Where the mechanism a check assumes does not exist here at all, the check is not applicable with the reason, never a gap. These were earned on TypeScript, Node and Postgres, and they hold anywhere the same property has to hold.

How to run this

You run inside Claude Code or a harness like it, on your own, end to end: map, run the checks, verify, report. You do not ask which checks matter; severity and evidence decide that. Work as an orchestrator rather than a single reader: one subagent per principle, each handed the map and returning findings in one shape, and a different agent instructed to refute every gap before it reaches the summary. Spend your own context on the map and the report, and delegate the reading.

The honesty rule, which is load-bearing

Every check gets one verdict:

• strong: the code does this, and you read the file that proves it.
• partial: it holds on some paths and not others. Name both.
• gap: the code does not do this, and you read enough to say so.
• unknown: you could not settle it from what you read.

Unknown is not a gap. Absence of evidence is a question, never a mark against the system, so an unknown becomes a sharp question the owner can answer in one sentence. A check you skipped is unknown, not strong. Guessing in either direction makes the whole report worthless, because the reader can no longer tell which verdicts were earned.

Absolute constraints

• Change no file except the report. No commits, no installs, no migrations, no deploys, no formatting and no fixes: what is worth fixing gets written down, not done.
• Read no secret value. Check that a variable exists, never what it holds.
• Run nothing that writes: no seeds, no scripts against a database, no call that spends money with a real provider key. Reading files, searching the tree, and the repository's own read-only commands are enough.
• Cite what you read. A verdict with no file and line behind it is an opinion.
• Report what you could not verify, and never soften a real finding.

Phase 1: know the repository first

Grep matches spelling, not meaning: a check written against one client finds nothing in a repository that uses another, and zero hits then reads as clean. So map before you judge, fanned out, and write the map at the top of docs/HARDENING-AUDIT.md:

• The stack: framework, HTTP layer, database client or ORM, auth, validation, test runner, CI and deployment target, read from the manifest and the lockfile rather than the README.
• What each dependency decides: does the database client throw or resolve its errors, is there a mutation path that bypasses the API, does the database authorise rows or connect as one pooled role, and where is the model actually called.
• The shape, from real files: one route, one service, one schema, one test, one model call.
• The instruction layer: AGENTS.md, CLAUDE.md, rules, agent definitions, hooks, cursor and copilot files, ADRs. Claims to check, not truth.

Then translate every check's Look into this repository's own paths and symbols. A check whose mechanism does not exist here (no agent, no tenancy, no money path) is not applicable with the reason, never a gap.

Phase 2: run the checks

One subagent per principle, in parallel. For every check it returns the id, the verdict, the evidence as path:line with at most three quoted lines, one sentence of why, and what it did not read. Read the files: do not answer from a framework's reputation, from a dependency's documentation, or from what the code is probably doing.

The checks

1. Own your context

Own your context, rent the model. The bar: The client owns their data and embeddings, exportable and portable, so the model is a swappable part.

• ownership.export-is-complete (high) Can a client take everything they own in one call?
  • Clears: One export path that enumerates every tenant-scoped table, with a test that derives the table list from the schema rather than restating it.
  • Fails: An export covering the tables someone remembered, which silently stops covering the ones added later.
  • Look: the export route; any export test; columns named tenant_id / workspace_id / account_id
• ownership.embeddings-are-yours (high) Do the vectors live in a database you control?
  • Clears: Embeddings stored in your own store, re-derivable from source documents you also hold.
  • Fails: The only copy of the index lives inside a vendor, so leaving means re-ingesting everything from scratch, if you still have it.
  • Look: the vector store client; the ingestion path; whether the source text is retained after embedding
• ownership.model-is-swappable (medium) Is the model behind one seam, or wired into call sites?
  • Clears: A single provider module and model ids as typed constants, so swapping a model is one edit.
  • Fails: A vendor SDK imported directly wherever generation happens, with model ids as loose strings.
  • Look: imports of provider SDKs across the tree; where model id strings appear
• ownership.deletion-is-real (critical) Does delete reach the things that are not rows?
  • Clears: One teardown primitive that destroys external resources before the database cascade, reused by every delete path.
  • Fails: A foreign-key cascade treated as the whole deletion contract, while provisioned projects, repositories, subscriptions and VMs keep running.
  • Look: delete services; any column holding an external id or ref; what happens to those on delete

2. Grounded or it doesn't answer

If it can't cite it, it shouldn't say it. The bar: Answers come only from retrieved sources, are cited, and the system refuses when the answer is not there.

• grounding.context-is-retrieved (critical) Does the answering call see only what was retrieved?
  • Clears: Retrieval runs first and its output is the only source material in the prompt.
  • Fails: A system prompt full of background the model can blend with, so you cannot tell a retrieved fact from a remembered one.
  • Look: the prompt assembly for the answer path; what else is concatenated alongside the retrieved chunks
• grounding.citations-are-derived (critical) Are citations built from retrieved ids, or written by the model?
  • Clears: The renderer resolves citations against the chunks actually retrieved, and a citation that resolves to nothing is caught.
  • Fails: The model emits citation markers as free text and the UI renders them, so a fabricated source looks identical to a real one.
  • Look: how citations are parsed and rendered; any check that a cited id was in the retrieved set
• grounding.refusal-is-code (critical) Is refusal enforced in code, or only requested in the prompt?
  • Clears: A predicate decides refusal from the retrieval result, independent of what the model chose to say.
  • Fails: The instruction says to refuse when unsure, and nothing verifies that it did.
  • Look: a grounding or refusal predicate; whether the answer path can return an ungrounded answer at all
• grounding.the-claim-is-checked (high) Does the interface claim a refusal only when one happened?
  • Clears: What the user is told is derived from the same signal that is recorded, and the showable claim is never weaker than the recorded one.
  • Fails: A reassuring card saying the system stopped rather than guess, rendered over an answer it actually generated.
  • Look: the refusal or trust component; the field it reads; whether that field is the recorded one
• grounding.evals-cover-refusal (high) Does the eval set contain questions that must NOT be answered?
  • Clears: A golden set with explicit refusal cases, run when retrieval, chunking, the model or the grounding prompt changes.
  • Fails: An eval that only measures recall, which a model that answers everything confidently passes.
  • Look: the eval set; the ratio of answerable to unanswerable cases

3. Untrusted content is data, not instructions

Retrieved and user content is data the model reads, never orders it follows. The bar: A hard boundary: target text, documents, and tool output cannot trigger tools or actions.

• injection.untrusted-is-fenced (high) Is retrieved and user content marked as untrusted where it enters the prompt?
  • Clears: Every untrusted block is delimited and labelled as reference material that is never an instruction.
  • Fails: Document text pasted into the prompt indistinguishable from your own instructions.
  • Look: how retrieved chunks are formatted into the prompt; any untrusted marker
• injection.no-tools-on-the-grounded-answer (critical) Can retrieved content reach a tool call?
  • Clears: The call that answers from untrusted context has no tools bound to it, so there is nothing for injected text to trigger.
  • Fails: One agent loop with both retrieval output and write tools available, where a crafted document can call them.
  • Look: the tool list on the answer call; whether retrieval and tool use share a loop
• injection.extractors-are-tool-free (critical) Does the component that reads hostile input have tools at all?
  • Clears: Anything parsing inbound email, uploads or third-party content returns structured data and can call nothing.
  • Fails: An extractor with tool access, so the content it parses decides what it does.
  • Look: inbound email, upload and webhook processing; the tool set on each
• injection.tool-args-are-validated (high) Are tool arguments validated server-side before they execute?
  • Clears: Every tool argument is schema-checked at the boundary, with bounds shared with the API schema rather than restated.
  • Fails: Arguments trusted because the model produced them, so a prompted id reaches a query.
  • Look: tool definitions; where their inputs are parsed; whether bounds are duplicated
• injection.commands-are-allowlisted (critical) If the agent runs commands, is the set allowed or merely filtered?
  • Clears: An explicit allowlist of permitted commands, denying everything else by default.
  • Fails: A denylist of dangerous patterns, which is a guess about every command you did not think of.
  • Look: any command execution path; how the permitted set is expressed

4. The agent proposes, a human decides

Autonomy stops at anything you can't undo. The bar: Destructive or irreversible actions need explicit approval, and every change is attributed and reversible. Audit trails are append-only.

• reversibility.writes-land-on-a-branch (critical) Does agent-authored code reach the default branch directly?
  • Clears: Changes land on a branch as a pull request a human reviews. The default branch is never written by the agent.
  • Fails: Commits pushed straight to main, where the review step is whatever the reviewer notices afterwards.
  • Look: the push path; which ref it targets; whether a PR is opened
• reversibility.approval-precedes-the-irreversible (critical) Is approval required before anything you cannot undo?
  • Clears: Irreversible actions block on explicit human approval. Reversible ones do not, so the gate keeps its meaning.
  • Fails: Either no gate at all, or a gate on everything, which trains people to click through it.
  • Look: the approval flow; which actions require it; which do not
• reversibility.approved-is-what-lands (high) Is the diff that was approved provably the diff that lands?
  • Clears: The base is compared between approval and push, and an ambiguous or truncated comparison blocks rather than proceeds.
  • Fails: Approval on a diff computed against a base that has since moved, so the merge contains work nobody reviewed.
  • Look: what happens between approval and push; any base comparison; what it does when it cannot tell
• reversibility.trail-is-append-only (high) Is the record of who did what append-only and attributed?
  • Clears: An audit trail nothing updates or deletes, recording actor, action and time, including the operator's own access.
  • Fails: A log with an update path, or one that records the client's actions but not the vendor's.
  • Look: the audit table; its grants; whether operator access is recorded too
• reversibility.secrets-are-scanned-before-they-leave (high) Is agent-written code scanned for secrets before it is pushed?
  • Clears: A scan runs before the push, and a hit blocks it rather than annotating it.
  • Fails: Scanning after the fact, or not at all, so a generated example key becomes a real commit.
  • Look: any secret scan; whether it gates the push or reports on it

5. Two layers or it's one bug from a breach

Auth in the app is not isolation. Enforce it at the database too. The bar: Every read and write is tenant-scoped, with row-level security and least-privilege grants underneath the app checks.

• isolation.every-row-is-scoped (critical) Does every content row carry a tenant id, and does every query filter on it?
  • Clears: One tenant column on every content table, and services that filter by the RESOLVED tenant on every read and write.
  • Fails: A table that joins its way to a tenant, so one missing join is a cross-tenant read.
  • Look: the schema for tenant columns; service queries for the filter; any table lacking one
• isolation.the-database-enforces-it-too (critical) Is there a second layer under the application checks?
  • Clears: Row-level security on tenant tables with policies, and closed grants on anything holding tokens, money or infrastructure state.
  • Fails: Application checks only, so one missing filter is a breach rather than a bug.
  • Look: which tables have RLS enabled; which have policies; which grants exist for the anonymous and authenticated roles
• isolation.tenant-comes-from-the-session (critical) Where does the tenant id come from?
  • Clears: Resolved from the authenticated user on every request. A tenant id in a request body is never trusted.
  • Fails: An id read from the body or a query parameter, which makes every endpoint a tenant switcher.
  • Look: route handlers; what the tenant id is derived from; any body field naming a tenant
• isolation.a-miss-is-a-404 (medium) Does a foreign id return 404 or 403?
  • Clears: 404, so the response does not confirm that the resource exists.
  • Fails: 403 on a resource belonging to someone else, which is an existence oracle you can enumerate.
  • Look: the not-found path in the tenant gate; what status a foreign id produces
• isolation.privileged-keys-stay-server-side (critical) Can a service-role or admin key reach the browser?
  • Clears: Privileged clients are constructed only in server modules, and the build would fail if one were imported into client code.
  • Fails: A single shared client, or a key in a variable the bundler exposes.
  • Look: where the privileged client is built; whether those modules are server-only; public environment variable names

6. One boundary: API-first, validated, typed

One server boundary that validates everything. No business logic in the client. The bar: Every input is schema-validated at a single API boundary; the backend is framework-agnostic; no mutations from the client.

• boundary.one-validated-entry (high) Is every input schema-validated at one server boundary?
  • Clears: One validation helper and schemas that live with the domain, imported by every surface that accepts that input.
  • Fails: Hand-rolled checks per handler, and bounds restated in each tool and form, which drift apart.
  • Look: the validation helper; where schemas are defined; any handler parsing input itself
• boundary.the-gate-runs-first (critical) Does authorization happen before any data work?
  • Clears: The tenant gate is the first statement in the handler, returning early on failure.
  • Fails: Validation, lookups or writes before the gate, so an unauthorized request still moves the database.
  • Look: the first lines of each handler; anything before the gate
• boundary.the-client-does-not-write (critical) Can the browser write to the database directly?
  • Clears: All mutations go through the server boundary. The client holds no write credential.
  • Fails: A public key with insert rights, where correctness depends on rules written somewhere else.
  • Look: client-side database calls; the grants on the public role
• boundary.handlers-declare-their-contract (medium) Do handlers state their return type?
  • Clears: An explicit return annotation on every handler, so a drift in what it returns is a compile error.
  • Fails: Inferred return types, which the compiler accepts happily while the response shape changes underneath.
  • Look: handler signatures; any without an annotation
• boundary.logic-lives-in-services (medium) Do route bodies do the work?
  • Clears: Handlers gate, validate and shape the response. Services own the work and the tenant scoping.
  • Fails: Database work inside handlers, or services returning HTTP responses, which puts the security boundary in two places.
  • Look: handler length; database calls inside handlers; HTTP types imported into services

7. Bounded by design

An agent with no limits is a bill and an outage waiting to happen. The bar: Step, retry, and cost caps; timeouts on external calls; idempotent and compensating side effects; fail closed.

• bounded.loops-have-caps (high) What stops an agent loop that does not converge?
  • Clears: Explicit step, retry and cost ceilings, enforced by the runtime rather than by the prompt.
  • Fails: A loop that runs until the model decides to stop, which is a bill with no upper bound.
  • Look: the agent loop; max step and retry settings; any per-run cost ceiling
• bounded.external-calls-time-out (medium) Does every outbound call carry a timeout?
  • Clears: A timeout on every external call, chosen deliberately rather than inherited from a default.
  • Fails: A hung provider holding a request open until the platform kills it, taking the work with it.
  • Look: fetch and SDK calls to third parties; timeout or abort signals
• bounded.retries-are-idempotent (critical) Is a retried unit of work charged and applied once?
  • Clears: A deterministic key derived from the CONTENT of the work, so a resumed or re-delivered job collapses onto the same record.
  • Fails: A key derived from position, a cursor or a parent id, which produces a fresh key on resume and bills the same work twice.
  • Look: idempotency keys; what each is derived from; whether a resumed batch would produce the same one
• bounded.spend-fails-closed (critical) What happens at zero balance, and what happens when the balance cannot be read?
  • Clears: A hard, explicit block at zero, and a read failure that fails closed rather than assuming credit.
  • Fails: Silent degradation, or a database error read as no balance found and treated as fine.
  • Look: the spend gate; every balance read; what each does with an error result
• bounded.stuck-work-is-reaped (high) What recovers work that dies between two states?
  • Clears: Every non-terminal status has a sweeper that does not depend on the in-request error handler, because a hard kill never runs it.
  • Fails: A status moved to terminal only inside a catch block, so an out-of-memory kill leaves it stuck forever.
  • Look: every non-terminal status value; what sets it to terminal; whether anything sweeps it
• bounded.scans-are-bounded-and-say-so (medium) Are unbounded reads capped, and does the surface admit when it truncated?
  • Clears: A cap on any query that grows with the data, and a surface that states it is showing a subset.
  • Fails: A page that loads everything, or worse, one that caps silently and renders the capped number as the total.
  • Look: queries with no limit; operator pages; whether a truncated count is labelled

8. You can tell when it's wrong

If you can't trace a bad answer back to its cause, you can't trust the good ones. The bar: Structured logs on success and failure with context, a trace from a finding back through the steps, and evals. Never a silent catch.

• observability.no-silent-catch (high) Is there a catch block that discards its error?
  • Clears: Every failure path logs with enough context to identify the tenant and the operation.
  • Fails: An empty catch, or one that swallows and returns a default, which is a failure nobody will ever see.
  • Look: catch blocks; which ones log; which return a fallback quietly
• observability.writes-check-their-result (critical) Does the data layer throw on failure, or resolve with an error you have to read?
  • Clears: If it resolves, every write reads the error and handles it, and webhook paths rethrow so the provider re-delivers.
  • Fails: An awaited write whose result is discarded, which is a fully silent data loss on a constraint or a timeout.
  • Look: awaited writes with no destructured error; webhook handlers; whether a failure there returns success
• observability.money-reads-fail-closed (critical) What does a failed read on a billing path return?
  • Clears: It throws. A transient database error is never allowed to read as zero, empty or not found on a money path.
  • Fails: A wallet or subscription read that swallows its error, so a blip becomes a wrong charge or a free tier granted forever.
  • Look: reads on wallet, subscription, ledger and customer records; what each does with an error
• observability.signals-reach-somebody (high) Where do the loud signals actually go?
  • Clears: Logs reach a drain someone can query after the fact, so a mitigation whose enforcement is that it logs loudly is real.
  • Fails: Structured error events written to standard output with no drain, which is detection nobody receives.
  • Look: the logger implementation; whether a drain or sink is configured
• observability.a-bad-answer-is-traceable (high) Can you get from a wrong answer back to what produced it?
  • Clears: A record per generation carrying what was retrieved, which model ran, what it cost, and whether it was grounded.
  • Fails: A response with no trace, so the only debugging tool is asking again and hoping it happens twice.
  • Look: the per-generation record; whether retrieval context is recoverable from it

Phase 3: refute

A different agent re-reads the evidence behind every gap and every critical verdict, and returns confirmed, refuted or downgraded with its reason. An author cannot check their own finding, for the same reason nobody proofreads their own writing well. Only confirmed findings reach the summary; refuted ones stay in the report, marked, so the next reader does not rediscover them.

Phase 4: the report

Write docs/HARDENING-AUDIT.md: the map, then the findings, then the questions.

• Lead with what it costs when it is wrong, not with what is easy to fix: critical first, then high, then the rest.
• One line per check (id, verdict, evidence, one sentence), grouped by principle, with a one-line verdict for each principle.
• A section of unknowns as questions, each phrased so the owner can answer it in a sentence.
• A section of what you did not read, and why.
• Close with the three things you would fix first, in order, and what each costs if it is left.

State plainly what this is: a point-in-time read of the code you could see. It is not a penetration test, not a compliance certification, and not a guarantee.