Your part
A 30-minute review
Nothing reaches main or a release until you merge it. That's the point of the integration branch (ADR 0009). Here's the order I'd do it in.
Read this page you
About 15 minutes. The decisions and what went wrong sections matter most.
Merge the three review pull requests with Rebase and merge you
vetted#25, sg-pii-rules#9, discreet#2. Rebase (not squash) keeps each change as its own commit on
main. Also merge the pull request that publishes this page on makoydev.github.io.Say "merged" Claude Code
Then I tag
v0.1.0onmain: sg-pii-rules first (byte-identical to the release candidate Vetted already vendors), then Vetted. I also publish the GitHub Releases, move Discreet's pin to the release, and close the milestone.Optional: switch on the real model optional
Create an OpenAI project with a monthly spend limit, then in your own terminal (not in a chat):
gh secret set OPENAI_API_KEY -R makoydev/vetted, and the same formakoydev/discreet. Until then everything runs on the free mock, and the evidence says so.14 October: Discreet goes from shadow to opt-in Claude Code
Two weeks of shadow mode, as the brief planned. The switch is a one-line pull request.
How we worked
One change you made to the rules
Before 30 September
Claude Code drafted each change as a pull request. You reviewed and merged every one, and answered decision questions as they came up. You merged 5 pull requests this way, and changed two of my proposals: the model (GPT-6 Luna instead of gpt-5-nano) and the NRIC look-alike detector (build it now).
From 30 September (ADR 0009)
You asked me to work through without waiting. You chose the integration branch option: each change still got its own pull request and full CI, and I merged it into a protected next branch labelled ai-merged. 17 pull requests went through that way. main never changed.
Why this matters for the portfolio: merges I make appear on GitHub as made by you, because I act through your account. The ai-merged label, the build log and ADR 0009 keep the record honest: they show what was automated, what gated it (tests, the canary suite, CodeQL), and that a human reviews the batch before anything ships.
What Vetted does
Eight steps on every pull request
Open a step to see why it exists and where its evidence is.
01Gate the runShadow mode reviews everything but only writes an audit record; opt-in reviews labelled pull requests and comments; forks get the free mock
Why it matters: a team can watch what the AI would say before letting it say anything. That's the gradual rollout IMDA's frameworks describe. Only pull_request events are accepted: pull_request_target would hand secrets to fork code.
Evidence: #15, a 9-row test table of mode × label × fork × key. Config is read from the base commit, so a pull request can't weaken its own review (ADR 0007).
02Keep sensitive files out.env files, keys, credential files, lockfiles, generated code and binaries are never sent; oversized files are skipped whole and listed
Why it matters: the cheapest leak to prevent is a file that never leaves. The deny list is a floor: a team's config can add to it but can never re-allow a .env. Files are never cut in half, because a truncated file would be reviewed as if it were complete.
Evidence: #16. A canary that only the deny list can stop leaks when the deny list is removed.
03Scrub secretsgitleaks' 221 rules run with a linear-time regex engine, plus an entropy check for secrets no rule names
Why it matters: using the industry's standard rule set is easy to defend, and proving the port faithful is what makes it credible: 0 misses against the real gitleaks binary. Where gitleaks would let a secret through (for example by honouring an "allow" comment written by the same person whose secret it is), Vetted is deliberately stricter.
Evidence: #17, ADR 0010, the parity table and the entropy calibration.
04Scrub Singapore personal dataNRIC/FIN (including mistyped ones), phone numbers and emails become <NRIC>, <PHONE>, <EMAIL>
Why it matters: PDPC's guidance treats NRIC numbers as especially sensitive, and sending them to an overseas AI provider is a transfer out of Singapore. The rules live in their own repository with checksummed releases, and Vetted's implementation passes 184 / 184 shared test cases, the same ones Discreet's Go version will have to pass.
Evidence: #18, sg-pii-rules v0.1.0-rc.1. VALIDATORS.md says plainly that the NRIC checksum was never officially published.
05Check the budgetEach review has a hard worst-case cost; today's worst case must fit the daily budget, or the free mock is used
Why it matters: the brief caps spend. Measuring spend after the fact can't stop a runaway, but a bound computed from hard caps can. Diff size is capped in bytes, and a token is never smaller than a byte, so the input cost has a true ceiling. Anything uncertain, such as an unknown model price, falls back to the mock.
Evidence: #19, ADR 0004, ADR 0011, and the cost bound below.
06Ask the model, safelyThe diff is wrapped in markers with a random nonce and labelled untrusted; no tools; strict JSON schema; store: false
Why it matters: prompt injection is OWASP's number one LLM risk. The model is told the diff is data, not instructions, and an attacker can't fake the end of the data block without guessing 64 random bits. The client refuses to exist inside the test suite with real network access, so tests can never spend money.
Evidence: #19, src/model/prompt.ts, src/model/openai.ts.
07Validate, detect, and commentInvalid output is discarded; injection attempts are reported by rules; comments are tagged [AI], always posted as COMMENT
Why it matters: the model's output is untrusted too. Links and images are removed before posting, because a rendered image URL is a known way to leak data out of an AI system. Injection findings are posted even when the model's output is unusable, so breaking the model can't also hide the attack. There's no code path that approves or requests changes, and a test checks for it.
Evidence: #20, ADR 0012. CodeQL caught my first HTML sanitiser, which could be bypassed; it's fixed and has a test.
08Leave an audit recordEvery run, including skipped and failed ones: prompt hash, model, tokens, cost, budget, scrub counts; never code
Why it matters: governance needs evidence of what happened, and the evidence mustn't become a leak itself. The record has hashes and counts only, is checked against a schema, and is uploaded as a workflow artifact kept for 30 days. It's the data Milestone 3's dashboard will be built on.
Evidence: #21. Real records: Vetted reviewing its own #21 and Discreet's first pull request.
Decisions
Twenty decision records
Each one records the options rejected. Four are drafted and waiting for you; two record a change you made to my proposal.
Evidence
What the numbers say
Every number here is measured and reproducible with the command named next to it, and each one states what it doesn't show. None of them come from a real AI model yet (see Not done yet).
The canary suite leaks nothing, and fails when a protection is removed
Leaks reported by npm run canary (144 canary values, 34 runs) with the code intact, and with one protection removed on purpose
Show as a table
| Configuration | Leaks |
|---|---|
| Everything on (as shipped) | 0 |
| Path deny list removed | 3 |
| Personal-data scrubber removed | 71 |
| All scrubbers removed | 165 |
Choosing the entropy threshold
Share of 2,000 random 32-character secrets caught, by threshold (npm run measure:entropy)
Show as a table
…against false alarms in real code
Tokens flagged per MB of real third-party JavaScript (28.4 MB), by threshold
Show as a table
Vetted uses 4.3: past it, the catch rate for 32-character secrets collapses, while false alarms have mostly flattened (most are base64 source maps, safe to redact).
Parity with the real gitleaks 8.30.1
npm run parity:gitleaks, with decoding off
| Corpus | Cases | gitleaks | Vetted | Missed |
|---|---|---|---|---|
| gitleaks' own test strings | 3,658 | 405 | 470 | 0 |
| Real source files (10 MB) | 2,028 | 2 | 2 | 0 |
All 65 extras are explained: 60 are gitleaks hiding a duplicate on a line another rule already covered (same redaction), 2 are "allow" comments Vetted ignores on purpose, and 3 are in a tar archive gitleaks skips as binary.
A spend cap by construction
Worst case, from hard caps (src/model/pricing.ts): not a measurement
| Most one review can cost | US$0.0104 |
| Paid reviews allowed per day (US$0.20 budget) | 19 |
| Most a repository can cost per month | US$6.20 |
| Spent so far (mock model only) | US$0 |
Worst case for both piloted repositories vs the brief's US$20 a month
Personal data: 184 / 184
Shared conformance cases pass (NRIC 53, look-alikes 46, phone 41, email 44). Honest limit: on real code the phone detector fires about 40 times per MB, almost all on numeric constants such as 67108864 (2²⁶). The cost is context, never data.
Injection: 6 / 6, 0 false alarms
Payloads detected in the canary suite, with no alarm on 11 clean pull requests. Its first real catch: the 7 attack examples in Vetted's own test files. Honest limit: these are pattern rules, and novel wording gets past them, which is why nothing depends on them alone.
Pilot: 1 pull request
Discreet's first pull request was reviewed in shadow mode and produced an audit record. Honest limit: a sample of one, a single maintainer and the mock model. There are no quality numbers yet; that's the job of Milestone 3's dashboard.
What went wrong
Every mistake I made, and what caught it
All of these are in the build logs. They're part of the evidence: they show AI-assisted work being checked, not just trusted. The recurring lesson was that checks only help if nothing can quietly bypass them.
Not done yet
Deliberately left for later
Needs you
- Real-model numbers (GPT-6 Luna vs gpt-5-nano, real cost per review, review quality): they need the
OPENAI_API_KEYsecret, which only you set. - Final v0.1.0 releases: after your merge. Releases come from
mainonly.
Milestone 3, as planned
- Metrics branch and dashboard: acceptance, overrides and cost per pull request. This closes the override-rate gap in CONTROLS.md.
- Default-on mode, suppressions with an expiry date, the Marketplace listing.
- Incident playbook, policy template, pilot plan.
Known gaps, stated in the docs
- Base64-encoded secrets aren't decoded (gitleaks does this); the entropy check covers long ones.
- Names and addresses aren't detected; that needs NER, planned for Discreet.
- A pull request can edit its own workflow file. Vetted is advisory, so it's deliberately not a required check; the batch review is the backstop.
Small calls I made on my own
- Closed four major-version Dependabot upgrades: TypeScript 7 breaks the build, and Node 26 types don't match the runtime.
- Made CodeQL and the canary suite required checks, after CodeQL caught two real issues.
- Size cap in bytes, not characters, so the cost bound is real. Entropy threshold 4.3. Look-alike NRICs scrubbed too.
- Vetted reviews its own pull requests from a pinned, merged version, never the code under review.
Risks
What to watch
| Risk | Now | What's being done |
|---|---|---|
R11 A flaw merged into next by AI that CI didn't catch | Medium | Your batch review; CodeQL and the canary suite are required; every AI merge is labelled |
| R12 No real-model results | High | Mock everywhere, and it says so; the comparison is pending your key |
| R9 Pilot too small to mean much | High | Sample sizes are stated; opt-in from 14 October; replay evaluation in M3 |
| R1 Schedule | Low (was High) | M1 was built on 30 September against a 25 October due date; the remaining time goes to review and M2 |
Frameworks
Mapped to Singapore's rulebook
18 Vetted features and 10 sg-pii-rules features are mapped to specific clauses, with page references from primary sources read on 30 September. Where a framework only applies by analogy, the matrix says why.
Vetted's CONTROLS.md · sg-pii-rules' CONTROLS.md · Threat model · Evaluation
Interview crib
If you're asked about Milestone 1
- "What is it?" A GitHub Action that lets a team use AI for code review without sending secrets or Singapore personal data to the provider, without letting text in the code steer the AI, and without the AI ever approving anything.
- "How do you know it doesn't leak?" A canary suite pushes pull requests full of fake passwords and identity numbers through the real code, even against a model that repeats everything back, and searches every output for them. Zero leaks. I also proved it fails when I remove a protection.
- "Did you write this yourself?" Claude Code drafted it under a written brief. Every change went through a pull request with CI. I merged the early ones myself, then chose an integration branch so the AI could merge into staging while I review the batch before main. Every AI mistake is logged with how it was caught.
- "What would you do differently?" I'd set the API key on day one: the biggest gap in the evidence is that no real model has been measured yet.
- "Why Singapore-specific?" The NRIC is treated as especially sensitive by PDPC, the checksum was never officially published, and sending personal data to an overseas AI provider is a transfer out of Singapore. The controls matrix maps each feature to the specific clause.