01At a glance
11 / 13
issues done; 2 wait for your Fly.io account and for 14 Oct
0
personal data values reached the model in gateway tests, logs scanned too
369 / 369
shared test cases pass in Go, written separately from the TypeScript version
100%
recall on every supported writing style (76.5% over a deliberately varied mix)
~3 ms
added per request; 2.9 ms of it is saving the audit record to disk
US$0
spent; at most US$0.0076 a request and US$0.20 a day with a real model
02What Discreet does
An app that already talks to OpenAI uses Discreet by changing one setting. Checked with OpenAI's own Python library.
The appSends a normal chat request, plus a header saying what it's for (for example "claims summary").
DiscreetRefuses eligibility decisions · finds NRIC, phone, email, card, postal code, unit number, date of birth · swaps them for placeholders · keeps the real values encrypted for minutes · writes an audit record.
The AI modelA free mock for everyone; a real model only for holders of an access token, under a daily budget.
App sendsCall S1234567D on 9123 4567 about claim C-2041
AI seesCall <NRIC_1> on <PHONE_1> about claim C-2041
App gets back…S1234567D… 9123 4567… plus a footer saying personal data was replaced
The NRIC is the synthetic worked example from sg-pii-rules. Card numbers are redacted and never put back.
03What was built
One pull request per issue, each with CI (race-checked tests, govulncheck, CodeQL, a container smoke test), reviewed by Vetted, merged into next by Claude Code with the ai-merged label.
| Issue | In plain words | Pull request | Status |
|---|---|---|---|
| D1 | Go project, checks in CI, evidence files | discreet#4 | merged |
| P1 | New rules: payment cards, postal codes, unit numbers | #11 #12 #13 | merged |
| P2 | Dates of birth; release candidate sg-pii-rules v0.2.0-rc.1 | sg-pii-rules#14 | merged |
| D3 | Go detection engine passing all 369 shared cases | discreet#5 | merged |
| D4 | Placeholders, encrypted vault, YAML policy | discreet#6 | merged |
| D7 | Tamper-evident audit log with verify and CSV export | discreet#7 | merged |
| D5 | The gateway, mock model, purpose refusal, footer | discreet#8 | merged |
| D6 | Real model behind access tokens, budget, rate limit | discreet#9 | merged |
| D8 | Four-pane demo page | discreet#10 | merged |
| D9 | Benchmark and latency | discreet#11 | merged |
| D10 | Container, Fly.io config, deploy workflow | discreet#12 | ready; needs your Fly.io account |
| D11 | Controls matrix, threat model review, risks, CV numbers, pilot numbers, this report | discreet#14 · vetted#45 · sg-pii-rules#16 | merged |
| V1 | Vetted pilot on Discreet: shadow → opt-in | discreet#13 | waits for 14 Oct |
04The evidence
5,000 synthetic samples from a generator written separately from the rules, on purpose including ways of writing personal data the rules don't handle yet. A value counts as caught only if it's covered completely.
Recall per kind of personal data
Share of true values fully caught, on 5,000 samples (7,531 values). Every supported writing style scored 100%; the gaps are styles not supported yet.
Show as a table
| Entity | Caught / total | Precision |
|---|
What the misses are
- NRICs written
S 1234567 Dor with hyphens - Phones written
91 23 45 67or912 34567 - Cards with double spaces between groups
- Emails written "name at example dot com"
- Postal codes with no clue, or after "S'pore"
- Unit numbers without
# - Dates of birth after "Birthday", with words in between, or with a two-digit year
False alarms: 482, all bare eight-digit order and batch numbers read as phone numbers, the trade-off accepted in sg-pii-rules ADR 0007. No other kind had any. Precision overall 92.3%.
| Latency (2,000 requests) | p50 | p95 |
|---|---|---|
| Baseline: same HTTP and JSON work, no Discreet | 0.04 ms | 0.11 ms |
| Through Discreet | 3.05 ms | 4.90 ms |
| of which: detection | 0.08 ms | 0.23 ms |
| of which: audit record saved to disk | 2.88 ms | 3.63 ms |
| Vetted's pilot on Discreet | Count |
|---|---|
| Pull requests reviewed (shadow, mock model) | 12 (37 runs) |
| Secret-shaped strings scrubbed before the model | 18 |
| Personal data values scrubbed before the model | 89 |
| Comments posted / cost | 0 / US$0 |
All values were synthetic. Synthetic data flatters rule-based detectors, so treat recall as an upper bound for the styles covered. Full numbers and method: Discreet's EVALS.md.
05Decisions to review
Each is an ADR marked "drafted by Claude Code, awaiting Michael's review". Reply "approve", or name the ones to change.
| ADR | Decision, in one line | What you might change |
|---|---|---|
| Discreet 0001 | Go's standard library only, no web framework: one small binary, little to explain or patch | - |
| Discreet 0002 | Same next integration-branch workflow as Vetted | - |
| Discreet 0003 | The rules are compiled into the binary from a checksummed sg-pii-rules release | - |
| Discreet 0004 | Numbered placeholders per request; values encrypted in memory for at most 10 minutes; strict YAML policy; cards redacted by default | Whether cards should be tokenised instead |
| Discreet 0005 | Plain-text audit log where each line holds the previous line's fingerprint; prompts stored only as keyed fingerprints; limits stated | When to anchor the head hash externally (M3) |
| Discreet 0006 | Callers must state a purpose; eligibility and benefits decisions refused before any model call; only known fields forwarded; footer on every answer | The refused-purpose wording |
| Discreet 0007 | Real model only with an access token; one pinned model; each request reserves its worst-case cost against US$0.20 a day; 30 requests a minute per address | Budget and rate defaults |
| Discreet 0008 | The demo page runs the API's own code path, always on the mock | - |
| Discreet 0009 | Public demo mock-only on one small Fly.io machine in Singapore, in a no-shell, non-root container | Region or app name |
| sg-pii-rules 0009 | Cards need both the Luhn check and a card network's prefix and length | - |
| sg-pii-rules 0010 | A rule can require context (like "Singapore") without redacting it | - |
| sg-pii-rules 0011 | Postal codes only after an address clue | - |
| sg-pii-rules 0012 | Dates of birth only right after a birth clue; the check never uses today's date | - |
06What the AI got wrong, and how it was caught
All logged in the repositories' build logs. None reached next uncaught.
| What went wrong | Caught by |
|---|---|
| A fake cloud key for the Vetted film contained a "9", which real keys never do | Running the sample through the real engine before animating |
| Checksums regenerated before the formatter rewrote a file; push and PR steps not chained after the commit | The gate script refused to commit |
| A release was created with a short commit ID and failed, while an unchained command closed its issue anyway | Reading the output straight away; release created a minute later, issue corrected |
| Two deliberate "break it" tests were written wrongly (one unbalanced bracket, one Go compile error), so they proved nothing at first | Suspiciously low failure counts; both redone so they test the real thing |
| The mock model said "exactly as a real model would see it" above text the reader sees restored | Reading the reply in OpenAI's Python client |
A test searched JSON-escaped text for <NRIC_1> | The failing test, read before touching gateway code |
| The merge script stopped waiting before CI had started | It then refused to merge, the safe outcome; it now waits for every required check |
A local check piped JSON through zsh's echo, which mangled it | The parse error, checked against the raw response |
07Left out, or not finished
- The public deployment (D10) is ready but needs your Fly.io account. The container was built and smoke-tested locally and in CI.
- The pilot switch (V1) waits for 14 Oct, the end of the two-week shadow period.
- No real model has been called. Everything ran on the mock because
OPENAI_API_KEYisn't set; Ollama is supported by configuration but untested against a real one. - Vetted still uses sg-pii-rules v0.1.0, so Discreet's synthetic cards, postal codes, unit numbers and dates of birth reached Vetted's mock model unscrubbed. Upgrade in M3.
- The shared tests had no case for the overlap tie-break; each implementation tests it, and vetted#44 tracks adding shared cases.
- Milestone 3 per the brief: hold-for-review queue,
sg-healthcareandsg-financepresets, Project Moonshot run, Vetted's dashboard, v1.0 of both, the toolkit page. Names (NER) after v1.
08Your part
- Review the three pull requests (about 30 minutes): discreet#15, sg-pii-rules#17, vetted#46. Try the demo with Docker (instructions in discreet#15).
- Decide on the 13 decisions above: reply "approve", or name the ones to change.
- Merge each with rebase merge, or tell me to. Then I tag Discreet v0.1.0 and sg-pii-rules v0.2.0 (byte-identical to the release candidate) and realign the
nextbranches. - Set up Fly.io (about 10 minutes, docs/DEPLOY.md): account, app, volume, key, deploy token. The deploy then runs from
mainand tests the live site. - Set
OPENAI_API_KEYwith a project spend limit, so the pilot and Discreet can produce real-model numbers. - On or after 14 Oct, say "merge V1", or I'll merge it in the next session after that date.