diff --git a/README.md b/README.md index 03d6a13..a251a9c 100644 --- a/README.md +++ b/README.md @@ -74,10 +74,10 @@ is byte-for-byte reproducible; `src/serve` imports nothing. | Milestone | State | | --- | --- | -| M0 — prove resolution end to end | Proven **locally**, not yet on a public domain. See below. | +| M0 — prove resolution end to end | Done, locally and publicly, including the cold OAuth path | | M1 — `corpora build` | Done | | M2 — `corpora/serve` and the Node shell | Done | -| M3 — Worker shell and templates | Done, including a real `workerd` smoke test | +| M3 — Worker shell and templates | Done, deployed live; container image built and verified | | M4 — `corpora verify` | Done | | M5 — atcr.io cutover | Blocked on two DNS records (see [ROADMAP](docs/ROADMAP.md)) | @@ -88,9 +88,15 @@ confirm the verification is real — flip a byte in a proof, or name a different `did.json`, and resolution fails. See [LOCAL_E2E](docs/LOCAL_E2E.md) for exactly which links are real and which are local substitutions. -**What localhost cannot prove**, and what M0 still wants: a real public hostname, real DNS, real -TLS, and a check from the Go implementation — `goat` rejects any `did:web` carrying a port, so it -cannot see a local authority at all. +Proven in public too. A throwaway authority at `did:web:lexicon-poc.jarrett.app` served four +lexicons across two authority domains from a Cloudflare Worker, and was verified four independent +ways against one record CID — `corpora verify`, `goat` (Go), `@atproto/lexicon-resolver` with +nothing injected, and a third-party browser tool. A permission set then resolved **cold** through +a real OAuth flow on a live PDS, whose consent screen rendered its title and permissions straight +out of the signed record. + +That last one is the failure this project exists to fix, tested rather than argued: the two +control scopes, naming lexicons that do not exist, came back `invalid_scope`. ## Getting started diff --git a/bin/corpora.js b/bin/corpora.js old mode 100644 new mode 100755 diff --git a/docs/PRIOR_ART.md b/docs/PRIOR_ART.md index 53eb996..88de370 100644 --- a/docs/PRIOR_ART.md +++ b/docs/PRIOR_ART.md @@ -99,6 +99,16 @@ Everything in the chain has been demonstrated: `did:web` lexicon authorities wor libraries build and sign a repo correctly, and Workers can serve the result. Nothing here is research. -The unbuilt part is the combination: an authority whose signing happens offline, whose serving +The unbuilt part was the combination: an authority whose signing happens offline, whose serving tier is stateless and keyless, and which is packaged as a tool other people can point at their own lexicons rather than a deployment of somebody's namespace. + +**Built and demonstrated 2026-08-03.** A throwaway authority signed offline and served from a +Cloudflare Worker holding no key, no state and no bindings resolved through `goat`, +`@atproto/lexicon-resolver`, a third-party browser tool, and a real OAuth flow on a live PDS whose +consent screen rendered the permission set's own title and permissions. + +The result worth recording is a negative one: **nothing on the read path noticed.** No consumer +cared that there was no database, no origin, and no signing key anywhere near the request. The +"Static PDS" pattern Eli Mallon named as missing is, for the read-only lexicon case, just a +compiler and a lookup table. diff --git a/docs/RESOLUTION.md b/docs/RESOLUTION.md index ca3d3d4..d4f15fe 100644 --- a/docs/RESOLUTION.md +++ b/docs/RESOLUTION.md @@ -115,12 +115,18 @@ portable and any host can serve them, but a *new* DID means every record must be ## The proof CAR is the critical path, not a nicety -**Reported, by two independent research agents reaching the same conclusion by different -routes.** The TypeScript resolution chain (`@atproto/lex-resolver` into `OAuthProvider` into -`LexiconManager` into `verifyRecordProof`) calls `com.atproto.sync.getRecord` and runs full -verification: commit signature against the DID document key, `commit.did === did`, MST walk to -`com.atproto.lexicon.schema/`, `$type` check, and `lexicon.id === rkey`. It does not call -`com.atproto.repo.getRecord` at all. +**Verified 2026-08-03**, having previously been reported by two research agents. Against a live +authority, resolving a lexicon produced **exactly one XRPC request — `com.atproto.sync.getRecord` +— and `com.atproto.repo.getRecord` was never called**, on any fixture. + +The TypeScript resolution chain (`@atproto/lex-resolver` into `OAuthProvider` into +`LexiconManager` into `verifyRecordProof`) runs full verification: commit signature against the +DID document key, `commit.did === did`, MST walk to `com.atproto.lexicon.schema/`, `$type` +check, and `lexicon.id === rkey`. + +That verification bites. Tampering tests confirm it rejects a proof CAR with a flipped byte +(`Malformed record proof`), a proof signed by a key the DID document does not name (`Invalid +signature on commit`), and one lexicon's proof served for another (`block not found`). Since an Auth Server resolving an `include:` scope is overwhelmingly a reference PDS, **the proof CAR is what OAuth actually consumes.** Signing correctly is load-bearing. @@ -159,18 +165,28 @@ body. ## Serving details that fail confusingly -**No redirects on `/.well-known/did.json`** (reported). The TypeScript resolvers fetch with -`redirect: 'error'`, so an apex-to-www redirect or a 301 breaks resolution outright. - -**DID document shape is strict** (reported). The parser requires a `verificationMethod` entry -whose id is exactly `#atproto` (or `#atproto`) carrying a `publicKeyMultibase`, and a -`service` entry with id exactly `#atproto_pds` and type exactly `AtprotoPersonalDataServer`. - -**Content type on `did.json`**: one agent reported a strict check matching -`^application/(did\+ld\+)?json$`. This is contradicted by observation. `stream.place`, a working -production lexicon authority, serves its `did.json` as `text/plain; charset=utf-8` (verified). -Either the check is not applied on this path or the report is wrong. Serve `application/json` -anyway; it costs nothing and removes the question. +**No redirects on `/.well-known/did.json`** (**verified**). `redirect: 'error'` is in +`@atproto/identity/dist/did/web-resolver.js`, and a tampering test confirms a 301 on that path +breaks resolution. An apex-to-www rule or a trailing-slash redirect is enough. + +**DID document shape is strict** (**verified for `id`**, reported for the rest). A document whose +`id` is not the DID that was resolved is rejected before anything else is read. The parser also +requires a `verificationMethod` entry whose id is exactly `#atproto` (or `#atproto`) carrying +a `publicKeyMultibase`, and a `service` entry with id exactly `#atproto_pds` and type exactly +`AtprotoPersonalDataServer`. + +**`did:web` must be a bare hostname** (**verified**). A path-based DID throws +`UnsupportedDidWebPathError`; the path branch survives in the source only as a commented-out line. +A port is permitted **only for `localhost`**, per the [DID spec](https://atproto.com/specs/did) — +and implementations disagree here in the dangerous direction: `@atproto/identity` accepts a port +on any host, while `goat` rejects any port at all, including localhost. A port-bearing production +DID would resolve for TypeScript clients and be invisible to Go ones. `corpora build` rejects it. + +**Content type on `did.json`**: an agent reported a strict check matching +`^application/(did\+ld\+)?json$`. **That report is wrong** — no such check exists anywhere in the +installed atproto stack; `DidWebResolver` sends an `accept` header and never inspects the response +type. This also matches observation: `stream.place` serves `text/plain; charset=utf-8` and works. +Serve `application/json` anyway; it costs nothing. **Response size cap** of roughly 1.03 MB (reported). Irrelevant at this scale, but it is another reason not to take the full-CAR shortcut. @@ -205,3 +221,8 @@ cached copy never notice while servers resolving fresh get the new signer. Note that this caching is the reference implementation's behaviour, not a guarantee of the protocol. A user population skewed toward self-hosted PDSes is also skewed toward varied implementations and versions, so some fraction may cache less aggressively or not at all. + +**You can still test the cold path**, despite not being able to flush anyone's cache: a +brand-new NSID has never been resolved by anybody, so its first request *is* the cold path, on +any Auth Server. Publishing a permission set under a fresh namespace and pushing one +authorization request is the whole test. See [ROADMAP](ROADMAP.md#the-cold-path-is-testable-after-all). diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index f35c69e..3a25cab 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -7,12 +7,24 @@ domain, which is the remaining gap before M5. | Milestone | State | | --- | --- | -| M0 — prove resolution end to end | Locally yes, publicly not yet | +| M0 — prove resolution end to end | Done, locally and publicly, including the cold OAuth path | | M1 — `corpora build` | Done | | M2 — `corpora/serve` and the Node shell | Done | -| M3 — Worker shell and templates | Done | +| M3 — Worker shell and templates | Done; deployed live | | M4 — `corpora verify` | Done | -| M5 — atcr.io cutover | Not started; blocked on DNS below | +| M5 — atcr.io cutover | Not started. Blockers below. | + +**M5's blockers, in order:** + +1. **`corpora build` has no cutover mode.** Its DNS check fails on both "record missing" and + "record names a different DID", and the only escape is `--no-dns-check`, which disables both. + That makes cutover steps 2 and 3 — stand up the new authority, verify, DNS untouched — + impossible without also giving up the missing-record check. Needs a third mode: records must + exist, may still name the old DID. +2. **`io.atcr.sailor.webhook` is a deletion.** It is live on the PDS and absent from the repo, so + the first Corpora build is the first publish that drops it. Decide before the flip. +3. **Corpora is not on npm**, so the container and standalone templates cannot install it. + Irrelevant if atcr.io deploys on a Worker with a path dependency, as the POC did. Findings that changed the design as written are recorded where they belong: [BUILD](BUILD.md) for the reproducible `rev` and the validation divergences, @@ -24,12 +36,19 @@ Findings that changed the design as written are recorded where they belong: ### M0. Prove resolution end to end before building a tool -> **Status: done locally, not publicly.** `npm run dev` and `test/e2e/` run the full chain against -> localhost — a real DNS server answering `_lexicon` TXT queries, `did:web:localhost%3A` -> over plain http, and `@atproto/lexicon-resolver`'s own verification path, with tampering tests -> that confirm the verification bites. What remains is the part localhost cannot supply: a public -> hostname, real DNS, real TLS, and a `goat` cross-check. The paragraph below still describes work -> to do, on a throwaway domain, before M5. +> **Status: done, locally and publicly, 2026-08-03.** +> +> Locally, `npm run dev` and `test/e2e/` run the full chain against a local DNS server and +> `did:web:localhost%3A`, with tampering tests confirming the verification bites. +> +> Publicly, `did:web:lexicon-poc.jarrett.app` served four synthetic `app.jarrett.poc.*` lexicons +> from a Cloudflare Worker over two authority domains. Verified four independent ways against one +> record CID: `corpora verify` (25 checks), `goat` (Go/indigo), `@atproto/lexicon-resolver` with +> nothing injected, and PDSls in a browser. +> +> **The cold OAuth path was tested too**, contrary to the caveat below. A permission set resolved +> through a real PAR against a live PDS, and the consent screen rendered its `title` and `repo` +> permissions from the signed record. See "The cold path is testable after all". Publish one lexicon under a throwaway `did:web` on a subdomain and confirm an independent resolver accepts it. No Corpora code required. @@ -45,6 +64,42 @@ but neither is your DNS and your key. Server caches, so verifying the cold path means standing up a fresh PDS you control and running a real OAuth flow through it, or making the resolution requests directly yourself. +#### The cold path is testable after all + +The paragraph above is too pessimistic, and the way around it is worth writing down. + +You cannot flush a cache, but you do not need to: **a brand-new NSID has never been resolved by +anyone, so it is cold by construction.** Publish a permission set under a namespace nobody has +seen and the first request is the cold path, on any Auth Server, including someone else's. + +A pushed authorization request is the whole test, and it is cheap — scope resolution happens at +the PAR, before any browser redirect: + +```console +$ curl -X POST https:///oauth/par -d 'scope=atproto include:' ... +``` + +Run it against a namespace you control, and twice more as controls. Verified 2026-08-03 against a +live PDS at version 0.4.5009: + +| Scope | Result | +| --- | --- | +| `include:app.jarrett.poc.authFull` — published, live | `201` + `request_uri` | +| `include:app.jarrett.poc.nope` — same authority, no such record | `400 invalid_scope` | +| `include:com.doesnotexist.nothing` — no DNS at all | `400 invalid_scope` | + +The controls are the point. Without them a `201` proves only that the server accepted a string. +With them it proves the server resolved DNS, fetched the DID document, pulled the proof CAR, and +verified a signature — and that `invalid_scope` is exactly the error that motivated this project. + +Note that a PDS may not advertise `include:` in `scopes_supported`; the one tested lists only +`atproto` and the `transition:*` scopes and resolves permission sets regardless. Do not conclude +from that metadata that permission sets are unsupported. + +Carrying it through the browser adds one thing worth seeing once: the consent screen renders +`title` and the `repo` permission's collections straight out of the signed record, which is the +concrete reason a forged permission set matters. + ### M1. `corpora build` — done Read a lexicon directory, inject `$type`, key each record by its `id`, build the MST, sign the @@ -65,17 +120,27 @@ one. Done when end-to-end resolution against `localhost` succeeds and responses match what a real PDS returns for the same records. -### M3. Worker shell and templates — code done, live deploy not attempted +### M3. Worker shell and templates — done `corpora/worker`, plus working templates in `examples/` for all three deployment shapes: Worker, container, standalone. Done when `wrangler deploy` produces a live authority and M0's verification passes against it. -> The shell, the three templates, and a `workerd` smoke test via miniflare are done — the Worker -> genuinely boots in the target runtime and serves `did.json` and a proof. The stated criterion is -> not met, because nothing has been deployed to Cloudflare. That step is the same step as M0's -> public half, and both want a throwaway domain. +> Criterion met 2026-08-03: `wrangler deploy` produced a live authority at +> `did:web:lexicon-poc.jarrett.app` and M0's verification passed against it. The bundle is 8.00 KiB +> gzipped with no bindings and no secrets. +> +> The container template was also built and run: the image starts, its `HEALTHCHECK` reports +> healthy, `corpora verify` passes 23 checks against it, and its `node_modules` contains only +> `corpora/src/serve` and `corpora/src/node` — zero third-party code, which is only possible +> because the serving core has no dependencies. The standalone shell is the same Node adapter, +> covered by tests over real HTTP; its systemd unit and reverse-proxy snippets remain unexercised. +> +> **Both the container and standalone templates are nonetheless broken for anyone but us.** Their +> `package.json` files declare `"corpora": "^0.1.0"`, and Corpora is not published to npm. Testing +> the container required packing a tarball by hand. Publishing, or documenting a vendoring path, +> is a prerequisite for those two templates being usable. ### M4. `corpora verify` — done