From c80c736a6aed3830f4d2f72f500f201bc4800de2 Mon Sep 17 00:00:00 2001 From: Evan Jarrett Date: Mon, 3 Aug 2026 17:40:17 -0500 Subject: [PATCH] docs: relays, and why this authority will never be on one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RELAYS.md answers the roadmap's third open question with a recommendation against: no subscribeRepos, no requestCrawl, keep the two sync endpoints already served. The decisive finding is that the question was never really about cost. A relay requires rev to increase between commits, and Corpora's rev is derived from the MST root precisely so that builds are reproducible — so roughly half of all publishes would carry a rev lower than their predecessor and be silently dropped by a conforming relay. Firehose participation and reproducible builds are mutually exclusive, and M1 chose reproducibility. That also collapses open question 4 into open question 3. The non-monotonic rev is safe only because there is no firehose, and the firehose is out of reach only because the rev is non-monotonic. They are one decision, now recorded as such. Two things worth knowing beyond the recommendation. There is no relay discovery protocol at all: atcr.io ships fifteen relay URLs as compile-time constants under a comment saying so, the reference PDS uses PDS_CRAWLERS, and the official installer writes exactly one hostname. And a relay never enumerates a host's repositories — it learns accounts only from firehose events — so "requestCrawl then getRepo for backfill" is not a middle path, it is nothing at all. The loss is real but bounded: a public relay's lexicon-schema index holds 491 repos and atcr.io is among them. The single consumer found, lexicon.store, accepts a deploy-time POST that does the same job, so PUBLISHING now ends the loop with it — alongside recording didDocHash out of band, which had been documented as a control and never given a step. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 1 + docs/PUBLISHING.md | 25 +++ docs/RELAYS.md | 395 +++++++++++++++++++++++++++++++++++++++++++++ docs/ROADMAP.md | 29 ++-- 4 files changed, 441 insertions(+), 9 deletions(-) create mode 100644 docs/RELAYS.md diff --git a/README.md b/README.md index bb162c0..59aff17 100644 --- a/README.md +++ b/README.md @@ -320,6 +320,7 @@ depending on whether they were checked against live infrastructure. | You are handling the signing key, or moving off `goat lex publish` | [PUBLISHING](docs/PUBLISHING.md) | | The build rejected something and you want to know why | [BUILD](docs/BUILD.md) | | A response differs from what a real PDS returns | [SERVING](docs/SERVING.md) | +| You are wondering why this never shows up on a relay | [RELAYS](docs/RELAYS.md) | | You want the design rationale, DID method, or key custody | [ARCHITECTURE](docs/ARCHITECTURE.md) | | You are moving an existing namespace off a PDS | [ROADMAP](docs/ROADMAP.md) | | You are wondering what already exists in this space | [PRIOR_ART](docs/PRIOR_ART.md) | diff --git a/docs/PUBLISHING.md b/docs/PUBLISHING.md index a7ae0ab..62a35fd 100644 --- a/docs/PUBLISHING.md +++ b/docs/PUBLISHING.md @@ -62,10 +62,35 @@ state of the namespace better than manual inspection: `goat lex check-dns` and ` 4. Verify: run the official resolver against the built artifacts before deploying. 5. Deploy the shell (`wrangler deploy`, or push a container image). 6. Verify again against the live URL. +7. Record `manifest.json`'s `didDocHash` and `mstRoot` somewhere that is not the serving host. +8. Optionally, tell the indexer (below). Steps 4 and 6 are not optional ceremony. A broken authority does not fail loudly on your side; it fails as `invalid_scope` on a stranger's login, hours later, on a cold cache. +Step 7 takes a second and is the whole of `did:web`'s tamper-evidence story. `did:web` has no +operation log, so a fingerprint published away from the host is what lets anyone — including you, +later — detect that the served document changed. `corpora verify --expect-did-doc ` is the +other half. See [ARCHITECTURE](ARCHITECTURE.md#did-method). + +### Telling the indexer + +A Corpora authority never appears on a relay, because relays learn about repositories from a +firehose and a static authority does not have one. [RELAYS](RELAYS.md) explains why that is +architectural rather than an omission, and why it costs less than it sounds like. + +The one known consumer that reads lexicon schemas off the firehose, lexicon.store, also accepts a +direct submission, which does the same job without any of the machinery: + +```console +$ curl -X POST https://api.lexicon.store/requestIndex \ + -H 'content-type: application/json' \ + -d '{"did":"did:web:lexicon.example.com"}' +``` + +Optional, and only worth doing if you want your schemas discoverable by people who do not already +know the NSID. Resolution does not depend on it in any way. + ## Key handling The signing key is a secp256k1 private key. It is an input to the build and nothing else. diff --git a/docs/RELAYS.md b/docs/RELAYS.md new file mode 100644 index 0000000..5a7f9ce --- /dev/null +++ b/docs/RELAYS.md @@ -0,0 +1,395 @@ +# Relays + +Whether a static lexicon authority can be relay-indexed, what it would cost, and what it would +buy. The short answer is that it can be indexed, that the cost is a live process holding sockets +plus a monotonic revision, and that the one real consumer already offers an HTTP endpoint that +does the same job in one POST. + +Facts marked **verified** were checked here on 2026-08-03, either against live infrastructure or +by reading the source named. Facts marked **reported** come from credible source or documentation +that was read but not exercised. Where a claim could have been tested live and was not, it says so. + +## 1. What a relay actually requires + +Traced through `bluesky-social/indigo`, `cmd/relay/`, read at `main` on 2026-08-03. Every step +below names the file and line where it was verified. + +**Step 1 — `com.atproto.sync.requestCrawl`.** A `POST`, no auth, one field: `{"hostname": "..."}`. +The lexicon describes its purpose as "new PDS instances declaring their existence to Relays" +(**verified**, `lexicons/com/atproto/sync/requestCrawl.json`). Handled at +`cmd/relay/handlers.go:19`. + +**Step 2 — the relay probes the host with `com.atproto.server.describeServer`.** Before anything +is written to its database, `handlers.go:57` calls `HostChecker.CheckHost`, which is exactly one +request: + +```go +func (hc *HostClient) CheckHost(ctx context.Context, host string) error { + client := hc.apiClient(host) + _, err := comatproto.ServerDescribeServer(ctx, client) + if err != nil { + return fmt.Errorf("%w: %w", ErrHostNotPDS, err) + } + return nil +} +``` + +(`cmd/relay/relay/host_checker.go:57`). A failure returns `400 HostNotFound`, "host server +unreachable", and the crawl request is discarded. + +**Corpora fails here, today, before any firehose question arises.** The live proof-of-concept at +`lexicon-poc.jarrett.app` answers `404 MethodNotImplemented` on +`/xrpc/com.atproto.server.describeServer` (**verified**), where the reference PDS at +`jarrett.app` answers `200` with a small JSON body (**verified**). So the first thing Corpora +would need is not a firehose at all but a seven-field static document — trivial, and worth +knowing, because it means `requestCrawl` cannot even be *attempted* against a Corpora authority +as built. Confirming the rejection live requires sending a `POST` to a public relay; that was not +done here, so the rejection itself is **reported** from source. It is one command to check: + +```console +$ curl -X POST https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.requestCrawl \ + -H 'content-type: application/json' -d '{"hostname":"lexicon-poc.jarrett.app"}' +``` + +**Step 3 — a host row, with limits.** `SubscribeToHost` (`cmd/relay/relay/crawl.go:11`) rejects +banned domains, enforces a per-day limiter on *new* hosts (`ErrNewHostsDisabled`), assigns an +account limit that is higher for configured trusted domains, and then subscribes. + +**Step 4 — the relay dials `wss:///xrpc/com.atproto.sync.subscribeRepos`.** With +`?cursor=` if it has seen the host before (`slurper.go:320-323`). Frames are two +concatenated DRISL-CBOR objects, a header and a payload (**verified**, `atproto.com/specs/event-stream`). + +**Step 5 — there is no backfill from `listRepos`.** This is the step people assume exists. The +relay learns which repos a host holds *only* from firehose events: accounts are created in +`preProcessEvent` (`ingest.go:63-73`), which is reachable only from the `#commit`, `#sync`, +`#identity` and `#account` handlers. Grepping `cmd/relay/` for calls to a host's `listRepos` +returns nothing; the relay *serves* `listRepos` and `getRepo` (the latter as a 3xx redirect to +the source PDS, `service.go:154`) but never calls them upstream. Bluesky's own relay notes say +the same thing from the operator side: "If hosts don't show up in listHosts, try making some repo +updates to emit new `#commit` messages" (**verified**, `docs.bsky.app/blog/relay-sync-updates`). + +**A host with no firehose is therefore a host with no repos, from the relay's point of view.** +`requestCrawl` plus a perfectly good `getRepo` buys nothing on its own. + +**Step 6 — per event, the relay does real work.** It resolves the DID (`ingest.go:80`); it +requires the resolved `#atproto_pds` endpoint's hostname to equal the crawled hostname, or fails +with "new account from a different host" (`account.go:127-170`); it checks account status, calling +the host's `com.atproto.sync.getRepoStatus` when the cached status is not active +(`host_checker.go:67`); and it verifies the commit signature, TID syntax, `prevData`, `since`, and +MST inversion (`verify.go`). + +**Step 7 — what happens when the firehose stops.** Dialling backs off; after 15 consecutive +failures the host is marked `offline` and the goroutine returns (`slurper.go:332-338`). On relay +restart only hosts with `status = "active"` are resubscribed (`crawl.go:62`), so an `offline` or +`idle` host stays disconnected until somebody sends a fresh `requestCrawl`. Nothing is deleted: +the accounts, revs and collection index the relay already built remain queryable. + +That last sentence is not theory. **Verified live**: `stream.place` — the production `did:web` +lexicon authority described in [PRIOR_ART](PRIOR_ART.md) — is on relay1.us-west with +`{"accountCount":1,"hostname":"stream.place","seq":4,"status":"offline"}`, and does not complete a +WebSocket upgrade on `subscribeRepos`. Its firehose delivered four events and stopped. Yet the +relay still answers `getRepoStatus` for `did:web:stream.place` with `active: true`, and +`did:web:stream.place` still appears in the relay's `listReposByCollection` index for +`com.atproto.lexicon.schema`. + +The relay's copy is simply frozen: it reports `rev: 3lx65qa7bb622` while the host's own +`listRepos` reports `rev: 3ms4lslqaca22` (**verified**, both fetched today). Nobody noticed, +because nothing in lexicon resolution reads either value. + +**Offline is the common case, not the exception.** The relay's full host list, paginated today, +is 5,777 hosts: 1,826 `active`, 3,668 `offline`, 251 `idle`, 32 `banned` (**verified**). Roughly +two in three hosts a public relay knows about are not currently streaming to it. + +## 2. How relays are discovered + +**They are not. Every PDS ships a hardcoded list.** The claim was correct. + +The clearest statement of it is in this project's own neighbour. `/home/data/atcr.io`, +`pkg/atproto/relays.go:20-38` (**verified in source**): + +```go +// KnownRelays is the hard-coded list of known public ATProto relays. +// There is no relay discovery protocol in ATProto — this list is manually maintained. +var KnownRelays = []KnownRelay{ + {Name: "Bluesky", URL: "https://bsky.network"}, + {Name: "Bluesky US-East", URL: "https://relay1.us-east.bsky.network"}, + {Name: "Bluesky US-West", URL: "https://relay1.us-west.bsky.network"}, + {Name: "Firehose NA", URL: "https://northamerica.firehose.network"}, + ... + {Name: "Blacksky", URL: "https://atproto.africa"}, + {Name: "Bassh", URL: "https://relay.bas.sh"}, +} +``` + +Fifteen entries, compile-time constants with no override — configuration can only *add* to them, +via `server.relay_endpoints`. `hold` fans `requestCrawl` out to the union of the two, once, in a +goroutine at process start (`pkg/hold/server.go:395-403`, `:444-483`): no retry, no periodic +re-announce, and no re-crawl when the repo changes. The project also carries +`docs/KNOWN_RELAYS.md`, a hand-maintained capability matrix for those fifteen, last verified +2026-02-08 and recording that one of them was returning 502. + +The reference implementation is the same pattern with the list moved into an environment +variable. `packages/pds/src/config/env.ts` reads `crawlers: envList('PDS_CRAWLERS')` with no +default (**verified**), `packages/pds/src/crawlers.ts` posts `requestCrawl` to each with a 20 +minute throttle (**verified**), and the official self-host installer writes exactly one hostname +into the generated environment file: `PDS_CRAWLERS="https://bsky.network"` (**verified**, +`bluesky-social/pds`, `installer.sh`). + +cirrus does it a third way and less well: its Worker never calls `requestCrawl` at all, and the +hostnames live in the CLI — `RELAY_URLS` of two Bluesky relays for status polling, and a separate +default argument `relayUrl = "https://bsky.network"` for the crawl itself, with no environment +override anywhere (**verified in source**, `packages/pds/src/cli/utils/pds-client.ts:1082` and +`:1141`). + +There is no registry, no `.well-known`, no DNS convention, and no discovery endpoint. The nearest +thing is relay-to-relay: indigo ships a `pull` command that reads another relay's `listHosts` and +adopts the hosts it finds (`cmd/relay/pull.go:100-160`) — which discovers hosts, not relays, and +only hosts that some other relay already had. + +So the honest description of "getting on the relays" is: *maintain a list of other people's +hostnames in your source tree, and hope it is current.* + +## 3. What Corpora would have to become + +The tension is not that a firehose is hard. It is that a firehose is a *property of a running +server*, and Corpora deliberately has none. Restating +[ARCHITECTURE](ARCHITECTURE.md#the-decision-everything-follows-from): signing happens at publish +time, serving is static bytes, and the artifact is replicated to many locations with no origin. + +Three things break against that. + +**The change happens when nothing is alive.** A firehose event describes a mutation. Corpora's +mutation is `corpora build` running on a laptop or in CI, which then exits; the deploy that +follows replaces bytes. There is no process present at the moment of change to emit anything. The +best a static artifact can do is carry a pre-baked event that it replays to whoever connects. + +**There is no single sequence — except that there is, and this is the interesting part.** The +usual objection is that a globally replicated artifact cannot maintain a monotonic counter. True +for a counter maintained at runtime; false for one *derived at build time*. `seq` could be the +build's Unix timestamp, embedded in `data.js`. Every replica then emits an identical event with +an identical `seq`, deterministically, and consistency across locations is free precisely because +the artifact is byte-identical everywhere. Cursor handling is a comparison: if +`cursor < embeddedSeq`, send the one event; otherwise send nothing; either way hold the socket +open. That is a genuinely stateless firehose, and it would be about forty lines. + +**The revision is the thing that actually disqualifies it.** See §6. + +Per shell, assuming the stateless design above: + +| Shell | What it costs | +| --- | --- | +| Worker | A Durable Object. Workers cannot hold a WebSocket outside one, and a DO is a single instance in a single region — cirrus's is literally `env.ACCOUNT.idFromName("account")` with an optional `locationHint`, and its own architecture notes say "there is exactly one running at a time, in one region" (**verified in source**). That reintroduces an origin, a binding, and mutable state: the three things [ARCHITECTURE](ARCHITECTURE.md) removes. The bundle stops being 20 KB gzipped with no bindings. | +| Container | A long-lived process and a socket count. Workable, but replicas behind a load balancer each terminate their own sockets; that is fine under the deterministic-`seq` design and not fine under any design where `seq` is generated at runtime. Health checks and restarts become firehose-visible. | +| Standalone | The least bad. It is already a live `node:http` process; adding `ws` makes it a server that holds connections. At which point the honest description is that you are running a small PDS. | + +And in every shell, the serving tier acquires its first dependency. `src/serve` imports nothing, +and [SERVING](SERVING.md#what-the-serving-core-may-import) has a test that enforces it. Workers +and Node have native WebSocket *server* primitives, so this is survivable — but frame encoding is +DRISL-CBOR, and Corpora has no encoder outside `src/build`. Either the build pre-encodes the +frames into `data.js`, which is the right answer and keeps the rule intact, or the rule breaks. + +## 4. The middle paths, evaluated + +Four were considered. Two are dead on inspection, one is technically sound and pointless, one is +what should actually be done. + +**(a) A separate publisher component that emits one event per deploy — dead.** The relay resolves +the repo's DID and requires the `#atproto_pds` endpoint's hostname to equal the hostname it +crawled, failing otherwise with "new account from a different host" +(**verified in source**, `cmd/relay/relay/account.go:127-170`, with `CreateAccountHost`'s own +comment: "We only create accounts associated with hosts we already know of, not remote hosts (aka, +no spidering)"). A publisher at `publish.lexicon.atcr.io` cannot emit events for a repo whose +service endpoint is `lexicon.atcr.io`. Making it work means pointing the DID document at the +publisher, which is where resolvers fetch `sync.getRecord` — so the publisher becomes the serving +tier and the separation was imaginary. + +**(b) A long-lived tiny notifier that is not the serving tier — dead, same reason.** It is (a) +with a smaller binary. + +**(c) `#sync` instead of a full commit stream — sound, and it is the right event.** `#sync` +"asserts the current status of an account's repository", carries only the signed commit block +(`maxLength: 10000` against `#commit`'s 2 MB CAR slice), and exists exactly for the case where +"upstream host does not know recent state of the repository" (**verified**, +`lexicons/com/atproto/sync/subscribeRepos.json`). Corpora's whole world is one repo whose current +state is known at build time, so a single embedded `#sync` frame plus an `#account` frame is a +complete and honest description of it. No `prevData`, no `since`, no MST inversion, no diff. + +It also happens to slip past the relay's revision check *today*: indigo's `processSyncEvent` has +no rev comparison, only a comment asking whether it should — `// TODO: should we load account +'rev' here and prevent roll-backs? or allow roll-backs?` (**verified in source**, +`cmd/relay/relay/ingest.go:169`). Building on that would be building on an open question in +somebody else's TODO, against a specification that says the opposite. See §6. + +**(d) `requestCrawl` at deploy time plus `getRepo` for one-shot backfill — does not work as +described, and something better does.** It does not work because the relay never enumerates a +host's repos (§1, step 5); `requestCrawl` without a firehose produces a host row, a dial, fifteen +retries and an `offline` marking. But the *goal* behind it — tell the things that care, at deploy +time, without holding a socket — is achievable directly, and §5 is where that lands. + +## 5. What it buys + +Skeptically, and with one genuine surprise. + +**Nothing on the resolution path.** [RESOLUTION](RESOLUTION.md) establishes this and it has been +verified end to end: DNS TXT, DID document, one `com.atproto.sync.getRecord`, done. No relay is +consulted. An Auth Server resolving `include:` scopes never touches one. + +**But lexicon repos on relays are not hypothetical, and this is bigger than expected.** Querying +relay1.us-west today (**verified**): + +```console +$ curl "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.listReposByCollection\ +?collection=com.atproto.lexicon.schema&limit=1000" +``` + +Paginated to exhaustion that is **491 repos publishing `com.atproto.lexicon.schema`**, of which +23 are `did:web` — including `did:web:stream.place`, `did:web:lexicon.store`, +`did:web:atproto-nix.org`, `did:web:firehose.chat` and `did:web:alt.science`. **atcr.io's own +`did:plc:wfj5kyialpmcv2fzk6uqwsln` is in the list**, confirming the report that the current PDS is +relay-indexed. `com.atproto.sync.listReposByCollection` is a relay endpoint, and its index is built +entirely from firehose events, so this is a real capability that a cutover to a static authority +gives up. + +**So who consumes it?** One service, found and confirmed live: **lexicon.store**, "The AT Protocol +Schema Registry". It is itself `did:web:lexicon.store`, it re-publishes every schema it verifies +into its own `com.atproto.lexicon.schema` collection, and it holds atcr.io's records already — +`io.atcr.authFullApp` at CID `bafyreic4alq6enwhdvcy5smx3rvq6utemlu2rlae3c5sgs6wbdppr2od3q`, which +is byte-for-byte the CID [RESOLUTION](RESOLUTION.md) records for the live record (**verified**). +Its own documentation says: "Once published to your PDS, lexicons are automatically discovered and +indexed via the AT Protocol Firehose. DNS authority verification is performed to ensure only +authorized publishers can register schemas for a namespace." Its live monitor connects to +`wss://nyc.firehose.stream/tap`, a Jetstream, which is fed from a relay (**verified**). + +**And it has a documented HTTP endpoint that does the same job with no firehose at all** +(**verified**, `api.lexicon.store`): + +```console +$ curl -X POST https://api.lexicon.store/requestIndex \ + -H 'content-type: application/json' \ + -d '{"type": "did", "value": "did:web:lexicon.atcr.io"}' +``` + +`type` accepts `nsid`, `did` or `domain`; the docs' own example for `did` is +`did:web:stream.place`. Indexing is gated on DNS authority verification, which a Corpora authority +passes by construction, since the build refuses to publish without the `_lexicon` TXT records. + +That is the whole payoff of the firehose, for this use case, reduced to one `curl` in the deploy +script. And the proof that it is sufficient is sitting in the registry: `place.stream.authFull` is +indexed at lexicon.store while `stream.place`'s relay host status is `offline` and its firehose +delivered four events, total (**verified**). A one-shot notification, or four events years ago, +is apparently enough — because a lexicon repo changes a few times a year and the index is durable. + +No other consumer was found. `lexidex.bsky.dev` appears in search results as a lexicon browser but +did not resolve when fetched. `pdsls.dev`, `goat`, `@atproto/lexicon-resolver` and the reference +PDS all resolve through DNS. `com.atproto.lexicon.resolveLexicon` exists in the lexicons +(**verified**) and is an NSID-to-schema query, not a firehose consumer. If there is a search tool +or index that requires a live firehose from a lexicon authority specifically, it was not found. + +## 6. The bug surface + +The instinct is right, and one item on the list is not merely a bug risk but disqualifying. + +**Non-monotonic `rev` is disqualifying.** [BUILD](BUILD.md#reproducibility-and-the-rev) explains +that Corpora derives the `rev` from the MST root so that identical inputs produce byte-identical +output. Concretely (`src/build/repo.js:117-130`), it is a fixed anchor of 2020-01-01 plus +`sha256(mstRoot)` reduced modulo `2**40` microseconds — a uniform draw from a window of about 12.7 +days. Two successive publishes therefore produce two independent uniform draws, so **each publish +has a fifty percent chance of a `rev` lower than the one before it.** + +The specification is unambiguous about what that means. "The revision must always increase between +commits for the same repository, even if the account migrates between hosts or has an extended +period of inactivity", and consumers "should ignore `#commit` and `#sync` events with a `rev` lower +or equal to the most recent successfully processed `rev` for that DID" (**verified**, +`atproto.com/specs/sync`). indigo enforces it on commits in two places — a fast check that logs +"dropping commit with old rev" (`ingest.go:114-120`) and a hard `ErrRevSequence` in +`VerifyCommitMessageStrict` (`verify.go:147`) — and does not yet enforce it on `#sync`, per the +TODO quoted in §4. + +So a Corpora firehose would be silently ignored by a conforming relay on half its publishes, in a +way that produces no error anywhere: the deploy succeeds, resolution keeps working, and the relay's +copy just stops advancing. The only fix is `--rev` with a wall-clock TID, which trades away the M1 +acceptance criterion the whole build was designed around. **Firehose participation and reproducible +builds are mutually exclusive.** That is a design-level conflict, not an implementation detail. + +(The complementary check is harmless: `verify.go:98` rejects revs more than five minutes in the +*future*, and Corpora's are six years in the past.) + +The rest of the surface, for completeness: + +- **Sequence numbers.** "Services should ensure that sequence numbers are not re-used, usually by + committing events (with sequence number) to robust persistent storage before transmitting them" + (**verified**, `atproto.com/specs/event-stream`). A rollback is expensive: a client reconnecting + with a cursor above the server's current position gets a `FutureCursor` error, and indigo + responds by marking the host `idle` and dropping the connection (`slurper.go:441-448`) — after + which, per §1, it will not resubscribe on restart. Both reference implementations have a live + version of this failure: `hold`'s `seq` is an in-process counter whose high-water mark is + recovered from SQLite at boot, and which restarts at 0 if the database fails to open + ("Events will not persist across restarts", `pkg/hold/pds/events.go:119-121`); cirrus's is a + SQLite `AUTOINCREMENT` inside its one Durable Object. +- **Cursor and backfill correctness.** cirrus's backfill is a single `LIMIT 1000` query with no + pagination loop, so a subscriber more than a thousand events behind silently loses the gap + (**verified in source**). `hold` handles a cursor ahead of its sequence by logging and sending + nothing, rather than the `FutureCursor` error frame the spec calls for — the relay just sees + silence (**verified in source**). These are careful projects and both got it subtly wrong. +- **Retention.** Neither reference implementation bounds its event log. cirrus documents a + 10,000-event window and ships the pruning function as dead code, called only from a test + (**verified in source**); `hold`'s `firehose_events` table stores a CAR blob per commit with no + deletion path anywhere in the tree. For a lexicon authority the log is tiny, so this is cheap + here — but it is a good indicator of how much of this machinery is easy to get wrong and hard to + notice. +- **Reconnection and liveness.** `hold` had to add a `readPump` purely so gorilla would answer + pings, because relays were timing the connection out otherwise (`pkg/hold/pds/events.go:474-476`). +- **What a relay does with a repo that regresses.** Per the spec, a mismatched `since`/`prevData` + means "the service will need to re-synchronize the repository. This often means fetching the full + repo CAR export" (**verified**). Every Corpora build is a fresh genesis commit with `prev: null` + and `since: null`, so *every* publish after the first looks like exactly that break. + +**Scale, for calibration.** The firehose in cirrus is ~784 lines across four files, carrying ~2,557 +lines of test and verification tooling — a 3.3:1 ratio, notably higher than the rest of that PDS. +In `hold` it is ~1,660 lines: a 1,027-line `events.go`, 292 lines of relay client and hardcoded +list, 202 lines of admin UI for it, plus the emission calls threaded through the mutation methods +(**verified in source**, both). Corpora's entire serving core is 328 lines with zero dependencies, +and the deployed Worker is 20,372 bytes gzipped. + +## 7. Recommendation + +**Do not implement `subscribeRepos`. Do not implement `requestCrawl`. Keep `listRepos` and +`getRepoStatus`, which are already served. Add `com.atproto.server.describeServer` only if +somebody asks.** + +The reasoning, in order of weight: + +1. **The `rev` conflict is decisive.** A content-derived `rev` is not monotonic and the sync + specification requires that it be. Half of all publishes would be dropped by a conforming + relay with no error surfaced anywhere. The only escape trades away reproducible builds, which + is a worse deal than losing relay indexing. +2. **The payoff is one consumer with an HTTP endpoint.** The single confirmed consumer of + `com.atproto.lexicon.schema` records off the firehose is lexicon.store, and it accepts a + deploy-time `POST /requestIndex` that does the same job, gated on the same DNS authority the + build already enforces. One line in a deploy script against a live process, a Durable Object or + a socket-holding server, an event log, cursor handling and a sequencing scheme. +3. **A firehose contradicts the architecture rather than extending it.** Every shell would need + either an origin instance or persistent state, and the Worker — the shell the first real + deployment uses — would need a Durable Object, which is one instance in one region. The design + exists to have neither. +4. **The loss is real but small, and it degrades gracefully.** Migrating atcr.io off its PDS does + drop it out of the relay's `listReposByCollection` index over time, and that index has 491 + entries, so it is a live thing and not a curiosity. But `stream.place` demonstrates the failure + mode directly: a lexicon authority whose firehose has been down long enough for the relay to + mark it `offline` is still indexed, still enumerable by collection, and still resolves perfectly + — because resolution never asked the relay anything. + +Which endpoints, and why: + +| Endpoint | Verdict | +| --- | --- | +| `com.atproto.sync.listRepos` | **Keep.** Already served: one repo, no cursor, fully determined at build time. Costs nothing and mirrors read it. | +| `com.atproto.sync.getRepoStatus` | **Keep.** Already served. A relay calls it on a host when re-checking account status, and it is a three-field constant. | +| `com.atproto.sync.requestCrawl` | **Exclude.** It is a client call, not a served endpoint, and making it would announce a host that cannot then be followed — fifteen dial retries and an `offline` marking. Revisit only alongside a firehose, never before one. | +| `com.atproto.sync.subscribeRepos` | **Exclude.** `MethodNotImplemented`, as now. | +| `com.atproto.server.describeServer` | **Optional, and cheap.** Seven static fields. It is the relay's gate check and it is the only reason `requestCrawl` fails at step 2 rather than step 4. Worth adding if a mirror or a tool wants it; not worth adding as a step towards relaying, because the steps after it are the problem. | + +If the position ever needs revisiting, the trigger is narrow and easy to state: **a consumer +appears that reads lexicon schemas from the firehose and offers no direct submission path.** Until +then, the deploy script should end with a `curl` to `api.lexicon.store/requestIndex` and nothing +else. See [PUBLISHING](PUBLISHING.md). diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 5f9c346..80dfe03 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -226,15 +226,26 @@ Run `goat lex check-dns` and `goat lex status` to confirm this independently bef several DIDs per deployment, or stay one-authority-per-deployment? Start with one. 2. **Publishing trigger.** Manual, or CI on lexicon changes? See [PUBLISHING](PUBLISHING.md#automation). -3. **Does anything expect lexicon repos on relays?** A static authority cannot serve a firehose. - Probably irrelevant, worth confirming. -4. **Is a non-monotonic `rev` acceptable?** Reproducibility and monotonicity are in direct - conflict: a `rev` is a clock-based TID, so a build that is byte-identical for identical inputs - cannot also produce an increasing `rev` across publishes. M1's acceptance criterion asked for - reproducibility, so that is the default, and `--rev` gives a monotonic one where it matters. - Nothing on the resolution path reads `rev` — every build is a genesis commit and there is no - firehose — so this is believed safe, but it is a deliberate departure from repo convention and - deserves a second look before a public deployment. See [BUILD](BUILD.md). +3. **Does anything expect lexicon repos on relays?** **Answered: yes, one thing, and it takes a + `POST` instead.** Confirmed 2026-08-03: a public relay's `listReposByCollection` index holds 491 + repos publishing `com.atproto.lexicon.schema`, atcr.io's DID among them, so relay indexing is + real and the cutover does give it up. The only consumer found is lexicon.store, which auto-indexes + from the firehose *and* accepts a deploy-time `POST /requestIndex` gated on the same DNS authority + the build already enforces. Against that, a firehose is disqualified rather than merely expensive: + the content-derived `rev` is not monotonic, the sync spec requires that it be, and a conforming + relay would silently drop half of all publishes. **Decision: no firehose, no `requestCrawl`.** See + [RELAYS](RELAYS.md). +4. **Is a non-monotonic `rev` acceptable?** **Answered, by question 3.** Reproducibility and + monotonicity are in direct conflict: a `rev` is a clock-based TID, so a build that is + byte-identical for identical inputs cannot also produce an increasing `rev` across publishes. + M1's acceptance criterion asked for reproducibility, so that is the default, and `--rev` gives a + monotonic one where it matters. + + The two questions turn out to be one. A non-monotonic `rev` is only safe *because* there is no + firehose — the sync specification's monotonicity rule exists for consumers following a stream, + and nothing on the resolution path reads `rev` at all. Equally, the firehose is out of reach + *because* the `rev` is non-monotonic. Reversing either decision forces the other, so treat them + as a pair rather than two independent knobs. See [BUILD](BUILD.md) and [RELAYS](RELAYS.md). ### Answered during implementation -- knotmirror 0.0.1