graph LR
accTitle: System overview
App
App -->|"GET /xrpc/{method}"| Query
App -->|"POST /xrpc/{method}"| Procedure
subgraph HappyView
Query["Query handler<br/><small>Script (optional)</small>"]
Procedure["Procedure handler<br/><small>Script (optional)</small>"]
Ingest["Ingest pipeline<br/><small>Record scripts</small>"]
Backfill["Backfill worker"]
Spaces["Spaces<br/><small>Permissioned data</small>"]
end
Procedure --> DB
Query --> DB
Spaces --> DB
Ingest --> DB
Procedure -->|proxy write| PDS["User PDS"]
DB[("SQLite / PostgreSQL<br/><small>records · lexicons · spaces</small>")]
Jetstream["Jetstream<br/><small>WebSocket</small>"] -->|record events| Ingest
Jetstream -->|archive replay| Backfill
Relay["Relay<br/><small>listReposByCollection</small>"] -->|account discovery| Backfill
Backfill -->|listRecords| PDS
Backfill --> Ingest
Labeler["Labeler<br/><small>WebSocket (out-of-band)</small>"] -->|label events| DB
Queries go through the query handler to the database (SQLite by default, or Postgres). Writes go through the procedure handler to the user’s PDS, then HappyView indexes the record locally. Real-time record events stream in via Jetstream, on its v2 subscribeEvents stream when the host offers it and v1 otherwise. They are applied in order for each repo, and the stream position is saved only once every earlier event has been applied. Historical records are backfilled in-process, either by replaying the Jetstream archive or by listing accounts through the relay’s listReposByCollection and fetching records directly from each PDS. Every source hands its events to one ingest pipeline (src/ingest.rs), which runs the record script and writes the record. Labelers are external services that emit content labels over a direct WebSocket connection, outside the relay and repository system. Spaces are permissioned data containers: membership gates access, each user’s repo state is tracked with an LtHash and signed commits, and credentials authenticate callers across services.
sequenceDiagram
accTitle: Reads (queries)
participant C as App
participant X as xrpc_get()
participant R as LexiconCatalog
participant L as Script
participant D as Database
C->>X: GET /xrpc/{method}?params
X->>R: Look up<br/>(must be a query)
R-->>X: Lexicon
X->>X: Rate limit, validate parameters
alt Script bound to the query
X->>L: Run in its interpreter
L->>D: happyview.db / happyview.sql
D-->>L: Results
L-->>X: Response
else No script
X->>D: List target_collection,<br/>or get by uri
D-->>X: Results
end
X-->>C: JSON response
A procedure is authenticated, routed and validated first, and only then reaches the code that writes.
sequenceDiagram
accTitle: Writes, from the request to the procedure handler
participant C as App
participant A as XrpcClaims
participant X as xrpc_post()
participant R as LexiconCatalog
participant H as handle_procedure()
C->>A: POST /xrpc/{method}<br/>DPoP auth, X-Client-Key
A->>X: Validated claims
X->>X: 401 if there are none
X->>R: Look up<br/>(must be a procedure)
R-->>X: Lexicon
X->>X: Rate limit, validate<br/>parameters and input
X->>H: Claims, input, lexicon
H->>H: Resolve the<br/>script, if any
A write then goes to the caller’s PDS the same way with a script or without one, and handle_procedure() answers the app with the script’s return value or the PDS’s response.
sequenceDiagram
accTitle: Writes, from the procedure handler to the PDS
participant H as handle_procedure()
participant L as Script
participant S as PDS session
participant P as User PDS
participant D as Record store
alt Script bound to the procedure
H->>L: Run in its interpreter
L->>S: happyview.record<br/>create, put or delete
else No script
H->>S: The lexicon's action
end
S->>P: createRecord, putRecord<br/>or deleteRecord
P-->>S: PDS response
S->>D: Write or delete<br/>the local copy
Without a script, the lexicon’s action picks the write. The default, upsert, puts when the input has a uri field and creates otherwise. The PDS session is DPoP for an app’s user and OAuth for the dashboard.
sequenceDiagram
accTitle: Admin endpoints
participant C as Caller
participant A as UserAuth
participant U as Users table
participant H as Admin handler
participant D as Database
C->>A: Request with a cookie,<br/>API key or DPoP session
A->>A: Validate the credential
A->>U: DID lookup
alt Users table empty, no SUPERADMIN_DID
U-->>A: Auto-bootstrap with superadmin
else User found
U-->>A: Load permissions
end
A->>A: Permission check (403 if missing)
A->>H: Authorized request
H->>D: Database operation
D-->>H: Result
H-->>C: JSON response
A DPoP session also needs the dev.happyview.adminFull grant, and an API key skips the bootstrap. A service-auth JWT is refused with 403.
sequenceDiagram
accTitle: Real-time indexing
participant J as Jetstream WebSocket
participant H as HappyView
participant D as Database
participant R as LexiconCatalog
H->>J: Connect (collection filters from indexed lexicons)
loop Stream events
J->>H: Record commit event
H->>H: Run the record script, if any
alt create / update
H->>D: UPSERT into records table
else delete
H->>D: DELETE from records table
end
end
J->>H: Lexicon schema event (com.atproto.lexicon.schema)
H->>D: Update tracked network lexicons
H->>R: Update the in-memory catalog
Note over H,D: Applied watermark persisted to instance_settings for resume on reconnect
Note over H,J: Reconnects on collection filter changes (lexicon add/remove)
Ordering. Events are split across JETSTREAM_WORKERS workers by account DID, so each repo’s events apply in the order they arrived while different repos apply in parallel. Each worker runs one event at a time, record script included, so script latency bounds throughput: with slow scripts on Postgres, raise JETSTREAM_WORKERS. The saved cursor is the applied watermark: the position below which every event has been applied. After a restart or reconnect HappyView resumes from it, so delivery is at-least-once, and writes are idempotent (a replayed record carries the same rev and is ignored).
Accounts. An account event with active=false is handled by its status:
deleted, takendown and suspended purge the account’s records and leave an account tombstone (jetstream.account_deleted, jetstream.account_takendown). A relay can still be relaying a PDS that a deleted account has left. So before purging a deleted account it holds records for, HappyView asks the account’s current PDS (com.atproto.sync.getRepoStatus, 5 second timeout). If that PDS says the account is active, nothing is purged or tombstoned and jetstream.account_status_mismatch is logged. If the PDS cannot be asked, the purge goes ahead. takendown and suspended are purged without asking, because a relay may impose them and the account’s PDS cannot see that.
deactivated, desynchronized and throttled keep the records (jetstream.account_inactive).
active=true lifts the account’s tombstone. Records that were purged come back only through a new backfill.
A #sync removes the repo’s records in registered collections, and the current ones that follow replace them. identity events change nothing, because HappyView caches no identities.
Failures.
A database outage is retried indefinitely with growing delays, at most every 16 seconds by default, so ingest pauses rather than skipping events. While paused, the consumer cannot notice a collection change or a shutdown until the queue drains.
Any other failure is retried for about 31 seconds, then logged as jetstream.apply_failed and skipped.
Invalid input (no record body, an unknown operation, data the database rejects) is skipped at once with one jetstream.record_invalid warning.
Recovering a cursor.
A cursor older than Jetstream keeps (CursorTooOld) is replayed from the Jetstream archive when it has one and BACKFILL_SOURCE is auto or replay: see Gap replay under Backfill. Otherwise it resumes from the stored witnessed time. Jetstream clamps that to its oldest event and announces the gap with #info OutdatedCursor, which is logged as jetstream.gap with reason outdated_cursor. Only when there is no stored time does HappyView resume from the live tip, logged as cursor_too_old. A backfill recovers what was missed.
If Jetstream renumbers its seqs (FutureCursor), HappyView resumes by witnessed time too.
An unexplained 400 first drops compression, then the seq cursor, then retries hourly and logs jetstream.invalid_request. The seq cursor is dropped only when a seq was sent and a witnessed time is saved; otherwise the consumer goes straight to the hourly retry and keeps the cursor.
Under auto, a 404 falls back to v1 until the process restarts.
Edge cases of revs and tombstones.
A record script can run for a write that a tombstone then refuses.
An account tombstone’s rev is a TID of the time Jetstream witnessed the event. A PDS whose clock differs from Jetstream’s can therefore let an older replay through, or block a newer write until the tombstone is swept (RECORD_TOMBSTONE_RETENTION_HOURS, 72 by default).
A stale record tombstone left after a newer recreate blocks writes without a rev (crawl writes) to that record until the sweep.
A crawl row fetched before a live update but written after it can overwrite the newer body while keeping the stored rev. A later replay at that rev heals it.
A delete with a rev removes a crawled row that has none.
A delete for a record HappyView does not hold leaves a tombstone only when the account holds records in that collection, or a running or pausing backfill covers it. Those are the cases where a create could still arrive. The network sends many deletes for records an instance never held, and tombstoning each would fill the table.
Operators can watch the consumer with GET /admin/jetstream/status and the About page’s Jetstream card. See Jetstream status.
sequenceDiagram
accTitle: Backfill
participant A as Operator
participant H as HappyView
participant D as Database
participant Relay as Relay
participant PLC as PLC Directory
participant PDS as User PDS
A->>H: POST /admin/backfill
H->>D: Create backfill_jobs record (status = running)
rect rgb(240, 255, 240)
note over H,PDS: Discovery, resolution and fetching run concurrently
par Discovery (network backfills)
loop Each collection, while the queue has room
H->>Relay: listReposByCollection (one page)
Relay-->>H: DIDs
H->>D: Queue (did, collection) units, save the relay cursor
end
and Resolver task
loop Unresolved units
H->>PLC: Resolve DID document
PLC-->>H: PDS endpoint
H->>D: Record the unit's PDS
end
and Fetcher task
loop Resolved units (as they arrive)
H->>PDS: listRecords for the unit's collection (paginated)
PDS-->>H: Records
H->>D: UPSERT each page
H->>D: Delete the completed unit, bump counters
end
end
end
Note over H,D: At most BACKFILL_DISCOVERY_WINDOW units are queued per job
H->>D: Mark job completed (or failed)
Two sources. A backfill job reads its records from one of two places, chosen by BACKFILL_SOURCE (default crawl) or the job’s source. A crawl runs the pipeline above, and every account backfill crawls. A replay asks the Jetstream archive for a plan (planSnapshot), downloads the segments and blocks that hold the job’s collections, and writes the records through the same path as live events. A second sweep then applies the account and sync events for the accounts that hold records. Markers are applied as of their own time, so the two sweeps give the same result in either order and never remove records written after them. auto replays when the Jetstream has an archive and crawls when it doesn’t. A replay returns only what the archive holds; some accounts’ history in Bluesky’s archive starts after their first record, which a crawl would find.
Gap replay. When the live consumer’s cursor is older than Jetstream keeps, HappyView creates a replay job for the gap between its last applied seq and the archive’s sealed tip. It resumes the live stream at that tip. The consumer waits at most 30 seconds for the archive’s plan; past that it resumes without a replay. Live events and the replay overlap safely, because the rev guard and tombstones make every write idempotent. Instances on BACKFILL_SOURCE=crawl (the default), or on a Jetstream without an archive, skip the gap.
Tables are named happyview_<name>, and the headings below drop the prefix. These are the tables a contributor meets first, not all of them. The Type column says what a value holds, not its SQL type. Both backends store ids, timestamps and JSON as TEXT, because sqlx’s Any driver has no native timestamp or JSON type, and most flags as INTEGER 0 or 1. migrations/ holds each backend’s definitions.
happyview_records is a view, and it is what reads and raw SQL see. The rows live in happyview_records_v2, with DIDs and collection NSIDs interned in happyview_dids and happyview_collections. A record’s body is its JSON text exactly as received: record on Postgres, and body on SQLite, which a background task compresses with a per-collection zstd dictionary.
Column
Type
Description
uri
text
AT URI (at://did/collection/rkey), derived
did
text
Author DID
collection
text
Lexicon NSID
rkey
text
Record key
record
text (JSON)
Record value
cid
text
Content identifier
indexed_at
timestamp
When HappyView indexed this version of the record. Redelivering an identical record (same CID and body) keeps the original value. A local write that Jetstream will echo back (an XRPC procedure, a happyview.record mirror, a linked-repo write) stores NULL, on insert and on update, until the echo stamps it. A plugin’s direct local-index write gets no echo, so an update through it keeps the existing value
created_at
timestamp
When the row was first written
rev
text
TID of the repo commit that wrote this version, from Jetstream. NULL for local writes, listRecords backfill rows and rows written before revs were stored. A delivery with an older rev than the stored one is ignored
Recently deleted records (uri, rev, deleted_at) and deleted accounts (did, rev, seq, deleted_at). A create or update at or before a tombstone’s rev, or one without a rev, is not written, so a late or replayed copy cannot bring a deleted record back. Tombstones are kept for RECORD_TOMBSTONE_RETENTION_HOURS (72 by default).
pending; resolving_pds (an account backfill that has not started); resolving_and_fetching (a crawl at work); replaying, replaying_accounts and retrying_blocks (a replay); completed. A failed or cancelled job keeps the stage it reached. Jobs from older versions can also show discovering_repos or fetching_records
total_repos
integer
Work units discovered (a unit is one account under one collection; one per account for account-targeted jobs)
resolved_repos
integer
Units with PDS endpoint resolved
processed_repos
integer
Units with records fetched
total_records
integer
Total records indexed
error
text
Error message if failed
scope
text
network (accounts from the relay) or dids (accounts given at creation)
End-to-end tests use wiremock to mock external services (PLC directory, PDSes) and a real database for full integration coverage. A test that needs TEST_DATABASE_URL or a built plugin fixture fails without it, so a green run means it ran. Set HAPPYVIEW_ALLOW_SKIPS=1 to skip those tests instead.
