Skip to content
HappyView

Backfill

Create and monitor historical backfill jobs. See the Backfill guide for background.

const TOKEN = "hv_..."; // your API key
const headers = { Authorization: `Bearer ${TOKEN}` };

Create a backfill job

Section titled “Create a backfill job”
POST /admin/backfill

Requires backfill:create.

interface BackfillJob {
  id: string;
  status: string;
}


const response = await fetch("http://127.0.0.1:3000/admin/backfill", {
  method: "POST",
  headers: {
    ...headers,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    collection: "xyz.statusphere.status",
  }),
});
const data: BackfillJob = await response.json();
FieldTypeRequiredDescription
collectionstringnoLimit to a single collection (backfills all if omitted)
didsstring[]noAccounts to backfill, as DIDs or handles (discovers all via relay if omitted)
didstringnoA single account; kept for compatibility and added to dids
sourcestringnocrawl, replay or auto. Defaults to BACKFILL_SOURCE (crawl unless set). A job with dids always crawls, and replay with dids is refused. See Replay and crawl

Every entry in dids must resolve, or the request fails with 400 listing each entry that did not and no job is created. Duplicates are ignored. A request can name at most 500 accounts. Any other source fails with 400 and the message source must be one of auto, replay, crawl. source: "replay" with dids fails with 400 and the message replay is only available for collection backfills; account backfills always crawl. Account backfills crawl even when BACKFILL_SOURCE is replay or auto. A replay can only return what the archive holds: some accounts’ history in Bluesky’s archive starts later than their first record. See Replay and crawl.

Response: 201 Created

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "running"
}

A job left with no collection to backfill completes at once, with error set to no backfill-eligible collections. A job left with nothing because every collection it covers belongs to a record source’s own backfill completes with no error and a note instead. A job for a collection that has no indexed record lexicon fails with error set to no indexed record lexicon for collection '<nsid>'.

Cancel a backfill job

Section titled “Cancel a backfill job”
POST /admin/backfill/{id}/cancel

Requests cancellation of a running backfill job. Requires backfill:create. The job status transitions to cancelling immediately; the background worker stops at its next checkpoint and sets the final status to cancelled, with error set to Cancelled by an operator. A paused job is cancelled at once, and a job already cancelling or cancelled answers with its status. See the Backfill guide for details on the two-phase process.

const response = await fetch("http://127.0.0.1:3000/admin/backfill/550e8400-e29b-41d4-a716-446655440000/cancel", {
  method: "POST",
  headers,
});
const data = await response.json();

Response: 200 OK

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "cancelling"
}

Returns 400 for a job in any other state, or 404 if the job ID is not found.

Pause a backfill job

Section titled “Pause a backfill job”
POST /admin/backfill/{id}/pause

Asks a running backfill job to pause. Requires backfill:create. The job status transitions to pausing immediately; the background worker stops at its next checkpoint and set the status to paused. Paused jobs retain all progress and can be resumed later.

const response = await fetch("http://127.0.0.1:3000/admin/backfill/550e8400-e29b-41d4-a716-446655440000/pause", {
  method: "POST",
  headers,
});
const data = await response.json();

Response: 200 OK

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pausing"
}

A job already pausing or paused answers with its status. Returns 400 for a job in any other state, or 404 if the job ID is not found.

Resume a backfill job

Section titled “Resume a backfill job”
POST /admin/backfill/{id}/resume

Resume a paused backfill job, or a failed replay job. Requires backfill:create. The job status transitions back to running, its error is cleared, and processing continues from where it left off. A replay job carries on from the last plan entry it finished, so a job that failed on an archive outage, or after 16 failed blocks in a row, doesn’t start over. A failed crawl job can’t be resumed and returns 400, as does a job that another request resumed first. So does a failed replay job of specific accounts, created before account backfills always crawled, once its account list is gone: clearing the job’s details removes it. A replay job that failed because its error log is full (10,000 failures) fails again when resumed; start a new backfill instead.

const response = await fetch("http://127.0.0.1:3000/admin/backfill/550e8400-e29b-41d4-a716-446655440000/resume", {
  method: "POST",
  headers,
});
const data = await response.json();

Response: 200 OK

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "running"
}

Returns 400 if the job is neither paused nor a failed replay, or if it is a failed replay whose account list is gone, and 404 if the job ID is not found.

List backfill jobs

Section titled “List backfill jobs”
GET /admin/backfill/status

Requires backfill:read. Returns every job, newest first.

interface BackfillJob {
  id: string;
  collection: string | null;
  did: string | null;
  scope: "network" | "dids";
  // "auto", "replay", "crawl", or a record source's own name.
  source: string;
  replay: {
    after_seq: number;
    through_seq: number | null;
    tip_seq: number | null;
    bytes: number;
    blocks_failed: number;
  } | null;
  status: string;
  stage: string;
  total_repos: number | null;
  resolved_repos: number | null;
  processed_repos: number | null;
  total_records: number | null;
  error: string | null;
  started_at: string | null;
  completed_at: string | null;
  created_at: string;
  discovery_complete: boolean;
  recent_completions_limit: number | null;
  note: string | null;
  hooks: ScriptStats | null;
  hook_stats: ScriptStats | null;
}


interface ScriptStats {
  done: number;
  failed: number;
  timeouts: number;
  dead_lettered: number;
  skipped: number;
  outcomes: Record<string, number>;
  rate_per_s: number;
  avg_rate_per_s: number;
  eta_s: number | null;
  record_ms_p50: number | null;
  script_ms_p50: number | null;
  script_share: number | null;
  stages: Record<string, { n: number; ms_mean: number | null }>;
}


const response = await fetch("http://127.0.0.1:3000/admin/backfill/status", {
  headers,
});
const data: BackfillJob[] = await response.json();

Response: 200 OK

[
  {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "collection": "xyz.statusphere.status",
    "did": null,
    "scope": "network",
    "status": "completed",
    "stage": "completed",
    "total_repos": 42,
    "resolved_repos": 42,
    "processed_repos": 42,
    "total_records": 1000,
    "error": null,
    "started_at": "2025-01-01T00:01:00Z",
    "completed_at": "2025-01-01T00:05:00Z",
    "created_at": "2025-01-01T00:00:00Z",
    "discovery_complete": true,
    "recent_completions_limit": 1000,
    "source": "crawl",
    "replay": null,
    "note": null,
    "hooks": null,
    "hook_stats": null
  }
]

A replay job looks like this:

{
  "id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "collection": "xyz.statusphere.status",
  "did": null,
  "scope": "network",
  "source": "replay",
  "replay": {
    "after_seq": 0,
    "through_seq": 8412000,
    "tip_seq": 9120000,
    "bytes": 734003200,
    "blocks_failed": 0
  },
  "status": "running",
  "stage": "replaying",
  "total_repos": 0,
  "resolved_repos": null,
  "processed_repos": null,
  "total_records": 18250,
  "error": null,
  "started_at": "2025-01-01T00:01:00Z",
  "completed_at": null,
  "created_at": "2025-01-01T00:00:00Z",
  "discovery_complete": false,
  "recent_completions_limit": 1000,
  "note": null,
  "hooks": null,
  "hook_stats": null
}

source is the source the job was created with. A job created with dids is crawl, except a retry of a replay job’s failed blocks. auto stays until the job’s first request to the archive, then becomes replay or, for a Jetstream with no archive, crawl. replay is null for a crawl job and for a record source’s own job. For a replay job, after_seq is where it started: 0, or the last applied seq of a gap replay. through_seq and tip_seq are the running sweep’s position and the tip it is working towards. bytes is what it has downloaded from the archive, and blocks_failed counts blocks that failed after retries. through_seq and tip_seq are null until the first plan answers.

The status field tracks the overall job state (running, pausing, paused, cancelling, cancelled, completed, failed). The stage field tracks the current processing phase: pending (a new network job), resolving_pds (a new job for specific accounts), resolving_and_fetching, replaying, retrying_blocks, replaying_accounts or completed. A job that fails or is cancelled keeps the stage it was in. A record source’s own job reports the record source’s stages, and a job created by an older version may still show discovering_repos, fetching_records or failed. A replay job counts no accounts, so its *_repos counters stay 0 or null, and total_records counts the records it has written. The resolved_repos counter tracks PDS resolution progress during the pipelined phase, while processed_repos tracks record fetching progress.

The three *_repos counters count work units. A unit is one account under one collection: a network backfill discovers accounts per collection, so an account found under two collections is two units and counts twice. A unit for an account-targeted job covers every collection the job targets, so there one unit is one account. total_repos is the number discovered so far while a network backfill is still listing accounts from the relay.

scope is network when accounts are discovered through the relay and dids when the job targets specific accounts. did is set only when a job targets exactly one account.

discovery_complete is false while a network crawl is still listing accounts from the relay. Discovery runs alongside resolving and fetching. A network replay job lists no accounts and reports false throughout. recent_completions_limit is set for jobs on the bounded queue, which keep only that many completed accounts for the fetched list. It is null for jobs created before the bounded queue.

A job HappyView paused itself, because it ran out of work with accounts it could not complete still queued, has status paused and the reason in error (for example "paused: 3 units could not be completed; resume to retry"). A job an operator paused has no error. Resuming clears it.

A record source can backfill some collections itself rather than crawl or replay them. Such a job’s source is the name the record source gives its backfills, and its counters are what the record source reports. Its note may carry a sentence from the record source about what the job did or found. note is null for every other job.

Script stats

Section titled “Script stats”

hooks is what the job’s records have cost in record scripts so far, while this process holds it: from the job’s first record until an hour after its last, and null before and after. hook_stats is the same block kept with the job when it completed, failed or was cancelled. It’s null until then, and for a job none of whose records reached a script decision. Neither is recorded while metrics are switched off.

{
  "done": 6,
  "failed": 1,
  "timeouts": 1,
  "dead_lettered": 1,
  "skipped": 2,
  "outcomes": { "proceed": 4, "skip": 2 },
  "rate_per_s": 0.57,
  "avg_rate_per_s": 0.855,
  "eta_s": 0.0,
  "record_ms_p50": 171.074,
  "script_ms_p50": 0.082,
  "script_share": 0.00014,
  "stages": {
    "resolve": { "n": 6, "ms_mean": 2.36 },
    "hook": { "n": 6, "ms_mean": 1254.153 },
    "write": { "n": 4, "ms_mean": 0.504 }
  }
}
FieldMeaning
doneRecords that reached a script decision, whether or not a script was set for them
failedRecords whose script failed at least once, retries included
timeoutsRecords whose script ran out of time at least once
dead_letteredRecords whose script was given up on. The record was still stored
skippedRecords not stored: decisions of skip or cid_mismatch
outcomesEvery record by what was decided: proceed, replace, skip or cid_mismatch
rate_per_sRecords a second over about the last 30 seconds; the average until a second has passed
avg_rate_per_sRecords a second since the job’s first record
eta_sSeconds left, from the job’s progress against its total. null for a crawl or a replay, which report no total
record_ms_p50A record’s time: its lookup, CID check, script and its share of the write
script_ms_p50The script’s own time: its code and the host calls it made
script_shareScript time over record time
stagesEach stage’s records and mean, as on /admin/metrics/ingest. A record source may add its own stages. A stage whose name ends in _skipped is a count, and its ms_mean is null

Backfill state of a collection

Section titled “Backfill state of a collection”
GET /admin/backfill/collections/{nsid}

How a collection’s backfill stands, as the record source describes it. Requires backfill:read.

Response: 200 OK

{
  "collection": "xyz.statusphere.status",
  "executor": "store-pass",
  "summary": { "message": "…", "records": 1840, "state": "running" }
}

executor names what backfills this collection when a record source does it itself, and is null for a collection HappyView crawls or replays; summary is then null too. A record source that says nothing about the collection also answers null. In a summary, records and state are null when the record source doesn’t know them, and state is otherwise running, complete or failed.

List errors for a job

Section titled “List errors for a job”
GET /admin/backfill/{id}/errors

Paginated failure detail for a backfill job, with exact totals for each kind. Requires backfill:read.

ParamTypeRequiredDescription
kindstringnoOnly failures of this kind
cursorstringnoThe cursor from the previous page
limitnumbernoPage size, 1 to 100 (default 50)
const response = await fetch("http://127.0.0.1:3000/admin/backfill/550e8400-e29b-41d4-a716-446655440000/errors?kind=replay_block", {
  headers,
});
const data = await response.json();

Response: 200 OK

{
  "errors": [
    {
      "did": "seg_0000000001.jss",
      "collection": null,
      "phase": "replay_block:17",
      "kind": "replay_block",
      "message": "HTTP 502",
      "attempts": 1,
      "last_at": "2025-01-01T00:03:00Z"
    }
  ],
  "cursor": null,
  "counts": [
    { "kind": "replay_block", "count": 1, "retryable": true }
  ],
  "capped": false,
  "cap": 10000
}

For a crawl job, did is the account that failed and phase is the step (resolve or fetch). For a replay job, did is the archive segment’s name, collection is null, and phase is replay_block:<index> for one block, or replay_segment for a whole segment that failed before its blocks could be listed.

kind is one of dns_failure, connection_failed, timeout, pds_server_error, rate_limited, did_doc_not_found, did_doc_forbidden, did_doc_invalid, repo_not_found, repo_deactivated, repo_takendown, other or replay_block. counts lists each kind that has failures, with its total and whether retry-failed retries it. The kinds dns_failure, connection_failed, timeout, pds_server_error, rate_limited and replay_block are retryable.

HappyView keeps detail for the first cap failures of a job. When capped is true, counts is still exact but errors holds only the first cap.

Retry failed

Section titled “Retry failed”
POST /admin/backfill/{id}/retry-failed

Starts a new job for the failures of a finished job. Requires backfill:create.

const response = await fetch("http://127.0.0.1:3000/admin/backfill/550e8400-e29b-41d4-a716-446655440000/retry-failed", {
  method: "POST",
  headers: {
    ...headers,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    kinds: [
      "replay_block"
    ]
  }),
});
const data = await response.json();
FieldTypeRequiredDescription
kindsstring[]noOnly retry failures of these kinds. Defaults to every retryable kind

The request needs a JSON body even when it sets nothing: send {} to retry every retryable kind.

What the new job does depends on the source of the job it retries:

  • Crawl: a dids job for the accounts that failed, in the same collection. It starts at the resolve step, so the accounts’ PDS lookups run again.
  • Replay: a replay job that downloads exactly the failed blocks again. It runs in the retrying_blocks stage, then, in the replaying_accounts stage, applies the account and sync events of every account those blocks wrote records for. For a dids job it keeps the original accounts.

Response: 201 Created

{
  "id": "0b8f0f0e-3a3d-4f6b-9d56-6f6fd2d3c1a4"
}

Returns 400 when there is nothing to retry: no failures of a retryable kind, or kinds names none that the job has. A record source’s own job also answers 400, because its failures are the record source’s to retry. A replay job of specific accounts also answers 400 when its account list is gone, which happens after its details are cleared. Start a new backfill instead. Returns 404 if the job ID is not found.

List accounts for a job

Section titled “List accounts for a job”
GET /admin/backfill/{id}/repos

Paginated per-account detail for a backfill job. Requires backfill:read.

ParamTypeRequiredDescription
phasestringnofetched: recently completed accounts. resolved: queued accounts whose PDS is known. Anything else: every queued account
cursorstringnoOpaque cursor from a previous response with the same phase
limitnumbernoMax results per page (default 50, max 100)

A running backfill keeps at most BACKFILL_DISCOVERY_WINDOW accounts queued and deletes each one when it finishes. fetched therefore lists the job’s most recent completions, newest first, up to recent_completions_limit (1,000). The other phases list what is queued now, ordered by collection and then DID; queued accounts have status pending. An account queued under two collections appears once for each. resolved and the other queued phases share one cursor format. A fetched cursor sent to a queued phase, a queued-phase cursor sent to fetched, or a DID cursor kept from a job created before the bounded queue gets 400.

Jobs created before the bounded queue (recent_completions_limit is null) keep a row for every account and list them all in DID order. For them, fetched lists completed accounts, resolved accounts whose PDS is known, and anything else every account. Their cursor is the last DID.

const response = await fetch("http://127.0.0.1:3000/admin/backfill/550e8400-e29b-41d4-a716-446655440000/repos?phase=fetched&limit=10", {
  headers,
});
const data = await response.json();

Response: 200 OK

{
  "repos": [
    { "did": "did:plc:abc", "pds_endpoint": "https://pds.example.com", "status": "completed", "records_fetched": 42 }
  ],
  "cursor": "18342"
}

cursor is null when there are no more results.

PDS summary for a job

Section titled “PDS summary for a job”
GET /admin/backfill/{id}/pds-summary

Aggregated PDS breakdown for a backfill job. Requires backfill:read. It isn’t paginated: it returns all PDS endpoints in one response, sorted by account count descending. For jobs on the bounded queue the counts are kept per PDS as the job runs. There total_repos counts work units resolved to that PDS, like the other counters, so an account queued under two collections counts twice.

const response = await fetch("http://127.0.0.1:3000/admin/backfill/550e8400-e29b-41d4-a716-446655440000/pds-summary", {
  headers,
});
const data = await response.json();

Response: 200 OK

{
  "pds_endpoints": [
    { "pds_endpoint": "https://morel.us-east.host.bsky.network", "total_repos": 1200, "completed_repos": 800, "total_records": 5000 }
  ]
}

Stream backfill events (SSE)

Section titled “Stream backfill events (SSE)”
GET /admin/backfill/{id}/events

Server-Sent Events stream of real-time backfill progress. Requires backfill:read. The connection stays open until the client disconnects, including after the job has finished. A keepalive comment is sent periodically to prevent timeouts.

Events are sent with event: event and a JSON data payload. Each event has a type field:

Event typeDescription
repo_discoveredA new DID was found during the discovery phase
repo_resolvedA DID’s PDS endpoint was resolved
repo_fetchedRecord fetching completed for a DID
job_countersUpdated progress counters
job_stage_changedThe job moved to a new processing stage
job_completedThe job finished (completed, failed, or cancelled)
job_noteThe record source set the job’s note
replay_progressA replay’s through_seq, tip_seq, bytes and blocks_failed moved
hook_statsThe job’s script stats: at most once a second while records arrive, and once more as the job ends
job_snapshotThe job’s current status, stage, counters, error counts and discovery_complete, sent on connect and after the stream falls behind

Flush job details

Section titled “Flush job details”
DELETE /admin/backfill/{id}/details

Delete the per-account detail of one backfill job: its recent completions, its PDS summary, any accounts left queued and, for jobs created before the bounded queue, its rows for every account. Requires backfill:create.

A job on the bounded queue can be flushed only once it is completed, cancelled or failed, since its queue is the work it still has to do. Flushing one that is running, pausing, paused or cancelling returns 409. A job created before the bounded queue can be flushed in any state.

await fetch("http://127.0.0.1:3000/admin/backfill/550e8400-e29b-41d4-a716-446655440000/details", {
  method: "DELETE",
  headers,
});

Response: 204 No Content

Flush all job details

Section titled “Flush all job details”
DELETE /admin/backfill/details

Delete the per-account detail and leftover queue of every completed, cancelled and failed backfill job. Requires backfill:create.

await fetch("http://127.0.0.1:3000/admin/backfill/details", {
  method: "DELETE",
  headers,
});

Response: 204 No Content

© HappyProto