Meilisearch sync

Push records to a Meilisearch search index whenever they are created, updated, or deleted on the network.

Trigger: record.index:<nsid>. Needs: happyview-http. Script variables: MEILISEARCH_URL, MEILISEARCH_API_KEY.

Lua

local json = require("internal.json")
local http = require("happyview.http")

function handle(input, ctx)
  -- Meilisearch ids may contain only letters, digits, `-` and `_`
  local id = (input.did .. "_" .. input.collection .. "_" .. input.rkey):gsub("[^%w%-_]", "_")

  local headers = {
    ["Authorization"] = "Bearer " .. ctx.env.MEILISEARCH_API_KEY,
    ["Content-Type"] = "application/json",
  }

  local res
  if input.action == "delete" then
    res = http.delete(ctx.env.MEILISEARCH_URL .. "/indexes/records/documents/" .. id, {
      headers = headers,
    })
  else
    res = http.post(ctx.env.MEILISEARCH_URL .. "/indexes/records/documents?primaryKey=id", {
      headers = headers,
      body = json.encode(json.to_array({
        {
          id = id,
          uri = input.uri,
          collection = input.collection,
          did = input.did,
          record = input.record,
        },
      })),
    })
  end

  if res.status >= 400 then
    error("meilisearch answered " .. res.status)
  end

  -- On delete `record` is null, which still lets the delete proceed.
  return input.record or true
end

JavaScript

import http from "happyview.http";

export default async function handle(input, ctx) {
  // Meilisearch ids may contain only letters, digits, `-` and `_`
  const id = `${input.did}_${input.collection}_${input.rkey}`.replace(/[^A-Za-z0-9_-]/g, "_");

  const headers = {
    Authorization: `Bearer ${ctx.env.MEILISEARCH_API_KEY}`,
    "Content-Type": "application/json",
  };

  const res = input.action === "delete"
    ? await http.delete(`${ctx.env.MEILISEARCH_URL}/indexes/records/documents/${id}`, { headers })
    : await http.post(`${ctx.env.MEILISEARCH_URL}/indexes/records/documents?primaryKey=id`, {
        headers,
        body: JSON.stringify([
          {
            id,
            uri: input.uri,
            collection: input.collection,
            did: input.did,
            record: input.record,
          },
        ]),
      });

  if (res.status >= 400) {
    throw new Error(`meilisearch answered ${res.status}`);
  }

  // `record` is null on delete, so this returns `true` and the delete proceeds.
  return input.record ?? true;
}

TypeScript

import http from "happyview.http";

interface RecordEvent {
  action: "create" | "update" | "delete";
  uri: string;
  did: string;
  collection: string;
  rkey: string;
  record: Record<string, unknown> | null;
}

interface Context {
  env: Record<string, string>;
}

export default async function handle(input: RecordEvent, ctx: Context) {
  // Meilisearch ids may contain only letters, digits, `-` and `_`
  const id = `${input.did}_${input.collection}_${input.rkey}`.replace(/[^A-Za-z0-9_-]/g, "_");

  const headers = {
    Authorization: `Bearer ${ctx.env.MEILISEARCH_API_KEY}`,
    "Content-Type": "application/json",
  };

  const res = input.action === "delete"
    ? await http.delete(`${ctx.env.MEILISEARCH_URL}/indexes/records/documents/${id}`, { headers })
    : await http.post(`${ctx.env.MEILISEARCH_URL}/indexes/records/documents?primaryKey=id`, {
        headers,
        body: JSON.stringify([
          {
            id,
            uri: input.uri,
            collection: input.collection,
            did: input.did,
            record: input.record,
          },
        ]),
      });

  if (res.status >= 400) {
    throw new Error(`meilisearch answered ${res.status}`);
  }

  // `record` is null on delete, so this returns `true` and the delete proceeds.
  return input.record ?? true;
}

How it works

1. On create or update: sends a POST request to Meilisearch’s document API with the record data wrapped in an array. Meilisearch upserts by id, so a document for the same record is replaced.
2. On delete: sends a DELETE request to remove the document from the index by its id.

A Meilisearch document id may contain only letters, digits, - and _, so an AT URI cannot be one. The script builds the id from the DID, collection and rkey, replaces every other character with _, and stores the AT URI in a uri field.

The primaryKey=id query parameter names id as the index’s primary key. Without it, Meilisearch infers the key from the first document’s attributes ending in id, finds both id and did, and refuses to add the document.

Meilisearch answers a write with 202 Accepted and applies it later as a task, so a document it refuses while processing the task does not fail the script. Check failed tasks in Meilisearch itself.

input is the record event. Meilisearch expects an array of documents, which Lua’s json.to_array guarantees (JSON). happyview.http returns a 4xx or 5xx answer as an ordinary result, so the script checks status and raises for it. A failed record script is retried and then dead-lettered.

Configuration

Set these as script variables from the dashboard or the admin API:

Variable	Value
MEILISEARCH_URL	Your Meilisearch URL, such as http://meilisearch:7700
MEILISEARCH_API_KEY	A Meilisearch API key with write permissions

A script reads them as ctx.env.MEILISEARCH_URL and ctx.env.MEILISEARCH_API_KEY.

Use case

This record script keeps an external search index in step with your indexed records as events arrive, with no polling or scheduled jobs.

Meilisearch can be self-hosted, so you can run it on the same network as your instance.

Combine this with a query script that searches Meilisearch instead of the local database for full-text search beyond the substring match of happyview.db’s search.