Building a client

Core is single-tenant, stateless about ownership, and asynchronous. Those three facts decide the shape of every client built on it. This page documents the patterns the reference frontend uses, and why.

1. Keep the client server-side

const CORE_URL = process.env.CORE_API_URL || 'http://127.0.0.1:8077'
const CORE_TOKEN = process.env.CORE_API_TOKEN || ''

Never import a core client into browser code. Core has no per-user auth, so every call must be made by something that has already checked ownership and resolved which video ids the caller may touch.

Wrap failures so "core said no" and "core did not answer" stay distinguishable:

export class CoreApiError extends Error {
  constructor(message: string, readonly status: number) { … }
}

A connection failure becomes a 503 naming the URL it tried; an HTTP error keeps core's own detail, which is written to be actionable.

2. Own the identity mapping

Core knows video_id. It does not know users, projects, titles or permissions. Keep a row per video with all of that, plus the ingest lifecycle.

core_video_id text NULL,           -- sha1(bytes)[:16], null until core has the bytes
status        text NOT NULL,       -- pending|uploading|queued|analyzing|ready|failed
job_id        text NULL,
stage         text NULL,
ingest_config jsonb NULL,          -- replayed verbatim on re-index
analyzers     text[] NOT NULL,     -- what it can be asked
aggregates    text[] NOT NULL

analyzers and aggregates are stored as arrays rather than JSON because the common query is a predicate: does this video have people?

3. Create the row before core has the bytes

4. Recover from a restarted core

Jobs live in memory. A poll that 404s does not mean the ingest failed:

job 404
 ├─ row has core_video_id, and GET /videos/{id} succeeds
 │    → the ingest finished before the restart. Complete the row from the video itself.
 └─ otherwise
      → genuinely lost. Fail the row with a message pointing at re-index.

Without this, a restart mid-ingest leaves rows stuck on analyzing forever with no way out.

5. Scope every call explicitly

Put one resolver between your callers and core, and pass everything through it:

async function resolveScope(context, requested?: string[]) {
  // 1. require a project on the request
  // 2. load that project's rows for that user  (RLS applies)
  // 3. keep only rows that are ready and have a core_video_id
  // 4. intersect with `requested`, and report what was dropped
}

In the reference app every agent tool but the pass-through display one goes through this. A model that hallucinates a video id, or repeats one it saw in another conversation, gets it filtered out here rather than answered from someone else's footage — and the rejected ids are reported back to the model so it corrects itself.

6. Choose detail deliberately

For a UI, detail=standard. For an agent, detail=minimal followed by GET /videos/{id}/chunks?chunk_ids=… on the few that matter — about 1.6k tokens against 19k for the same five moments at full detail.

Do not expose core's filters to a language model. Every filter is a hard AND against exact stored labels, so it can only remove results, and models reliably send speculative ones that silently empty a good result set.

7. Delete with a reference count

// count other rows pointing at the same core_video_id
// only if none remain: DELETE /videos/{core_video_id}

The reference routes

The frontend exposes these; each wraps core with an ownership check.