The agent
Nine tools, how the agent chooses between them, and the scoping that keeps one project's footage out of another's answers.
The chat is a streaming multi-step tool loop built on the AI SDK. The model is given an inventory of the project's videos and nine tools, and runs up to 15 steps per turn — retrieving, reading, then answering.
What the model is told
The system prompt carries a live inventory: every video in the project with its id, duration, status, which analyzers ran, and which video-level results exist.
That last part is load-bearing. Which analyzers ran decides what a video can be asked at all — one
analysed without people cannot answer who was there, one without diarization has no speech to
quote. Listing them means the model picks a workable tool first rather than discovering the gap
through an empty result.
Videos tagged by the user for the current message are marked TAGGED BY USER; anything not
ready is marked NOT SEARCHABLE YET or FAILED.
The tools
| Tool | Use |
|---|---|
ask_video | The default. "What / why / how / summarise / explain." Routes the question to whichever video-level results address it and returns a grounded answer with sources |
search_moments | "Find the part where… / when does… / show me…" — timestamped moments |
get_video_insights | Counts, structure and whole-video conclusions: how busy, what stands out, which brands, chapter breakdown, who talked most |
get_video_entities | Who was in the video, what each person did, how long they were present |
get_video_transcript | Exact wording and quotes, optionally over a time range |
read_chunks | The full stored analysis for specific moments |
list_project_videos | Resolve a video named in words into its id, or check what it supports |
show_clips | Open the artifact panel with a playable reel |
show_artifact | Long written output — a full summary, a written-up transcript. Not for clips |
ask_video
Wraps core's POST /ask. Returns the answer, the source moments, and consulted — which
video-level results core routed to, per video. That last field is worth reporting: it explains why
an answer knows something no single segment says.
search_moments
Wraps POST /query. One search tool where an earlier design had three.
Parameters: query, video_ids, analyzer, field, limit, score_threshold, detail.
Synthesis is off — the chat message does the explaining, and a second model call here would pay
for prose the agent is about to rewrite anyway.
Core's filters dict is deliberately not exposed. Every filter is a hard AND against exact
stored labels, so it can only ever remove results — and the model reliably sent speculative ones
("white shirt" as an object, a people count on an analyzer that stores none), each of which
silently reduced a good result set to nothing. Prose in the tool description did not stop it. The
semantic query already covers what those filters were being used to express.
The same reasoning applies to score_threshold, which is documented to the model as the one
parameter that can empty a good result set: correct matches routinely score 0.55–0.60.
read_chunks
The second half of the cheap retrieval pattern: run search_moments at detail="minimal" to
decide which moments matter, then read only those in full. Reading five moments this way costs a
fraction of asking for full detail on every hit.
It also works as a plain reader — give a time range instead of chunk ids to see what happened between two points. Capped at 12 chunk ids per call.
get_video_transcript
Asks for diarization first and falls back to transcript, so the model does not have to know
which one ran. Output is markdown lines of [m:ss] SPEAKER: sentence, budgeted to 12,000
characters and cut on a segment boundary so no sentence is left half-quoted. When truncated, the
result says where to resume from.
speakers_attributed tells the model whether it may name who said something.
get_video_insights
Reads one aggregate by name, or lists what the video has when called without one — returning twelve full results would swamp the context for nothing.
get_video_entities
Defaults to min_appearances: 2. Someone seen once is already fully described by that moment; the
value of this tool is the people who were followed.
Scoping
Core has no users, no projects and no row-level security: whoever can reach it can read every video in it. Scoping is entirely the frontend's job.
Every tool except show_clips resolves the ids it may touch through a single choke point, which:
- Requires a project on the conversation.
- Loads that project's
video_corerows for that user — RLS applies. - Keeps only rows that are
readyand have acore_video_id. - Filters the model's requested ids against that set, and reports what it dropped.
A model that hallucinates a video id, or repeats one it saw in another conversation, gets it filtered out here rather than answered from someone else's footage. Core is then always called with explicit ids.
Rules the agent follows
- Retrieve before answering. Any claim about a video must come from a tool result in the conversation.
- Cite timestamps as
m:ss. - Check the analyzer list before choosing a tool. If the needed analyzer never ran, say so and suggest re-indexing — do not answer from a different signal and imply it is the same thing.
- A video that is not
readyis reported as still analysing, not searched. - Ambiguous across several videos → search all of them, and name which video each finding came from.
- Empty results are retried once with the query alone, then a different analyzer or a video-level insight. If that is also empty, say so plainly. Never invent a moment, a quote or a timestamp.
- Clips go in the panel, never as raw URLs in chat.
- Attribute quotes to a speaker when the transcript is diarized, and do not attribute when it is not.
Models
The model picker sits above the input. Providers are OpenAI, Google, Groq, Cerebras and LM Studio;
the list lives in app/agent/lib/ai/models.ts and any model whose provider key is set can be
selected. LM Studio needs no key — it talks to a local server at LMSTUDIO_BASE_URL, so its
models only work while that server is running with the model loaded.
Tagging videos
The Videos control next to the model selector picks exactly which videos this message should
search. Tagged videos appear as chips above the input and are passed to the agent as
selectedVideoIds, which the system prompt turns into "search these unless the user clearly means
others". With nothing tagged, the default scope is every searchable video in the project.