Architecture

How a question travels from the chat box to a vector search and back as a playable clip.

FalconVQA architecture

The shape

Next.js  ──►  /api/agent (AI SDK tool loop)  ──►  core (FastAPI)  ──►  Qdrant
   │                    │                              │
   │                    └──► Supabase Postgres         └──► Supabase Storage
   └──► Supabase Storage (uploads → public URL)             (video bytes, posters)

Four processes, and a clean split of responsibilities:

Component	Owns
Next.js app	Auth, projects, ownership, conversations, the agent loop, all UI
core (FastAPI)	Chunking, analysis, indexing, aggregation, search, question answering
Supabase Postgres	Projects, conversations, messages, `video_core` rows, RLS
Supabase Storage	Video bytes and posters — core's bucket, plus the app's upload bucket
Qdrant	Chunk vectors, five named fields per point, embedded in core's process

HTTP is the external boundary; composition is in-process. The UI and any future MCP server talk to core over HTTP, but aggregators call analyzer output directly. Core never calls itself over HTTP.

Ingesting a video

browser ──upload──► Supabase Storage (app bucket)
   │                        │
   └──POST /api/videos──────┘  (public URL)
              │
              ├─► insert video_core row (status: pending)
              └─► POST /videos/url ──► core: 202 + job_id
                                        │
                                        └─► background thread:
                                            fetch → chunk → analyze → index → aggregate
                                                │
                                                ├─► Supabase Storage (core bucket): mp4 + poster
                                                ├─► data/records/: the analysis
                                                └─► Qdrant: one point per chunk per analyzer

browser ──poll /api/videos──► reconcile row against GET /jobs/{id} ──► status: ready

The row exists before core has seen a byte, because the video's id is the hash of its contents. Everything identifying arrives later, through the job.

Answering a question

user message
   │
   ▼
POST /api/agent
   ├─ load the project's video_core rows  (RLS)
   ├─ build a system prompt carrying each video's analyzers and aggregates
   └─ streamText with 9 tools, up to 15 steps
        │
        ├─ tool call ──► resolveScope()  ──► allowed core_video_ids
        │                                     │
        │                                     ▼
        │                              core: POST /query | /ask | GET /chunks | /aggregates | /entities
        │                                     │
        │                                     └─► Qdrant search / record read
        │
        ├─ show_clips ──► (no backend call) ──► artifact panel renders the reel
        │
        └─ stream text + tool parts ──► client
                                          │
                                          └─► persist messages to Postgres

The agent never receives a raw core response. Every tool reshapes it: timestamps into m:ss, video_id into the row's playback_url and title, and core's error messages passed through verbatim because they are written to be the correction the model needs.

Inside core

chunk_video()            three modes: preset | weights | interval
  boundaries/            speaker, silence, cut, semantic detectors
  chunking/chunker.py    weighted fusion → boundaries → chunks
analyzers/               per-chunk passes,  registry in __init__.py
aggregators/             video-level passes, registry in __init__.py
vectordb/                BGE embeddings + Qdrant (named vectors)
api/core.py              upload / query / ask / aggregate  (the real logic)
api/app.py               a thin HTTP layer over it
api/ui.py                the built-in web UI, mounted only when VIDEOMIND_UI != 0
storage.py               Supabase Storage ↔ local cache
paths.py                 every path, all env-overridable

api/app.py is deliberately thin. All logic lives in api/core.py, which is importable — so a future MCP server, a CLI or a batch script is a new caller, not a new implementation.

Decisions worth knowing

Videos are URLs outside, paths inside. Every request and response speaks Storage URLs; everything below storage.py gets a local path and does not know Storage exists. That boundary is one module because it is the only place both forms are valid at once.

The video is downloaded, not streamed. Analyzers each re-open the file and seek per chunk — an access pattern priced for a local disk. Frame reading uses grab()/retrieve() rather than per-frame seeking, because seeking makes H.264 restart from a keyframe, measured at ~22× the cost per frame. Paying that to the network instead would be far worse.

No local path is ever stored. A stored path goes stale the moment the data directory moves. Records hold storage_path; the local file is re-derived from the content hash on demand and re-downloaded if the cache is cold.

Analyzers and aggregators are registries. Adding either is one module and one line. Nothing in ingest, the store, or the API changes — and because the frontend builds its analyzer list from /api/core/capabilities at runtime, nothing in the frontend changes either.

A clip is a range, not a stream. There is no per-clip URL anywhere. The panel loads the mp4 once and seeks, so several clips from one video cost one load between them.

Deployment shape

Piece	Where
Next.js app	Vercel, or any Node host
core	A GPU host — it runs Whisper, pyannote, YOLO, EasyOCR and CLIP locally
Postgres + Storage	Supabase Cloud
Qdrant	Embedded in core's process by default; can be pointed at a server

Core's port must not be publicly routable. It has no users and no row-level security — VIDEOMIND_API_TOKEN is a boundary, not an authorisation model. See Authentication.

Known limitations

Cross-video entity linking is not built. Entity ids are meaningful within one video only.
Within-chunk person tracking fragments. IoU tracking at 1 fps loses people who move between sampled frames; box_id is a referent within a chunk, not an identity claim.
Qdrant payload indexes are inert in embedded mode. Filtering is correct but scans. Fixed by running Qdrant as a server — no code change needed.
Jobs live in memory. A restart loses job history; vectors and records survive on disk.
Chapters can return a single chapter on unbroken single-location footage. Arguably correct, but not useful in a timeline.