Aggregators

An analyzer describes one chunk. An aggregator reads every chunk at once and produces a conclusion about the whole video — a summary, a chapter breakdown, who was present throughout, what stands out.

This is the part retrieval structurally cannot do. "How busy was the store" cannot be answered by finding the five most relevant moments; it needs all of them, counted.

The twelve

Dependencies and ordering

depends_on names either an analyzer (a requirement on the video) or another aggregator (which gets pulled in and run first). Execution order is derived from the graph, so a new aggregator that reads another's output just declares it.

people ──▶ entities ──▶ entity_timelines
                    └─▶ cooccurrence
diarization ──▶ speaker_stats
            └─▶ sentiment
object_detection ──▶ object_entities

Caching

Results already stored are reused unless force is set. This matters: summary, chapters, events, entities and object_entities each bill API calls, so re-uploading a video to add one analyzer would otherwise re-buy its whole summary.

Ingestion runs aggregators with force=false for exactly that reason. The response reports llm_calls_saved so you can see what the cache was worth.

Notable outputs

summary

A hierarchy built by halving the video until a leaf covers a few chunks. tiers[0] is the finest level and the last tier is the whole video. Depth follows length rather than a fixed block size — a 39-chunk video produced 4 tiers of 8 / 4 / 2 / 1 sections.

{
  "summary": "…a few sentences covering the whole video",
  "key_points": ["…"],
  "topics": ["…"],
  "depth": 4,
  "tiers": [{ "level": 0, "section_count": 8, "sections": [{ "summary": "…", "start": 0.0, "end": 24.0, "chunk_ids": [0, 1, 2] }] }],
  "sections": [ "…the finest tier, repeated for retrieval" ],
  "based_on": ["default_video", "diarization"]
}

The whole-video summary is a reduction over merged sections, not one huge prompt.

chapters

Consecutive chunks grouped into titled sections, each with title, summary, first_chunk, last_chunk, and resolved start / end / chunk_ids. Chapters naming a chunk id that does not exist are dropped rather than trusted.

events

Discrete occurrences — someone arriving or leaving, a transaction completing, an object changing hands. One entry per event, not one per chunk.

{
  "events": [{ "event": "A customer approaches the register", "chunk_id": 12,
               "actor": "woman in a grey coat", "category": "arrival",
               "start": 240.0, "end": 260.0 }],
  "categories": { "arrival": 4, "transaction": 2 },
  "based_on": ["default_video", "people"]
}

stats

Free, and the right answer to almost every "how many / how often / how busy" question. Contains duration, chunks, and — depending on which analyzers ran — a people series with min/max/mean/busiest/quietest, object frequencies, and speech totals.

novelty

Every chunk scored by how far it sits from the video's own mean. Outliers are those more than two standard deviations out, so "unusual" scales with how varied that particular video is rather than using a threshold tuned on one clip.

entities

The one that turns sightings into people. Embeddings are clustered under constraints, and only then does an LLM write narratives.

• People co-visible in one chunk cannot be the same person.
• Descriptions too generic to identify anyone are left unlinked.
• The signature is clothing only. Blending in appearance dragged same-person scores to 0.69–0.87 because it drifts and sometimes contains meta-commentary ("same woman as box 3") that the embedding treats as content.

Asked to match people directly, an LLM confidently merges anyone in dark clothing. The clustering is what stops that.

object_entities

Reuses the same clusterer with two filters people do not need:

• Static fixtures (counters, shelves) are counted but not tracked — "the counter was present throughout" is true of every frame and tells a search nothing. They come back under fixtures.
• Person-like detections are handed to the people aggregator, which has clothing to identify them with.

Running them

• Automatically, at the end of every ingest, with force=false.
• On demand, with POST /videos/{video_id}/aggregates — returns 202 and a job id.
• From the frontend, on the video detail page, which proxies the same call.

Adding an aggregator

One module with id, depends_on and aggregate(ctx), plus a line in videomind/aggregators/__init__.py. Order is derived from depends_on; an aggregator whose analyzer is absent is skipped, not failed.