Video indexing and search

Two graph nodes do the work. IndexVideo samples frames from a video and embeds them into an index; EmbeddingSearch embeds one or more query strings into the same vector space and ranks the indexed frames against them.

Both are ordinary nodes in a graph, so search sits inside a pipeline rather than beside it. When the nodes shipped in v1.9.7, a demonstration that had been a few hundred lines of indexing code became a graph of about twenty lines.

Indexing samples the video at a fixed rate instead of processing every frame, which keeps the cost proportional to what you actually need.

• sample_fps sets the rate — 1.0 frame per second by default; raise it for fast action, lower it for long recordings.
• Frame mode embeds one sampled frame at a time and is the default; clip mode embeds a window of frames as a single vector, which is what encoders with temporal attention are for.
• The index is a first-class artifact (VideoIndexData), so it can be stored, passed between nodes and reused by more than one query.
• Any embed-capable model works: CLIP at 512 dimensions, X-CLIP for video-language, and Qwen3-VL-Embedding up to 4096 dimensions, truncatable through the embed_dim parameter.

Each query string is embedded into the same space as the frames and answered by cosine nearest-neighbour search, so a query is written the way the moment would be described to a person.

• One query or a list of queries per call, each returned separately with its own matches.
• top_k caps the matches per query (5 by default); threshold drops anything below a similarity floor.
• Every match carries its similarity score and a start and end timestamp, so a reviewer can jump straight to the moment.
• Two metrics are recorded — indexed_frames while the index is built, num_queries while it is searched.

CLIP is the fast, familiar choice and works well for objects and scenes described in ordinary words. X-CLIP embeds a window of frames with a temporal transformer, which is what a query about movement needs — a cyclist weaving through traffic rather than a cyclist.

Qwen3-VL-Embedding places text, images and video in one vector space, and its output can be truncated to a smaller dimension when an index has to stay compact.

The query does not have to know the vocabulary of the model: "vehicle making an abrupt lane change near pedestrians" is a valid string, and returns matches or nothing.

The documented example is dashcam footage: several behaviour queries run against one index, each returning the moments that match. The same shape applies to any recording where finding a moment by description is cheaper than watching it — review of incidents, of long shifts, of archives nobody has time to re-watch.

Because the index is an artifact, a review workflow can build it once and query it many times, including with questions written after the footage was captured.

• Queries are assembled per review, not per recording, so a new question does not mean a new pass over the video.
• Matches arrive as timestamps, which is what an editor or an investigator needs to open the clip.
• The same graph can carry other tasks, so search and detection can describe the same footage.

Install datamata, load an embed model, and run the graph with the video path; mata.infer accepts video as an input alongside images, and results come back as search results you can iterate, print or persist with the rest of the pipeline output.

The index lives in memory for the run, and because it is an artifact it can be serialised where a workflow needs to keep it.

• The embed model is a provider like any other, so it can be swapped without touching the rest of the graph.
• One index answers many queries, which is what makes a review session cheap.
• Search results are ordinary artifacts, so they can be exported or displayed like any other result.