Zero-shot vision

Zero-shot here means running a vision task with text and prompts rather than a trained head over a fixed label set. The class list becomes an argument, so a pipeline can be built before the labelled data exists and edited afterwards by changing a string.

It covers classification, detection and segmentation in the same framework, which means a first version of a pipeline can be assembled from three models that need no training step at all.

GroundingDINO takes a space-dot prompt — "cat . dog . person" — and reads it as the class list. OWL-ViT v2 accepts a plain list and trades a little accuracy for speed. Both ignore the fixed label space that a conventional detector would force on you.

• IDEA-Research/grounding-dino-tiny — 48M parameters, the highest accuracy of the documented zero-shot detectors, strongest on complicated scenes and small objects, at roughly 2–5 seconds per image.
• google/owlv2-base-patch16 — 93M parameters, fast enough for interactive work at roughly 0.5–1 second per image.
• google/owlvit-base-patch32 — the original OWL-ViT, kept for compatibility and baseline comparison.
• Thresholds default to 0.3 and can be overridden per call, so precision and recall can be traded without reloading the model.

SAM segments from a point or a box prompt, which makes it the natural second stage after any detector. SAM3 goes further and accepts text prompts over a very large concept vocabulary, so a phrase can stand in for the click.

The grounding_dino_sam preset wires both stages together: detect with text, filter by score, segment each box, refine the mask with a morphology pass. The result is a set of instances that carry a box and a mask at the same time.

Wording changes accuracy more than most teams expect, and the framework's guidance is specific about it.

• Use specific singular nouns with describing words — "red apple", "delivery van" — rather than bare category names.
• Include the variations you expect to see instead of one canonical name.
• Write object names in the language the model was trained on; English object names are the reliable default.
• Avoid heavy plural forms and very broad terms: a generic word finds less than the specific names you actually care about.
• Keep prompts short — long multi-clause descriptions degrade accuracy instead of sharpening it.

There is no class list to retrain: the prompt is the class list.

A prototype no longer waits on a labelling project. Categories that shift with the season, the supplier or the campaign can be renamed in a config file, and a new one can be tried on a single frame.

Detection and classification can run on the same frame, so a scene is described both as a whole and object by object. When the two do not depend on each other, the graph planner runs them side by side.

Load a detector with text prompts and a classifier with labels, then compose them in a graph — or start from a preset that already does it. Nothing on that path needs training data, a dataset directory or a labelling tool.

Trying a different category is one argument away. The effort goes into deciding what to ask for rather than preparing what to train on.

• A detector plus a classifier is a two-provider graph, and the pipeline runs in the order the data requires.
• Prompts and thresholds can be adjusted per call, so tuning does not mean reloading models.
• The same call pattern works from the CLI while a configuration is still being explored.

On a fixed, repetitive task with a clean labelled dataset, a task-specific model is still the stronger tool; zero-shot earns its place where labels do not exist yet or the classes keep moving.

Accuracy also depends on image quality, object size and prompt phrasing, so the setup is decided by results on your own frames rather than by benchmark numbers. Training and fine-tuning are not part of MATA today — the training module is on the v2.0 roadmap.