Exploring Vision-Language Models for
Open-Vocabulary Zero-Shot Action Segmentation

Abstract

Downloads

VLM features — available now

We release the pre-extracted per-frame VLM features for all 14 VLMs across all three datasets (GTEA, 50 Salads and Breakfast), together with the cached action-label text embeddings. These are the exact features behind every number reported on this page.

Extracting features from large VLMs demands substantial compute — ours were extracted on an NVIDIA A6000. By releasing ready-to-use embeddings we hope to remove that barrier, so that the community can iterate on open-vocabulary action segmentation and other action understanding tasks without repeating the extraction cost.

The archive is large. Google Drive cannot virus-scan a file this size, so the browser will warn before downloading; for a transfer this big we recommend a resumable command-line client such as gdown or rclone rather than the browser.

Method

OVTAS is a training-free, zero-shot pipeline that requires only a set of candidate action labels (the action-set supervision) and the video frames as input. Action-set supervision assumes that, given the high-level activity (e.g. “making tea”), we know the set of possible fine-grained actions (e.g. “boil water”, “pour tea”, “add sugar”) — but neither their order nor their boundaries.

Stage 1 — FAES

Let \(T\) be the number of frames, \(N\) the number of action labels and \(C\) the embedding dimension. With row-wise \(\ell_2\)-normalised frame embeddings \(\mathbf{X}\in\mathbb{R}^{T\times C}\) and action-label text embeddings \(\mathbf{A}\in\mathbb{R}^{N\times C}\), the similarity matrix and the per-frame classification probabilities are

Stage 2 — SMTS

We adopt the Action Segmentation OT (ASOT) decoder of Xu and Gould. Given \(\mathbf{S}\), we define a visual cost \(\mathbf{C}=\mathbf{1}-\mathbf{S}\) and a diagonal temporal prior \(R_{ij}=\left|\tfrac{i}{T}-\tfrac{j}{N}\right|\), which encourages monotone alignment. We then solve for a coupling \(\Pi\in\mathbb{R}^{T\times N}\):

where \(U(\mathbf{u},\mathbf{v})\) is the transport polytope and \(H(\Pi)\) the entropy of \(\Pi\). The entropy term makes the problem convex with a unique solution; \(\Pi^{\star}\) is computed with log-stabilised Sinkhorn iterations, which scale linearly in \(T\!\times\!N\) per iteration. Each frame is finally assigned the action with maximum transported mass, \(\hat{y}_t=\arg\max_{j}\Pi^{\star}_{t,j}\).

What “training-free” means here

Training-free refers strictly to the absence of backpropagation-based weight updates. Both encoders are used off-the-shelf and are never fine-tuned. Optimal-transport hyperparameters are selected by grid search on a small set of held-out videos per dataset and then fixed for all experiments — we use \(\varepsilon = 0.07\), \(\alpha = 0.5\), \(r = 0.04\), \(\lambda_{\text{frames}} = 0.11\) and \(\lambda_{\text{actions}} = 0.01\). Unlike the unbalanced formulation of prior work, we find the balanced formulation gives the strongest results. Because the action order is unknown under action-set supervision, the ordering of action embeddings used to build the temporal prior is randomised.

Positioning against prior supervision regimes

Scroll the table horizontally →

Table 1. Comparison with other TAS methods by generalization ability and training-data needs. ✓ denotes applicability. OVTAS occupies a regime no prior method does: training-free, zero-shot and open-vocabulary at once.

Ablations

Stage ablation

We ablate Stage 1 by randomly permuting the frame and action-embedding features, and Stage 2 by taking per-frame argmax predictions instead of the OT decoding. Both stages prove critical: removing either produces significant drops across every metric on every dataset.

Scroll the table horizontally →

Table 6. Stage ablations. Ablating Stage 1 corresponds to Random-ASOT; ablating Stage 2 corresponds to per-frame argmax decoding. Drops from the full pipeline are shown in blue.

Temporal prior and \(\ell_2\) normalisation

We independently ablate the \(\ell_2\) normalisation in Stage 1 and the temporal prior in Stage 2, reporting on GTEA. Both design choices turn out to be load-bearing: without either one, the Avg falls to 2.69 and 3.95 respectively — close to the random baseline (1.56) and well below the strongest training-free baseline (7.27).

Table 7. Ablation on the temporal prior and \(\ell_2\) normalisation, on GTEA. Arrows show the drop from Best.

Analysis

Which VLM family works best?

Averaging each family's performance across the three benchmarks yields a stable ranking: SigLIP outperforms all others, followed by CLIP, with OpenCLIP and PECore trailing. This ordering is consistent across all three datasets, suggesting the relative strength of each family is stable across domains.

Does scaling the VLM help?

As future directions to close this gap, we point to (i) stronger text prompting and (ii) video-frame pre-processing such as cropping — together these may extract frame and action-label embeddings that do improve with increasing VLM size.

Where does OVTAS get confused?

Examining per-action confusion on GTEA (SigLIP-M1) shows that take acts as a dominant attractor for most manipulation actions, since its reaching motion overlaps with the onset of nearly every other action. shake is almost entirely misclassified, reflecting the limitation of image-trained VLMs on dynamic motions.

Scroll the table horizontally →

Table 8. Action-type confusion on GTEA (SigLIP-M1).

Effect of video length and segment density

Performance decreases consistently as video length increases, across all three datasets: longer videos bring greater temporal variability and amplify error propagation in training-free segmentation.

Table 9. Performance variation by video duration.

The density of fine-grained action boundaries matters just as much. GTEA videos comprise many brief segments (mean ≈ 36 per video) and the model performs worst there; Breakfast averages ≈ 5 segments and performs best; 50 Salads sits between the two. Tightly packed sequences of short actions are especially difficult.

Table 10. Performance variation by the number of ground-truth action segments per video.

Qualitative Results

Each pair of colour bars shows the ground-truth segmentation (top) above the OVTAS prediction (bottom) for one video, with segments coloured by action label.

Figure 5. Qualitative segmentation results of OVTAS. Top bar: ground truth. Bottom bar: prediction.

Limitations & Future Work

• We evaluate on three prominent action segmentation datasets; future work can extend the study to further benchmarks and domains beyond kitchen and egocentric activities.
• Prompt engineering is barely explored here. Our label normalisation is deliberately simplistic (“pour_coffee” → “pour coffee”, with verbs attached to build short phrases on GTEA). Richer prompting is a promising and cheap lever.
• Image-trained VLMs struggle with dynamic motions — shake is almost entirely misclassified on GTEA, and take acts as a dominant attractor because its reaching motion overlaps with the onset of nearly every other action.
• Strengthening the temporal modelling ability of the optimal-transport decoder is important for training-free segmentation pipelines generally, especially for long videos and densely packed short segments.

BibTeX

@article{unmesh2026ovtas,
  title   = {Exploring Vision-Language Models for Open-Vocabulary
             Zero-Shot Action Segmentation},
  author  = {Unmesh, Asim and Ramesh, Kaki and Patel, Mayank and
             Jain, Rahul and Ramani, Karthik},
  journal = {arXiv preprint arXiv:2602.21406},
  year    = {2026}
}

Acknowledgements

We acknowledge the Feddersen Distinguished Professorship Funds. This work was also supported by NSF under the Partnership for Innovation: Technology Transfer (PFI-TT) 2329804. Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the funding agency.