An AI system that sends each video-and-prompt example through a task-specific expert reported a 62.9 average accuracy across five COIN procedural-video tasks, the highest mean among four routing choices tested on that benchmark.
VideoLLM-MoTE-1B+5E scored 65.1 on Step, 50.3 on Next, 50.5 on Proc., 94.4 on Task and 54.2 on Proc.+. The five scores were averaged arithmetically to produce 62.9.
A route built into the decoder
The architecture keeps the visual encoder, projector, token embeddings and attention path shared. In the decoder, an always-active shared expert works alongside selected task experts, and each video-prompt example follows one sample-level route through all decoder layers.
In the main COIN setup, task identity was an explicit routing variable, with one routed expert assigned to each of the five tasks. That means the route was set for the whole sample rather than changing from token to token.
Lower active compute in the reported profile
On the reported profile, VideoLLM-MoTE-1B+5E had 5.3 billion total LLM parameters and 2.0 billion active parameters, compared with 8.0 billion total parameters and 8.0 billion active parameters for VideoLLM-online-8B-v1+. Its listed compute was 28.9 TFLOPs versus 98.7, latency was 1.8 seconds versus 2.8, and decoding throughput was 37.7 tokens per second versus 27.8.
The reported models used four A100-80GB GPUs, sampled video at 2 frames per second and capped inputs at 1,200 frames. The language model was fine-tuned with LoRA, while the MLP projector remained trainable.
The cost of adding capabilities
Explicit task-index routing had a mean COIN accuracy of 62.9, versus 62.1 for learned sample routing, 61.5 for dense all-expert activation and 61.4 for learned token routing.
In a deliberate Step-Next expert swap, the two swapped tasks scored 21.5 and 17.4, while the untouched Proc., Task and Proc.+ routes retained scores of 50.5, 94.4 and 54.2. The paper reports similar locality for other swaps.
Adding experts sequentially preserved the earlier task scores in the reported test, but the final stage averaged 61.0 across five active tasks, compared with 62.9 for joint training.
An expanded VideoLLM-MoTE-1B+6E checkpoint combined COIN and Ego4D routes. It reported a Fluency score of 51.0—the highest in that comparison—alongside PPL of 2.57 and a TimeDiff of 2.25; Ego4D-only baselines had lower PPL and TimeDiff.
A controlled test beyond video
An auxiliary document evaluation applied the routing design to SROIE and CORD receipt-image key information extraction. GLM-OCR-MoTE-0.9B+2E reached 87.90% Micro-F1 on SROIE and 95.79% on CORD, versus 55.04% and 20.07% for the corresponding dense baselines. The retained OCR route preserved the reported OmniDocBench v1.5 performance.
When a task index was unavailable, a 22.7-million-parameter MiniLM selector was evaluated on 50 generated prompt variants for each COIN task. With a 70/10/20 train-validation-test split, it reached 100% test F1 and took 4.35 milliseconds per prompt.
What the results leave open
The figures are reported benchmark results, and the evaluation gives no confidence intervals, repeated-run variation or inferential tests. The efficiency profile also comes from one stated hardware and input configuration.
The prompt-selector score came from generated variants of predefined, clearly separated intents; ambiguous, overlapping and compositional prompts were not tested. The document results were auxiliary, and the combined cross-dataset model was compared with separate task-specific checkpoints.
The work is an arXiv version 1 preprint dated 25 Aug 2026.
Paper data and sources
Original title: MoTE: Mixture of Task Experts for Multi-Task Video Understanding
Authors: Muhammad Asad Ali, Umar Khan, Nadia Robertini, Didier Stricker
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text