A study of robot learning found that ranking several visual observations at once can be a competitive alternative to comparing observations in pairs. Across Drawer Open, Door Close and Button Press, at least one listwise ranking size matched or outperformed the other methods in mean success rate.
The central question was whether a visual reward model could learn from listwise VLM preferences using the Plackett–Luce objective, a way of learning from an ordered list of choices. The framework combines VLM-generated preferences with a Plackett–Luce reward model inside a reinforcement-learning loop.
How the comparison worked
The benchmark used three Meta-World tasks: Drawer Open, Door Close and Button Press. They were selected as visually interpretable object-manipulation completion tasks.
For the main experiments, GPT-5.6 Luna generated visual rankings, using original image detail and no reasoning effort. The primary setup used four feedback groups per iteration. Listwise PL and BT-Kwise used K = 3, 4 or 5, where K is the ranking size; pairwise methods were limited to K = 2.
Each task and method configuration used five independent seeds, numbered 0 through 4. Each run lasted 100,000 environment steps, organized as 200 iterations of 500 steps, with 18 warm-up iterations and 182 subsequent iterations. Policies were evaluated every 10,000 steps, including the final checkpoint, in 20 deterministic episodes; success rate was the number of successful episodes out of 20.
Results depended on the task
On Drawer Open, the RL-VLM-F baseline had the highest final success rate at 92%. Listwise training with a ranking size of four reached 86%, tying the Oracle baseline.
Door Close produced a different ordering. Listwise training with five observations ranked at a time reached 54% mean final success, compared with 47% for the BT-Kwise baseline at the same ranking size. The RL-VLM-F and BT-Pairwise baselines each reached 39%.
Button Press was less consistent. The BT-Kwise method with four observations reached 45%, while the listwise method reached 41%. With five observations, however, listwise training reached 26% compared with 7% for BT-Kwise.
Those differences did not amount to a clear overall statistical win. Matched tests using the five seeds found no significant difference between listwise and BT-Kwise training at any ranking size on Drawer Open or Door Close. On Button Press with five observations, the listwise advantage was nominally significant before adjustment, with t(4) = 2.881 and p = 0.045, but not after Holm correction for the multiple tests, when p = 0.405.
Fewer judgments, but not a guaranteed gain
An ablation cut the feedback groups to one per iteration. On Drawer Open, final success averaged 0.49 ± 0.17 for PL and 0.55 ± 0.23 for BT-Kwise across five seeds, with the ± values representing the reported standard error. Both methods failed completely on at least one seed.
The lower-budget listwise setup used 182 calls to the vision-language model, compared with 728 for the pairwise baseline and 1,456 for RL-VLM-F. The smaller number of calls therefore came with no established performance advantage.
Two exploratory analyses raised the possibility that how observations are selected may matter as much as how they are ranked. In one Drawer Open setting with one feedback group and rankings of three observations, state-diverse sampling reached 93% ± 7%, compared with 27% ± 20% for uniform sampling across three matched seeds. In a separate Door Close analysis using state-diverse sampling and rankings of three observations, increasing the feedback groups from one to five was associated with mean success changing from 25% to 62%, while the number of rankings rose from 182 to 910.
What the tests leave open
The comparison covered three Meta-World tasks and used five seeds for each task and method configuration, so the results were sensitive to the particular test setting. The one-group feedback analysis also reported high seed dependence, with both methods failing completely on at least one seed.
The manuscript states that it has been submitted to the IEEE for possible publication. It reports support from the National Science Foundation under Grant No. 2521982 and the NSF CISE REU Student Funding Program.
Paper data and sources
Original title: Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning
Authors: Srivalli Katkuri, Maxwell Kawada, Juan Wachs
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text