An AI routing system was substantially better at picking the right skill when it used information about the user, but only in cases where that profile information changed which skill suited the task, according to an arXiv preprint. On those profile-sensitive cases, Hit@1, the share of cases where the first-ranked skill matched the benchmark's annotated target, was 0.630 with profile input and 0.444 with task-only input. On CF-Switch Accuracy, a measure of whether the selected skill changes when the profile changes, the scores were 0.377 and 0.026. For profile-insensitive cases, the figures were nearly identical: 0.832 with profile input and 0.838 with task-only input.
The paper frames personalized skill routing as a retrieval problem. Relevance is judged jointly from the task and the task-relevant parts of the user's profile, rather than from the task in isolation. The central question is whether that extra context helps precisely when a profile change alters the suitable skill, while leaving profile-insensitive requests largely alone.
A test built around changing profiles
To examine that question, SkillFeed-Bench assembled 228,432 candidate skills across 10 categories and 329 test samples. Each sample paired a task with a structured user profile, one annotated reference skill and semantically similar hard negatives. Of the test samples, 162 were profile-sensitive, including 77 profile-counterfactual cases and 85 standard-with-constraints cases. The other 167 were profile-insensitive.
SkillFeed's inference process has two stages. It first retrieves candidates from the repository at scale, then applies fine-grained reranking to the shortlist. The retriever was trained in two phases, starting with 2,712 task-centric instances for task-only alignment and then using 7,459 profile-conditioned instances for profile-aware adaptation. The models were initialized from Qwen3-Embedding-0.6B and Qwen3-Reranker-0.6B.
Across the full benchmark, the pipeline recorded Hit@1 of 0.751. Its MRR@20 was 0.803, a ranking score based on how high the target appears among the first 20 results. The 95% confidence interval for Hit@1 ran from 0.706 to 0.794, while the interval for MRR@20 ran from 0.760 to 0.846. The key comparisons used bootstrap confidence intervals and paired permutation tests.
Compared with the Qwen3-Embedding-0.6B plus Qwen3-Reranker-0.6B baseline, the full pipeline's Hit@1 was 0.231 higher, with p < 0.0001. That gap is a comparison between routing configurations on this benchmark.
Benchmark construction included explicit quality checks. Samples with confidence below 0.80 or task-skill lexical overlap above 0.50 were discarded, followed by consistency checks and manual review. Each retained original and counterfactual instance was independently reviewed by two annotators. Disagreements were resolved by an expert reviewer, and ambiguous cases were discarded.
The gains had boundaries
One comparison exposed a trade-off in the retrieval design. The profile-chunk variant had Hit@1 of 0.739, slightly below the full pipeline's 0.751, but Hit@10 of 0.918 and nDCG@20 of 0.838. The figures show that a system can rank the target within its leading results while still trailing on the first position.
Performance was less even across domains. Hit@1 was 0.790 for General queries, 0.614 for Tech, 0.533 for Science and 0.455 for Other Domains. The subgroup sizes were 248, 44, 15 and 22, respectively. The supplied subgroup results did not show uncertainty intervals, and the specialized categories had lower scores than General.
The result is therefore a focused one, not a blanket advantage for adding profile data. The largest difference appeared when the profile changed the suitable skill, while the profile-insensitive comparison remained almost unchanged. This matches the paper's framing of routing as a task-and-profile relevance problem. The document is an arXiv preprint, version 1, dated 28 August 2026.
Paper data and sources
Original title: Beyond Task-Only Matching: Personalized Skill Routing with Counterfactual Evaluation
Authors: Tianle Wang, Yanghe Zou, Xiang Liu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text