An arXiv preprint dated 20 Aug 2026 reports a method that turns computer-use traces — screenshots and mouse or keyboard events — into structured task models. Rather than treating a session as one uninterrupted workflow, Task Model Induction, or TMI, tries to identify the separate tasks underway and link each task’s objective with the procedure used to pursue it.
From events to tasks
TMI grounds events in visual context, segments them into semantic actions and larger activities, infers latent tasks, and reconciles objective and procedure models into a single task model. In plain terms, it tries to keep track of both the goal of a piece of work and the steps that belong to it when several activities are interleaved.
All pipeline stages used gpt-5.4 at temperature 1.0. The authors used gpt-5.5 and claude-sonnet-5 as intrinsic judges, while gpt-5-mini generated the reusable skills and executed held-out tasks.
A strong result in synthetic mixtures
For a robustness test, the authors built synthetic interleavings from recorded sessions, varying the number of mixed tasks from 2 to 15 and the interleaving difficulty from 2 to 4. Three trajectories were used for each condition, giving 126 synthetic trajectories overall.
Across those conditions, the adjusted Rand index (ARI) — a measure of agreement between the recovered grouping and the known task partition — was 0.974 ± 0.028. The mean absolute error in the predicted task count was 0.48 ± 0.54. ARI ranged from 0.980 ± 0.024 at difficulty 2 to 0.968 ± 0.029 at difficulty 4.
The paper reports these figures as means with standard deviations; it does not report confidence intervals or significance tests for the comparison.
Task identities mostly lined up
The HumanWork evaluation comprised 38 recorded sessions spanning 15 tasks across five professional domains, totaling 42.8 hours and 48.7K raw events. SkillsBench contained 86 software-engineering tasks across three agent harnesses and three skill conditions. For this analysis, the authors selected the 15 tasks with the highest average action count, yielding 195 runs.
When the induced task sets were compared with the assigned tasks, they matched task identities in 94.74% of the 38 HumanWork runs under gpt-5.5. Among 74 successful SkillsBench runs, the match rate was 93.24%.
The picture changed with the judge model: claude-sonnet-5 reported 89.47% for HumanWork and 98.65% for SkillsBench. The SkillsBench identity analysis included successful runs only and relied on language-model judges, so the figures apply to that evaluation setup rather than every benchmark run.
The generated procedures scored better than a cited baseline
The researchers also compared how accurately the task model described the work with a cited baseline. Under the gpt-5.5 judge, step-description accuracy was 74.9% for TMI versus 30.3% for the baseline, while operator correctness was 88.5% versus 52.7%. The same ordering was reported under claude-sonnet-5.
A smaller test of skill transfer
The downstream test used 20 SkillLearnBench task families. For each family, one successful demonstration was used to induce a task model, which was then deployed as a reusable skill on held-out instances from the same family.
Skills built from TMI scored 67.65% on executability and 18.57% on held-out task accuracy, compared with 59.35% and 14.29% for workflow summaries. The authors described the accuracy difference as a 30% relative gain.
The benchmark comparison has no reported confidence interval or significance test.
What remains unresolved
Repeated induction was similar but not identical. Across three independent pipeline runs, task-identity match averaged 91.2 ± 4.0 under gpt-5.5 and 90.4 ± 4.0 under claude-sonnet-5.
Privacy is another unresolved issue. The traces may contain personally identifiable information, so the authors recommend redacting screenshots and keyboard events before induction. They do not report how that redaction would affect induction quality. The work used three publicly available research datasets and reports no new human-subject data collection or private user-data access for benchmark evaluation.
The robustness test used synthetic interleavings rather than direct observations of naturally concurrent work. The SkillsBench analysis also focused on 15 tasks chosen for their highest average action count from 86 base tasks, leaving open how the method would perform on broader, unselected traces.
Paper data and sources
Original title: Inducing Task Models from Computer-Use Traces
Authors: Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text