A new preprint has assembled a broad map of the visualization-design skills that instructors want students to learn, then turned that map into a 14-question multiple-choice assessment. The tool is meant to sample common design skills across a field that includes deciding how information should be shown, judging whether a visual explanation works, and creating visualizations.
The paper does not show whether students have mastered those skills or whether the questions measure them accurately. Its evidence comes from instructors’ stated course objectives and feedback during development, making the proposed inventory an early framework that still requires empirical validation.
A framework built from classroom objectives
The researchers began with course syllabi rather than student test scores. They invited 114 instructors, including 27 from private universities and 87 from public universities. Of those invited, 35 consented to have their syllabi included.
The syllabus analysis covered 38 syllabi from 35 instructors. Seven were submitted by email and 31 through a survey. The courses included 17 undergraduate courses, 10 graduate courses and 11 mixed undergraduate/graduate courses; three instructors supplied syllabi for two courses each.
The team extracted learning objectives, coded them and repeatedly grouped related objectives into clusters. Ten instructors also took part in semi-structured interviews, whose feedback was woven into the development and refinement of the inventory.
The coding was carried out independently by members of the research team before disagreements were discussed and resolved manually. The reported Cohen’s kappa score was 0.92, a measure of how closely the coders agreed when classifying the objectives.
Interview feedback led to major changes early in the process and smaller refinements later. The authors interpret that shift as near saturation, meaning that later interviews appeared to produce fewer major changes, although the study did not report a formal saturation metric.
Three areas made the final inventory
The final hierarchy contained seven top-level clusters and 15 subclusters. It was intended to organize a wide range of visualization-design principles and tasks into a structure that could guide both teaching and assessment.
Three broad areas remained in the final inventory: critiquing and evaluating design, understanding visualization-design principles, and designing visualizations. Together, they define the paper’s central view of visualization design as something students should be able to examine, explain and put into practice.
Developing visualizations was left out of the final scope after instructors treated programming as separate from visualization design. The paper therefore does not present the inventory as a measure of programming, development or related technical work.
The authors chose a breadth-first strategy. Instead of trying to test every observed skill in depth, the assessment uses cornerstone questions—representative examples intended to sample key skills across the hierarchy.
That choice gives the proposed tool a wide reach, but it also sets a clear boundary: performance on the questions would not, by itself, provide an in-depth picture of every individual design ability represented in the framework.
A broader lens than a single literacy test
The paper compares its proposed inventory qualitatively with existing visualization-literacy assessments. The review found substantial overlap, alongside visualization tasks that were unique to the new inventory.
The paper describes the additional tasks as a way to cover design skills that may receive less attention in existing assessments. This was a coverage review, not a head-to-head study of how people performed on different tests.
The result is a tool aimed at breadth rather than a finished diagnostic instrument. It is designed to help represent the range of concepts instructors emphasize, not to claim that a single short test can capture the full process of constructing a visualization.
What the study still cannot establish
Because the project analyzed syllabi and instructor feedback, it does not show that students possess the identified skills, learn them in a particular way or improve after taking a course. It also does not support causal claims about teaching methods or course effectiveness.
The inventory has not yet been empirically validated with students. The study therefore does not establish its measurement validity, diagnostic accuracy, reliability for student responses or sensitivity to instruction.
The multiple-choice format is another constraint. The authors note that questions of this kind may not adequately assess visualization construction or other performance-based abilities, including the ability to build a new visualization.
The framework also reflects the setting from which it was developed: instructors in North American and English-language contexts. The authors identify that restriction as a limitation.
Several areas were partly or wholly outside the final inventory’s scope, including programming, development, interactivity, domain-specific visualization, data transformation and visualization research. The paper therefore leaves open how design should be assessed alongside programming and development.
The researchers also acknowledge possible bias from personal connections between some participants and members of the research team, as well as the influence of pre-existing coding themes. Those disclosures are part of the context in which its clusters were produced.
The next test is with students
The authors’ next phase is planned to involve student think-aloud interviews and questionnaire responses. Those studies are intended to examine how students understand the questions and whether the items function as intended.
Future analysis may also examine item quality and construct validity—the extent to which the questions measure the skills they are meant to represent—using item-response methods and comparisons with external assessments.
The authors identify several other open tasks: translating and revalidating the inventory for languages and populations outside North America, deciding how to combine multiple-choice questions with construction tasks, and developing reusable templates or automated ways to generate new questions.
Until that work is done, the inventory is best understood as a proposed map of visualization-design education. It can offer educators and assessment researchers a shared starting point, but the study does not yet show how well the map predicts what students know or can do.
The paper’s status and materials
The document is a version 1 preprint posted on arXiv on 20 August 2026.
Supplemental materials and instructions for accessing the concept inventory are available on OSF. The supplied analysis does not report whether the underlying coded objectives, interview data or other study data are available.
The work was supported in part by the National Science Foundation through Awards #2402718, #2514565, #2141506 and #2313998.
The study’s survey and interview process was reviewed by the University of Washington institutional review board and determined to be exempt.
Paper data and sources
Original title: What Do Visualization Instructors Want Students to Learn? Introducing a Concept Inventory for Visualization Design
Authors: Medina Lamkin, Heer Patel, Sayamindu Dasgupta, Leilani Battle
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text