A preprint describes a texture-space diffusion system for reconstructing and generating physically based material maps on known 3D objects from posed views, text prompts or low-resolution maps. With reference geometry, it reported the best listed results in synthetic reconstruction and relighting tests. But on real examples under the original lighting, a competing method performed slightly better.
The system treats the task as texture completion. It uses world-space positions, the direction each surface patch faces, and text as conditions, then outputs maps for base color, height, roughness and metallicity. Because the completion is carried out in texture space, the paper is testing whether one object-linked material can remain consistent as the view changes, rather than creating a separate answer for each image.
For training, the implementation fine-tuned all of its diffusion-transformer layers for 15,000 iterations on 32 A100 GPUs. It accumulated gradients over four steps for an effective batch size of 128. The training set contained 120,000 object-centric videos. Each object had 17 frames at 2,048 by 2,048 pixels, rendered with a path tracer using three bounces.
Synthetic results lead, but real data are mixed
The clearest gains appeared in the synthetic tests. The main multi-view evaluation averaged results across 32 extracted materials, using eight views for each material and eight HDR probes for relighting. With reference geometry, reconstruction reached 31.24 PSNR, 0.960 SSIM and 0.0370 LPIPS. Relighting reached 30.46 PSNR, 0.950 SSIM and 0.0401 LPIPS. PSNR and SSIM are read as better when higher, while LPIPS is better when lower. The authors reported the system as the top performer on all the synthetic metrics. These are averages, and the paper reports no confidence intervals or significance tests.
The real-data comparison was more modest. Across eight DTC examples under the original lighting, DiffPT scored 30.67 versus 29.03 PSNR, 0.959 versus 0.947 SSIM, and 0.0307 versus 0.0331 LPIPS for the proposed system. Using LSRM geometry in the multi-illumination test, the proposed system improved slightly over LSRM on all three measures. The small real comparison means the synthetic lead does not carry across every setting.
Material generation from less information was also tested. In the single-view test, the system had CLIP-FID 1.520, CMMD 0.0081 and LPIPS 0.0325. In the text-to-material test, the corresponding values were 3.338, 0.0230 and 0.0542. Those were the lowest listed values for every metric in the two tests. The analysis reports no repeated-run variability or significance testing for this comparison.
More views and higher resolution come with a cost
The system also includes an inference-scaling procedure. It progressively rolls noise from a 2K result to an 8K result and, when many views are available, keeps the top two expert predictions for each texel, or small texture element, then averages them using coverage weights. Across the first four real-dataset samples, moving from 17 views at 2K to 117 views at 8K left the reported metrics close: 28.42 dB, 0.919 and 0.0668 versus 28.34 dB, 0.917 and 0.0639 for PSNR, SSIM and LPIPS. PSNR and SSIM dipped slightly, while LPIPS improved.
In a supplementary synthetic g-buffer evaluation, which checks rendered material channels, the system had the highest listed scores across 32 scenes and 17 views: 29.19 siPSNR for base color, 27.00 base-color PSNR, 24.32 roughness PSNR and 18.24 metallicity PSNR. The authors present this as a g-buffer comparison, not a direct physical measurement of material accuracy.
The preprint's limitations are practical as well as technical. It says specular appearance remains difficult, and that its 16 by 16 attention patches can leave overlapping texture-atlas segments indistinguishable. Suggested next steps include better data curation, patch-aware texture unwrapping and pixel-space diffusion.
Runtime is another obstacle. The implementation is described as not optimized and took 130 seconds for the 17-view model at 2K on a GB300 GPU. The paper points to acceleration and distillation as ways to reduce the cost.
Taken together, the preprint makes the strongest case for texture-space material generation when the object geometry is known and synthetic evaluation is the main test. It also reports that the approach can preserve similar scores as resolution and view count rise. But the real-data results are mixed, the tested real set is small, and the method is not yet cheap to run.
Paper data and sources
Original title: Texture Space Material Diffusion
Authors: Jacob Munkberg, Peter Kocsis, Jon Hasselgren
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text