DiffuLLM — Training a Diffusion Language Model From Scratch
A from-scratch discrete diffusion language model, trained and evaluated as an alternative to left-to-right text generation.
Designed and built the DiffuLLM system: a bidirectional Transformer, D3PM-style absorbing-mask corruption, masked reconstruction, iterative unmasking, DDP/SLURM training, checkpointing, and qualitative evaluation. The comparative study also includes two independently built SEDD systems by collaborators.
Model training from scratch, distributed GPU infrastructure, and failure-aware evaluation.
ENGINEERING BREAKDOWNPROBLEM · METHOD · EVIDENCE · LIMITS
Autoregressive generation is the default for text. Can a bidirectional model learn language by denoising masked tokens in parallel?
Absorbing [MASK] corruption, logit-normal timestep sampling, masked-only cross-entropy, and confidence-based unmasking from a fully masked sequence.
The selected checkpoint (best_step503000.pt) was evaluated on structured recipe completion. Short and medium generations stay coherent; longer outputs show repetition, malformed control tokens, and drift. The README's comparison table also references a 1.2M-step run, so the page treats 503k as the evaluated checkpoint until that discrepancy is reconciled.
A recipe-domain research study with qualitative evaluation, not a broad benchmark or a production model. The 1.2M-step figure is not used as a headline result.
