Back to Projects

DiffuLLM — Training a Diffusion Language Model From Scratch

A from-scratch discrete diffusion language model trained on recipe corpora.

PyTorchTransformersD3PMDDPSLURM

The question

Autoregressive language models generate one token at a time, left to right. DiffuLLM asks a different question: can a language model learn text by denoising in parallel, starting from a fully masked sequence and iteratively filling in the blanks?

This was a from-scratch research build, not an API wrapper. The system was trained on recipe text because recipes have a strong, inspectable structure: a title, ingredients, and an ordered instruction sequence.

System design

DiffuLLM is a bidirectional Transformer trained with a D3PM-style discrete diffusion objective.

LayerChoice
BackboneBidirectional Transformer encoder
CorruptionAbsorbing [MASK] state
ObjectiveMasked-only cross-entropy over corrupted tokens
Timestep samplinglogit_normal, to emphasize mid-noise states
InferenceIterative confidence-based unmasking
TrainingPyTorch Distributed Data Parallel via torchrun, submitted to SLURM
CheckpointingValidation-loss checkpoints; the evaluated checkpoint is best_step503000.pt

The important design distinction is that the model does not receive a causal left-to-right history. It sees the whole partially masked sequence and learns to reconstruct missing tokens from bidirectional context.

Training and infrastructure

  • Trained for more than 500,000 iterations on the recipe corpus.
  • Distributed training through PyTorch DDP; cluster submission through SLURM.
  • A local smoke-test path verifies the data pipeline, tokenizer, and local inference before a cluster run.
  • Checkpointing and validation-loss monitoring were used to select the evaluated checkpoint.

The repository's comparison table also lists a 1.2M-step figure for DiffuLLM, while the evaluation section documents the selected 503k checkpoint. Those numbers have not been reconciled as the same run. This page therefore treats 503k as the evaluated checkpoint and does not use 1.2M as a headline result.

Evaluation

Evaluation focused on structured recipe completion: provide a title and ingredients, then generate the instruction sequence. The repository documents that short-to-medium generations are coherent and follow the recipe structure; generations at longer lengths are less reliable.

No numeric benchmark is asserted here. The evidence is the committed training/evaluation code, the selected checkpoint, the generated examples in the README, and the final report. The report covers the broader three-way comparison, including two SEDD systems.

Failure modes

Longer generations exposed real limits:

  • Repetition and cyclical instruction loops.
  • Malformed control-token output.
  • Topic drift as the sequence grows beyond a few hundred tokens.

The README attributes these to the difficulty of maintaining a global narrative state without a strict causal hidden state. That is a useful negative result: parallel denoising can model local structure while still struggling with long-range coherence.

My contribution and collaborators

I designed and built the DiffuLLM system: the architecture, absorbing-mask diffusion setup, masked reconstruction objective, training loop, DDP/SLURM workflow, checkpointing, and qualitative evaluation.

The comparative report is co-authored with Urvi Desai and Aparajita Sarkar. The SEDD-SE and SEDD-KL systems in the comparison were independently implemented by Urvi; the three systems share a study and report, not a single authorship claim.

Reproducibility and links

  • Source and report
  • Final report PDF
  • Dataset: B2111797/recipenlg-text-256
  • Setup: scripts/setup_ilab.sh
  • Cluster run: scripts/submit_ilab_ddp.slurm
  • Local smoke test: scripts/run_local_smoke.sh

Limits

This is a recipe-domain research system with qualitative evaluation. It is not a production language model, and it is not evidence that discrete diffusion beats autoregressive models across general text. The 1.2M-step documentation discrepancy should be resolved before that figure is used publicly.

Ask AIABOUT MY WORK