Introspective Diffusion Language Models

Decoding paradigm comparison. I-DLM is a drop-in replacement within AR serving infrastructure.

Throughput-latency tradeoff compared with DLMs across batch sizes (1, 4, 16, 64). I-DLM delivers 2.9-4.1x higher throughput than LLaDA-2.1-mini and SDAR at C=64.

I-DLM acceptance rate (p):0.90

R-ISD LoRA overhead (α):1.12

Gated LoRA adds compute at MASK positions for bit-for-bit lossless output. α=1.12 matches empirical overhead.

Memory-bound: Speedup ≈ TPF = (2+p+...+pN-2) / (2-pN-1)

How do DLMs perform as they approach compute-bound?

At high concurrency, forward pass latency scales with query count per forward. We can measure compute efficiency as TPF²/query_size — how much useful output each FLOP produces relative to AR (efficiency = 1):

• SDAR (N=4, p=0.5): TPF ≈ 1.1, processes N=4 queries/forward → compute efficiency = 1.1²/4 ≈ 0.31. Each FLOP produces only 31% as much output as AR. This pushes SDAR into compute-bound early, and its throughput plateaus (batching efficiency slope = 84, see motivation figure).
• I-DLM (N=4, p=0.9): TPF ≈ 2.9, processes 2N−1=7 queries/forward → compute efficiency = 2.9²/7 ≈ 1.22. Each FLOP produces more useful output than AR — I-DLM stays in the memory-bound regime at concurrency levels where SDAR is already saturated (batching efficiency slope = 549).

Efficiency > 1 means parallel decoding actually saves total compute vs. AR. This is why I-DLM's throughput scales with concurrency while SDAR and LLaDA plateau in the throughput figure above.

Installation

git clone https://github.com/Introspective-Diffusion/I-DLM.git
cd I-DLM/inference
bash install.sh

See inference/README.md for detailed environment setup.

Quick Start

1. Launch server:

python -m sglang.launch_server \
    --model-path yifanyu/I-DLM-8B \
    --trust-remote-code --tp-size 1 --dtype bfloat16 \
    --mem-fraction-static 0.85 --max-running-requests 32 \
    --attention-backend flashinfer --dllm-algorithm IDLMBlockN \
    --dllm-algorithm-config inference/configs/idlm_blockN4_config.yaml \
    --port 30000

2. Generate:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Prove that sqrt(2) is irrational."}],
    "max_tokens": 4096, "temperature": 1.0
  }'

Training

Convert a pretrained AR model into I-DLM via introspective-consistency training:

• Input: Concatenate fully-masked sequence with clean sequence [x_t | x_0]
• Attention: Strict causal masking across all positions
• Loss: Auto-balanced CE on both masked and clean positions
• Data: 4.5B tokens, 8 H100 GPUs, 2 epochs with stride curriculum (N=2 then N=3)

See training/README.md for scripts and configs.

Inference & ISD

Introspective Strided Decoding (ISD) generates and verifies in a single forward pass:

• MASK positions: Propose new tokens (distribution q)
• Clean positions: Verify prior tokens (anchor distribution p)
• Acceptance: min(1, p(x)/q(x)) guarantees AR-distribution output
• Stride N=4: TPF=2.96, ~3x wall-clock speedup in memory-bound regime

See inference/README.md for algorithm configs.

Serving (SGLang)

I-DLM uses strict causal attention, enabling direct integration into SGLang with no custom infrastructure:

• Paged KV cache and continuous batching
• CUDA graph capture (+42-76% throughput)
• Stationary-batch decode-loop scheduling (+11-21%)
• Argmax proposals (+11-15%)
• Paged-only attention kernel (+10-14%)

Full system achieves 2.1-2.5x throughput over naive baseline.

Lossless R-ISD

Residual ISD (R-ISD) adds a gated LoRA adapter for bit-for-bit lossless acceleration:

• LoRA active only at MASK positions; verify positions use base-only weights
• Output is identical to the base AR model by construction
• LoRA rank=128, overhead factor ~1.12x
• Model: yifanyu/I-DLM-8B-lora-r128

Model Zoo

All models use trust_remote_code=True (custom SDARForCausalLM architecture).

Benchmarks

We evaluate on 15 benchmarks across 4 categories with thinking mode enabled:

• Knowledge: ARC-C, MMLU, MMLU-Pro, GPQA-D, GPQA
• Math: GSM8K, MATH-500, MathBench, AIME-24, AIME-25
• Code: HumanEval, MBPP, LiveCodeBench-v6
• Instruction: IFEval

See inference/eval/ for reproduction scripts.