Google DeepMind's DiffusionGemma: Parallel Text Generation Hits 1,000 Tokens Per Second

Google DeepMind has released an experimental model that breaks from the standard autoregressive approach to text generation. DiffusionGemma produces output in parallel blocks rather than one token at a time, achieving up to 4x faster inference on GPUs - over 1,000 tokens per second on a single H100.

The trade-off is quality, which lags behind the standard Gemma 4. But for real-time applications where latency matters, this opens new possibilities. The model is open-weight under Apache 2.0, letting developers test locally.

Built on the Gemma 4 26B A4B Mixture-of-Experts architecture, DiffusionGemma has roughly 25.2 billion total parameters, with only 3.8 billion active during inference. That efficiency keeps memory usage at 18GB VRAM.

Instead of predicting the next token sequentially, DiffusionGemma starts with a canvas of random tokens and refines the entire block - typically 256 tokens - through a denoising process borrowed from image diffusion. Bidirectional attention lets the model see the full block at once, self-correcting errors on the fly. Early demos show clean markdown formatting without the drift common in streaming autoregressive output.

Performance numbers are strong for targeted workloads. NVIDIA optimizations push past 700 tokens per second on an RTX 5090, and over 1,100 on Hopper H100 hardware with FP8 precision. vLLM added native support immediately, and the inference engine handles this first discrete diffusion language model (dLLM) using shared Gemma 4 backbone weights for encoding and denoising.

Context length reaches 256K tokens, and multimodal inputs - text, images, video - can condition the output. Support spans over 140 languages. Developers are already experimenting with inline editing, long-document iteration, and non-linear text creation.

This model isn't a universal replacement. It excels in low-batch, interactive settings, not high-concurrency cloud serving. At scale, autoregressive models pack more queries onto the same hardware. DiffusionGemma's strength is local deployment and latency-sensitive tasks.

The release marks one of the first practical open implementations of diffusion for text. Questions remain about training stability and whether quality can match autoregressive leaders with further scaling. Expect hybrid approaches that mix autoregressive drafting with diffusion refinement. For now, it's a powerful tool for developers building fast, local, interactive AI.

Source: Webpronews