diffusiongemma

A small launcher + setup for running DiffusionGemma 26B-A4B locally on llama.cpp.

DiffusionGemma is Google DeepMind's block-diffusion variant of Gemma 4. Instead of generating one token at a time, it denoises a whole 256-token canvas in parallel over a handful of diffusion steps, then commits the block and moves to the next — so a reply arrives as a few block resolves rather than a left-to-right stream.

Heads up — this is CLI-only

DiffusionGemma runs through the dedicated llama-diffusion-cli runner from llama.cpp PR #24423, which at the time of writing is a draft proof-of-concept. Consequences:

• The standard llama-cli / llama-server cannot generate from it.
• There is no HTTP server, so it does not drop into llama-swap / OpenAI-compatible tooling. It's an interactive terminal chat (and one-shot -p) only.
• It's multimodal on paper, but this PR wires up text generation only — no vision yet.
• Code generation is its weak spot (Codeforces ~1429 / LiveCodeBench ~69%, below Gemma 4). It's great for chat and for watching the diffusion process; reach for a standard model for serious codegen.

Setup

Requires cmake, a C++ toolchain, the HuggingFace CLI (hf), and — for GPU — CUDA.

./setup.sh

This builds llama-diffusion-cli (PR #24423) and downloads the Q8_0 GGUF (~27G). Useful env knobs:

# Reuse an existing llama.cpp checkout via an isolated git worktree
# (keeps your current build/serving untouched):
LLAMA_SRC=~/Documents/llama.cpp ./setup.sh

# Smaller quant, CPU-only build, custom locations:
QUANT='*Q4_K_M*' CUDA=OFF BUILD_DIR=~/llama-diffusion MODEL_DIR=~/models/dg ./setup.sh

Quant	Size	Notes
Q8_0	~27G	near-lossless, recommended (spans 2× 24G GPUs)
Q6_K	~21G	fits a single 24G GPU
Q4_K_M	~17G	smallest, fits a single 24G GPU

Usage

Put bin/diffusiongemma on your PATH (e.g. symlink into ~/.local/bin):

ln -s "$PWD/bin/diffusiongemma" ~/.local/bin/diffusiongemma

diffusiongemma                 # Q8, conversation mode, live canvas denoise
diffusiongemma q4              # smaller/faster quant
diffusiongemma --list          # installed quants + sizes
diffusiongemma -h              # help
diffusiongemma q8 -- -n 4096   # pass extra flags to llama-diffusion-cli

-ngl 99 offloads all layers; with multiple GPUs visible the model splits across them automatically (Q8 needs two 24G cards). --diffusion-visual (on by default) renders each canvas denoising in place. stderr is routed to ~/.local/state/diffusiongemma.log so the canvas stays clean.

Path overrides: DG_MODEL_DIR, DG_BIN, DG_LOG.

Useful diffusion flags

Passed through after --:

• -n N — target tokens; derives the block count and grows batch/context to fit.
• --diffusion-visual — live in-place canvas view.
• --diffusion-kv-cache {auto,on,off} — prompt-prefix KV cache (auto = on for single GPU).
• Entropy-bound sampler (default): --diffusion-eb-max-steps (48), --diffusion-eb-t-max/--diffusion-eb-t-min (0.8→0.4), --diffusion-eb-entropy-bound (0.1), --diffusion-eb-confidence (0.005).

License

Apache-2.0 (matching the model and llama.cpp). See LICENSE.

The model weights are © Google DeepMind, released under Apache-2.0; the GGUF quants are by Unsloth. This repo only contains the launcher/setup glue.