Ggml v1.0.0

Overview

ggml 1.0.0 prunes the deprecated public API and the legacy DIAG_MASK ops, adds configurable YaRN truncation and FP8 support, and brings broad performance work across backends: few-row quantized matmuls and MoE fusion on CUDA/Metal, a Vulkan prefill FlashAttention path, a device-side MoE scheduler on SYCL, decomposed Adreno FlashAttention in OpenCL, new WebGPU kernels, expanded Hexagon op coverage, and scheduler improvements that avoid uploading unused inputs.

API changes

• Pruned deprecated API and removed the legacy DIAG_MASK_INF/DIAG_MASK_ZERO ops, bumping the RPC protocol to reject incompatible peers (llama/29801, llama/31946)
• Added ggml_rope_set_truncate/ggml_rope_get_truncate to control YaRN correction-bound rounding (llama/30206, llama/30802)
• Implemented the backward pass for TRI (llama/30712)
• Added MUL_MAT src1 precision tests and gating for ggml_prec_set_acc (llama/31896, llama/31851)
• Bumped the RPC protocol to v9 with faster loading and graceful failure on unreachable endpoints or failed handshakes (llama/29413, llama/30415)

Core changes

• Scheduler skips uploading inputs that are not read and prefetches only inputs produced by earlier splits (llama/31419, llama/32100)
• MoE cache can compute a share of the cache misses on the CPU and copy the results back asynchronously (llama/30966)
• Meta backend raises the compute view limit for buffers with few tensors (llama/31447)

Backend changes

CUDA

• Added few-row MMQ for batch sizes 2-24 and MMVF for BF16 few-row matmuls (llama/31533, llama/31959)
• Added OCP FP8 support and fused NVFP4 gate/up matmuls with GLU in MMQ (llama/28413, llama/31296)
• Extended FlashAttention with an sm70 head-size-256 config and MMA head sizes 40/72 (llama/27997, llama/30285)
• Improved graph handling: skip compat checks on reuse, re-instantiate on cluster-dim changes, move graph state (llama/32004, llama/31284, llama/32001)
• MoE: skip empty MMQ stream-k tiles, linearize the MoE layout, and sanitize post-bias NaNs (llama/32067, llama/31047, llama/31005)
• Added a debug buffer pool and halved the VMM pool reservation with reserve retries (llama/31919, llama/30542)
• HIP: use a fixed BLAS workspace and add an MFMA lightning indexer for CDNA2 (llama/31838, llama/29050)
• Added a d_state=64 SSM_SCAN fast path and made radix Top-K deterministic for ties (llama/31818, llama/31084)

Metal

• Fused MoE routing and kept fusable node groups inside a single command buffer (llama/31615, llama/31688)
• Made radix Top-K deterministic for ties, added mul_mm for 32<=K<64, and switched TRI to select (llama/31084, llama/30982, llama/30711)
• Fixed GROUP_NORM for ne03>1 and out-of-bounds accesses in the FA output and SOLVE_TRI kernels (llama/31852, llama/31662, llama/32103)

Vulkan

• Added an Intel FlashAttention prefill kernel (llama/29357)
• Added batched tensor copies, a test-fusion interface, and ReBAR detection (llama/31418, llama/30769, llama/31658)
• Enabled dequantized KV with CM2, YaRN truncation, RDNA3/4 large-tile split-K, and large coopmat matmul on newer AMD drivers (llama/31526, llama/30802, llama/30280, llama/31587)
• Added a tiled-transpose concat shader and FWHT kernels for block widths above 512 (llama/30591, llama/29772)
• Fixed MoltenVK out-of-bounds accesses, GROUP_NORM workgroup limits, and FA scratch sizing (llama/31890, llama/31930, llama/30487)
• Capped scalar FlashAttention at 8 rows for large heads on Qualcomm (llama/30829)

SYCL

• Added a device-side MoE scheduler (llama/30156)
• Made the oneDNN FA cache thread-safe and un-gated fused SDPA for Alchemist at head_dim=64 (llama/31138, llama/31574)
• Cached the device id and reject buffers that would exceed VRAM (llama/31126, llama/32013, llama/31285)
• Added exact rounding for CPY to Q4_1/Q5_1/Q8_0, strided CLAMP, and f16 SWIGLU_OAI (llama/31465, llama/31784, llama/30857)
• Fixed AOT for allocations above 4GB (llama/31873)

OpenCL

• Added decomposed Adreno FlashAttention and improved decode performance (llama/32046, llama/31633)
• Added q6_k tiled, q8_0, and f32/q5_k multi-column GEMV kernels and f32 MUL_MAT_ID (llama/31758, llama/30947, llama/31783)
• Added arange, non-contiguous sigmoid, repeat for f16/i32/i16/bf16, and DSV4 HC pre/post kernels (llama/31829, llama/31687, llama/31782, llama/31623)
• Enabled multi-column GEMV by default for Adreno X2 (llama/31694)

WebGPU

• Added batched tensor API support and returned maxBufferSize from get_max_size (llama/31473, llama/31348)
• Added GROUP_NORM, MEAN, COUNT_EQUAL, and PAD_REFLECT_1D kernels (llama/31490, llama/31591, llama/31578, llama/31877)
• Supported the YaRN truncation config (llama/30802)

Hexagon

• Added Q5_0/Q5_1 support and optimized conv_2d while adding conv_3d (llama/32094, llama/31814)
• Fused post-norm residual add into RMS_NORM_MUL and added set_tensor_n (llama/31822, llama/31744)
• Added automatic libhtp loading and fixed 64-bit FastRPC handle truncation (llama/31781, llama/30952)
• Fixed the host build with clang-cl on Windows ARM64 (llama/32011)

CPU

• Switched MUL_MAT to F32 src1 and accumulate FlashAttention V in F32 (llama/31854, llama/29810)
• Added SpacemiT GATED_DELTA_NET RVV and Q4_1 32x256 HP kernels and enabled IME2 MXFP4 repack (llama/31897, llama/31929, llama/31776)
• Enabled the 4x4 simd_gemm tile on ARM NEON and added bf16/q8_0 VXE to llamafile (llama/32128, llama/31866)

CANN

• Fail early when a buffer type is unsupported (llama/31270)

ZDNN

• Accept non-zdnn tensors, invalidate compute tensors after each op, and fix release backend lib detection (llama/31824, llama/31905)

OpenVINO

• Supported the YaRN truncation config (llama/30802)

More info

• Releases and versioning of ggml-org projects

Changelog since v0.29.0

417a024 ggml : bump version to 1.0.0 (#1690)