Zyphra unveils zyfa: fp8 attention kernels achieve significant speedups on amd mi355x

Zyphra has introduced ZYFA, an open-source set of FP8 and BF16 prefill attention kernels designed for AMD Instinct MI355X GPUs. This engine, which powers the Zyphra Optimization Engine (ZOE) used in Zyphra Cloud, demonstrates substantial performance gains over AMD's default AITER library. On real model activations, ZYFA's FP8 kernels show speedups ranging from 1.16x to 1.76x, with error rates being 62-78% lower than AITER at a head dimension of 128. ZYFA employs three key optimizations: an e4m3 bit pattern, a lean P implementation for softmax that reduces vector cycles from 896 to 384, and a scaling technique leveraging CDNA4's matrix instructions. While the MI355X clock speed can vary based on operand bit flips, ZYFA still requires fewer total cycles than AITER, despite running at a lower clock (1,628 MHz vs 2,108 MHz for Mistral-7B). Performance benchmarks show ZYFA reaching 2,672 TFLOP/s on Gaussian benchmarks at 32K, compared to AITER's 1,944 TFLOP/s. Furthermore, ZYFA achieves superior energy efficiency, delivering 1.94 TFLOP/J on captured Mistral-7B layers versus AITER's 1.60 TFLOP/J. The FP8 error relative to fp64 attention is notably low, with ZYFA showing 0.22x AITER's error on Mistral-7B.