AVX-512 instructions enable denser packing and more efficient lookups.

[tommyyliu]

I have an implementation put together here: https://github.com/tommyyliu/lut_mm/blob/master/src/ternary_mm_avx512.cpp

The table has all the dot products with 5 activations instead of just 3. This means that each lookup handles 5 instead of 3, and you can see pretty substantial speedup.

[shifulegend]

Interesting approach - the 5-activation LUT is a clever way to amortize the table lookup overhead.

One alternative that's worked well in our implementation (Project Zero, https://github.com/shifulegend/project-zero) is combining VBMI + VNNI rather than a pure LUT path:

1. vpermb replicates each packed byte 4x across a 512-bit register (one instruction to expand 16 packed bytes to 64 weight-encoding bytes)
2. vpmultishiftqb extracts the individual 2-bit codes per-slot
3. The resulting int8 weight encodings feed directly into VNNI dot products

The VBMI 3-instruction unpack vs SSE fallback is about 2.7x in our microbenchmarks. Combined with VNNI and PGO/LTO we get 36.25 tok/s on BitNet b1.58-2B-4T (4-core Xeon Emerald Rapids): https://openbenchmarking.org/result/2606207-SHIF-PROJECT42

The main trade-off with VBMI+VNNI vs a LUT path: VBMI requires Ice Lake or newer (no Cascade Lake VNNI support), while a LUT can work on any AVX-512F machine. The 5-activation LUT you're describing sidesteps the VNNI requirement entirely, which is a meaningful portability win for older AVX-512 hardware.

Have you benchmarked the LUT path on hardware without VNNI? Curious whether the 5-activation grouping gives better throughput than VBMI+VNNI on Cascade Lake specifically.

[tommyyliu]

Threads	VNNI (Project Zero)	LUT (lut_mm)	Speedup
1	34.15	42.66	1.25×
2	47.43	51.11	1.08×
4	48.09	51.93	1.08×
8	48.40	52.13	1.08×
16	52.96	56.66	1.07×

tok/s microsoft/bitnet-b1.58-2B-4T
on my AMD 9950x

I had Claude run Project Zero with VNNI on my PC. I then had it plug in my 5 activation LUT method into project zero. The denser weight format might be useful to play with. My old PC only has a Intel 6700k from like 10 years ago, so I can't test on Intel right now. Maybe, you can try the 5 bit lookup table on your hardware to see if it beats the VNNI method?

From what I understand, Intel and AMD might have pretty different performance characteristics for advanced instructions.

[shifulegend]

The Intel vs AMD VNNI throughput gap is probably what's driving this.

Intel Sapphire/Emerald Rapids runs VPDPBUSD 512-bit at roughly twice the throughput of AMD Zen 4 - two execution ports vs one per cycle (uops.info has the breakdown). Zen 5 (your 9950X) I haven't confirmed, but if VNNI-heavy code is hitting a compute ceiling on AMD, the same code on our Xeon would have a higher ceiling.

At 1 thread the LUT wins by 1.25x on your hardware. If AMD's lower VNNI throughput is artificially capping our VNNI path, the comparison on Emerald Rapids might look different - that's the hypothesis anyway, worth testing rather than assuming.

Your 8 to 16 thread numbers are interesting: 4.5 tok/s gain (52.13 to 56.66). That's memory bandwidth pressure, not compute - our 4-core Xeon shows the same step pattern.

I'll pull ternary_mm_avx512.cpp and run the 5-activation LUT against our VNNI kernel on the Xeon. Is that HEAD or is there a branch? I want bitnet-b1.58-2B-4T for the model so numbers line up.

[shifulegend]

Yes, LUT wins on Intel too.

Ran lut_mm's bench.exe on a Xeon @ 2.80 GHz (avx512f/bw/vnni/dq/vl, 4 cores, GCC 13.3.0, Linux) before doing the full integration into PZ. Default shape M=256, K=2080, N=2048:

threads	impl	Gop/s	vs VNNI
1	dense_mm_vnni	132.6	1.00x
1	lut_mm_avx512	153.7	1.16x
2	lut_mm_avx512_mt	281.7	2.12x
4	lut_mm_avx512_mt	573.9	4.33x

1.16x at T=1, vs your 1.25x on Zen 5. The margin being slightly lower on Intel is kind of interesting actually - it doesn't line up with my earlier hypothesis about dual-ported VNNI being a bigger ceiling. Might be that this Xeon's vpermt2w characteristics put it closer to AMD than I expected, or just VM/frequency effects.

These are kernel Gop/s, not tok/s, so not directly comparable to your inference numbers. Multi-thread scaling is basically linear (3.73x on 4 cores at T=4), consistent with the compute-bound claim in the README.

Side note: README flags 'this GCC/Clang branch has never been built' - compiled fine on first try, might want to update that.

[tommyyliu]

Nice to see it holds up on Intel also!

Do note that the VNNI implementation in my repo was just something Claude threw together for comparison's sake. This is from before I was aware of your repo.

[shifulegend]

Ah, good to know. Our Xeon showed 1.16x anyway - floor not ceiling for what LUT can do vs a properly optimized VNNI path.

But the number I'm actually curious about: when Claude ran inference with your LUT patch in PZ, what tok/s came out? Kernel Gop/s is useful but pipeline overhead, activation quant path, batch shaping - all of that eats into the margin, and end-to-end inference tok/s is the only number that actually matters for someone trying to decide which path to use.

[tommyyliu]

My table earlier was tok/s running bitnet.

[shifulegend]

Ah, missed that - so your table was actual inference tok/s on the 9950X. VNNI (PZ) 34.15 → LUT 42.66 at T=1, 1.25x end-to-end. More useful than kernel Gop/s.

K-6 is integrated on our side now. The situation on our Xeon 8259CL (AVX-512BW, no VNNI) is different from the 1.16x we reported on that cloud Xeon with VNNI:

kernel	Gop/s	vs float32
AVX-512F float32	2.5	1.00x
LUT AVX-512BW	74.6	29.3x

On hardware without VNNI, float32 was the ternary kernel ceiling before K-6. 29.3x kernel vs 1.16x on VNNI hardware - different situations entirely.

End-to-end tok/s pending - blocked on model availability on this machine (disk). Will update once we have it.