gguf_alg_ext_acc.md

We use lm-eval for evaluation. For LLaMA, we enabled add_bos_token and removed @use_kernel_forward_from_hub("RMSNorm") in modeling_llama.py to stabilize accuracy during evaluation. All other settings follow the default configurations of AutoRound and lm-eval.

Results of V>0.12.3

lm-eval 0.4.9.1

transformers 4.57.6

1. Qwen3-8B 评测结果

Model / Quant	Average	arc_challenge	arc_easy	boolq	hellaswag	lambada_openai	mmlu	openbookqa	piqa	truthfulqa_mc1	winogrande
bf16	0.6329	0.5563	0.8354	0.8670	0.5717	0.6439	0.7292	0.3160	0.7671	0.3647	0.6780
opt-rtn (2bit)	0.5879	0.4753	0.7904	0.8431	0.4901	0.6352	0.6559	0.2840	0.7323	0.3133	0.6598
q2ks	0.6060	0.5111	0.8026	0.8462	0.5107	0.6266	0.6736	0.3160	0.7492	0.3329	0.6906
opt-rtn (3bit)	0.6200	0.5486	0.8152	0.8550	0.5500	0.6247	0.7030	0.3160	0.7606	0.3611	0.6654
q3ks	0.6287	0.5580	0.8316	0.8566	0.5516	0.6385	0.7090	0.3300	0.7661	0.3488	0.6969
opt-rtn (4bit)	0.6303	0.5469	0.8308	0.8645	0.5647	0.6454	0.7255	0.3120	0.7639	0.3709	0.6788
q4km	0.6353	0.5606	0.8388	0.8673	0.5695	0.6392	0.7285	0.3180	0.7677	0.3770	0.6867

---

2. Qwen2.5-7B-Instruct 评测结果

Model / Quant	Average	arc_challenge	arc_easy	boolq	hellaswag	lambada_openai	mmlu	openbookqa	piqa	truthfulqa_mc1	winogrande
BF16	0.6574	0.5282	0.8144	0.8642	0.6201	0.6953	0.7161	0.3480	0.7938	0.4823	0.7119
opt-rtn (2bit)	0.6258	0.5094	0.7992	0.8612	0.5490	0.6994	0.6557	0.3220	0.7633	0.4015	0.6977
q2ks	0.6388	0.5205	0.8127	0.8636	0.5652	0.6975	0.6741	0.3300	0.7688	0.4431	0.7127
q3ks-opt-rtn	0.6380	0.5009	0.7870	0.8569	0.5964	0.7054	0.6908	0.3400	0.7688	0.4578	0.6756
q3ks	0.6482	0.5358	0.8123	0.8575	0.6024	0.6986	0.7029	0.3340	0.7769	0.4663	0.6953
opt-rtn (4bit)	0.6574	0.5435	0.8241	0.8609	0.6177	0.6897	0.7121	0.3440	0.7976	0.4761	0.7080
q4km	0.6585	0.5478	0.8190	0.8630	0.6204	0.7004	0.7141	0.3420	0.7916	0.4774	0.7088

---

3. Llama-3.1-8B 评测结果

Model / Quant	Average	arc_challenge	arc_easy	boolq	hellaswag	lambada_openai	mmlu	openbookqa	piqa	truthfulqa_mc1	winogrande
BF16 (Base 1)	0.6235	0.5017	0.7997	0.8095	0.6017	0.7568	0.6216	0.3460	0.7960	0.2705	0.7316
BF16 (Base 2)	0.6295	0.5154	0.8165	0.8217	0.6007	0.7535	0.6323	0.3340	0.7998	0.2864	0.7348
opt-rtn (2bit)	0.5575	0.4121	0.7487	0.7807	0.5167	0.6699	0.4762	0.2860	0.7546	0.2338	0.6961
q2ks	0.5849	0.4522	0.7740	0.7844	0.5415	0.7083	0.5527	0.3080	0.7677	0.2632	0.6969
q3ks-opt-rtn	0.6054	0.4573	0.7845	0.8000	0.5779	0.7526	0.5929	0.3300	0.7878	0.2534	0.7174
q3ks	0.6165	0.4889	0.8098	0.8116	0.5858	0.7396	0.6216	0.3300	0.7873	0.2681	0.7222
opt-rtn (4bit)	0.6283	0.5247	0.8178	0.8193	0.6017	0.7398	0.6342	0.3320	0.7905	0.2815	0.7411
q4km	0.6279	0.5179	0.8190	0.8174	0.6025	0.7436	0.6352	0.3360	0.7943	0.2766	0.7364

Results of V<=0.12.3

Average accuracy across lambada_openai, hellaswag, piqa, winogrande, truthfulqa_mc1, openbookqa, boolq, arc_easy, arc_challenge and mmlu.

method	scheme	Llama-3.1-8B	Qwen2.5-7B-Instruct	Qwen3-8b	Qwen3-30B-A3B-Instruct-2507
BF16	-	0.6295(100%)	0.6571(100%)	0.6322(100%)	0.6746(100%)
Optimized RTN	q2_k_s	0.5535(87.92%)	0.6266(95.35%)	0.5901(93.35%)	0.6386(94.66%)
AutoRound+alg_ext	q2_k_s	0.5740(91.18%)	0.6349(96.62%)	0.5962(94.31%)	0.6460(95.77%)
Optimized RTN	q3_k_s	0.6040(95.95%)	0.6382(97.12%)	0.6128(96.94%)	0.6598(97.82%)
AutoRound+alg_ext	q3_k_s	0.6081(96.59%)	0.6503(98.97%)	0.6252(98.89%)	0.6622(98.17%)
Optimized RTN	q3_k_m	0.6083(96.63%)	0.6418(97.68%)	0.6194(97.97%)
AutoRound+alg_ext	q3_k_m	0.6127(97.33%)	0.6533(99.42%)	0.6197(98.02%)
Optimized RTN	q4_k_s	0.6228(98.94%)	0.6560(99.83%)	0.6303(99.70%)	0.6762(100.24%)
AutoRound+alg_ext	q4_k_s	0.6239(99.11%)	0.6605(100.51%)	0.6320(99.98%)	0.6777(100.46%)
Optimized RTN	q4_k_m	0.6252(99.32%)	0.6558(99.80%)	0.6296(99.59%)
AutoRound+alg_ext	q4_k_m	0.6257(99.40%)	0.6575(100.06%)	0.6340(100.29%)

Time cost

model	Optimized RTN	AutoRound+alg_ext
Llama-3.1-8B	1m25s	29m43s
Qwen2.5-7B-Instruct	1m20s	35m35s
Qwen3-8b	1m29s	47m58s
Qwen3-30B-A3B-Instruct-2507	25m12s	12h47m39s