AygeaandClaude 3507d33006 Grade 12-model backlog + fix native-API TTFT capture ...

Grading (12 new entries, 18→30 total in benchmark_history.json):
- KAT-Coder v2.5 Dev XL: lfu 49 / tts 52 / webhook 72 / rust 36 / automation 60
- Qwen3 Coder 30B: lfu 45 / tts 44 / webhook 62 / rust 54 / automation 58
- Qwen 3.6 35B-A3B uncensored (automation): 46
- Gemma 4 26B-A4B (data): 86 [tests pass]
All "coder" models scored Critical Bugs across prompts — plausible-looking
async code with fatal bugs (broken LFU eviction, in-flight cancel no-op,
submit() raising instead of backpressuring, un-awaited async read-through).

grade_run.py: switch from OpenAI-compat /v1/chat/completions (empty stats)
to native /api/v1/chat — returns full stats incl. time_to_first_token_seconds.
Verified on Gemma-26B (53.5 t/s, ttft 1.13). Two native-API gotchas handled:
input (string) not messages; max_output_tokens not max_tokens (that 400s).

New capture: outputs/gemma-4-26b-a4b-data.py (native-API run).

Co-Authored-By: Claude <noreply@anthropic.com>

2026-07-28 21:17:08 -07:00

..

.last_response_schema.json

Grade 12-model backlog + fix native-API TTFT capture

2026-07-28 21:17:08 -07:00

gemma4-26b-a4b-8bit-mlx-automation.py

Gemma 4 26B-A4B head-to-head vs Qwen (4 prompts, via grade_run.py API)

2026-07-28 18:42:58 -07:00

gemma4-26b-a4b-8bit-mlx-rust.rs

Gemma 4 26B-A4B head-to-head vs Qwen (4 prompts, via grade_run.py API)

2026-07-28 18:42:58 -07:00

gemma4-26b-a4b-8bit-mlx-tts.py

Gemma 4 26B-A4B head-to-head vs Qwen (4 prompts, via grade_run.py API)

2026-07-28 18:42:58 -07:00

gemma4-26b-a4b-8bit-mlx-webhook.py

Gemma 4 26B-A4B head-to-head vs Qwen (4 prompts, via grade_run.py API)

2026-07-28 18:42:58 -07:00

gemma4-26b-a4b-8bit-mlx.py

Grade 3 more models + dashboard v2 layout (quant/format as first-class)

2026-07-28 16:51:50 -07:00

gemma4-31b-gguf.py

Initial benchmark suite: 8 graded models + cyberpunk dashboard generator

2026-07-28 14:00:40 -07:00

gemma-4-12b-coder-fable5-composer2.5-v1-uncensored-heretic-mxfp8-mlx.py

Initial benchmark suite: 8 graded models + cyberpunk dashboard generator

2026-07-28 14:00:40 -07:00

gemma-4-26b-a4b-data.py

Grade 12-model backlog + fix native-API TTFT capture

2026-07-28 21:17:08 -07:00

gemma-4-31b-qat-gguf.py

Initial benchmark suite: 8 graded models + cyberpunk dashboard generator

2026-07-28 14:00:40 -07:00

kat-coder-v2.5-dev-xl-mlx-automation.py

Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet)

2026-07-28 19:22:41 -07:00

kat-coder-v2.5-dev-xl-mlx-lfu.py

Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet)

2026-07-28 19:22:41 -07:00

kat-coder-v2.5-dev-xl-mlx-rust.rs

Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet)

2026-07-28 19:22:41 -07:00

kat-coder-v2.5-dev-xl-mlx-tts.py

Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet)

2026-07-28 19:22:41 -07:00

kat-coder-v2.5-dev-xl-mlx-webhook.py

Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet)

2026-07-28 19:22:41 -07:00

kat-coder-v2.5-dev-xl-mlx.py

Initial benchmark suite: 8 graded models + cyberpunk dashboard generator

2026-07-28 14:00:40 -07:00

qwen3-coder-30b-6bit-mlx-automation.py

Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet)

2026-07-28 19:22:41 -07:00

qwen3-coder-30b-6bit-mlx-lfu.py

Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet)

2026-07-28 19:22:41 -07:00

qwen3-coder-30b-6bit-mlx-rust.rs

Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet)

2026-07-28 19:22:41 -07:00

qwen3-coder-30b-6bit-mlx-tts.py

Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet)

2026-07-28 19:22:41 -07:00

qwen3-coder-30b-6bit-mlx-webhook.py

Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet)

2026-07-28 19:22:41 -07:00

qwen3-coder-30b-6bit-mlx.py

Grade 3 more models + dashboard v2 layout (quant/format as first-class)

2026-07-28 16:51:50 -07:00

qwen3.6-27b-8bit-mlx.py

Grade 3 more models + dashboard v2 layout (quant/format as first-class)

2026-07-28 16:51:50 -07:00

qwen3.6-35b-a3b-4bit-mlx.py

Initial benchmark suite: 8 graded models + cyberpunk dashboard generator

2026-07-28 14:00:40 -07:00

qwen3.6-35b-a3b-6bit-mlx-rust.rs

Grade Qwen 6-bit on Rust prompt: 50/100 Critical (does not compile, 7 errors)

2026-07-28 17:53:15 -07:00

qwen3.6-35b-a3b-6bit-mlx-tts.py

First multi-prompt result: Qwen 6-bit TTS = 49 (vs 82 LFU) + per-prompt schema

2026-07-28 17:43:38 -07:00

qwen3.6-35b-a3b-6bit-mlx-webhook.py

Grade Qwen 6-bit on webhook prompt: 75/100 Minor Flaws (runs, all 4 tests pass)

2026-07-28 18:03:50 -07:00

qwen3.6-35b-a3b-6bit-mlx.py

Initial benchmark suite: 8 graded models + cyberpunk dashboard generator

2026-07-28 14:00:40 -07:00

qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-automation.py

Capture uncensored-Qwen-35B automation (others hung via API)

2026-07-28 20:21:13 -07:00

qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf.py

Initial benchmark suite: 8 graded models + cyberpunk dashboard generator

2026-07-28 14:00:40 -07:00