modelTesting

21 Commits 1 Branch 0 Tags

2 Commits

Author	SHA1	Message	Date
AygeaandClaude	2f99dd1e35	Grade muse-glimmer-28b full 7-prompt battery — new benchmark leader 7 entries (30→37 total). Muse Glimmer 28B (GGUF) avg 80.7 — the strongest model in the benchmark, 6/7 prompts Minor Logic Flaws: lfu 76 \| webhook 81 \| automation 89 \| rust 85 \| data 88 \| tts 58 \| mcp 88 Standout results: - rust 85 (KAT 36, Qwen3-Coder 54) — real tokio channels (mpsc::channel, not hallucinated mpsc::bounded), two-tier CancellationToken, zero clippy lints; one-line E0507 compile fix. - automation 89 — first model to print a correct summary (98/2/0/100); atomic temp+fsync+rename checkpointing. - data 88 edges out Gemma-26B's 86; mcp 88 sets the bar on a new prompt. Only weak spot: tts 58 (backpressure raises instead of awaits, like Qwen3-Coder). Captured via the native /api/v1/chat fix (real tok/sec + TTFT). Slow deep-thinker: ~17-19 t/s, 5-9 min/prompt, ~5-9k tokens incl. reasoning. Also gitignore checkpoint.json (automation test runtime artifact). Co-Authored-By: Claude <noreply@anthropic.com>	2026-08-10 13:13:54 -07:00
AygeaandClaude	3507d33006	Grade 12-model backlog + fix native-API TTFT capture Grading (12 new entries, 18→30 total in benchmark_history.json): - KAT-Coder v2.5 Dev XL: lfu 49 / tts 52 / webhook 72 / rust 36 / automation 60 - Qwen3 Coder 30B: lfu 45 / tts 44 / webhook 62 / rust 54 / automation 58 - Qwen 3.6 35B-A3B uncensored (automation): 46 - Gemma 4 26B-A4B (data): 86 [tests pass] All "coder" models scored Critical Bugs across prompts — plausible-looking async code with fatal bugs (broken LFU eviction, in-flight cancel no-op, submit() raising instead of backpressuring, un-awaited async read-through). grade_run.py: switch from OpenAI-compat /v1/chat/completions (empty stats) to native /api/v1/chat — returns full stats incl. time_to_first_token_seconds. Verified on Gemma-26B (53.5 t/s, ttft 1.13). Two native-API gotchas handled: input (string) not messages; max_output_tokens not max_tokens (that 400s). New capture: outputs/gemma-4-26b-a4b-data.py (native-API run). Co-Authored-By: Claude <noreply@anthropic.com>	2026-07-28 21:17:08 -07:00

Author

SHA1

Message

Date

AygeaandClaude

2f99dd1e35

Grade muse-glimmer-28b full 7-prompt battery — new benchmark leader

7 entries (30→37 total). Muse Glimmer 28B (GGUF) avg 80.7 — the strongest
model in the benchmark, 6/7 prompts Minor Logic Flaws:
  lfu 76 | webhook 81 | automation 89 | rust 85 | data 88 | tts 58 | mcp 88

Standout results:
- rust 85 (KAT 36, Qwen3-Coder 54) — real tokio channels (mpsc::channel, not
  hallucinated mpsc::bounded), two-tier CancellationToken, zero clippy lints;
  one-line E0507 compile fix.
- automation 89 — first model to print a correct summary (98/2/0/100);
  atomic temp+fsync+rename checkpointing.
- data 88 edges out Gemma-26B's 86; mcp 88 sets the bar on a new prompt.
Only weak spot: tts 58 (backpressure raises instead of awaits, like Qwen3-Coder).

Captured via the native /api/v1/chat fix (real tok/sec + TTFT). Slow
deep-thinker: ~17-19 t/s, 5-9 min/prompt, ~5-9k tokens incl. reasoning.

Also gitignore checkpoint.json (automation test runtime artifact).

Co-Authored-By: Claude <noreply@anthropic.com>

2026-08-10 13:13:54 -07:00

AygeaandClaude

3507d33006

Grade 12-model backlog + fix native-API TTFT capture

Grading (12 new entries, 18→30 total in benchmark_history.json):
- KAT-Coder v2.5 Dev XL: lfu 49 / tts 52 / webhook 72 / rust 36 / automation 60
- Qwen3 Coder 30B: lfu 45 / tts 44 / webhook 62 / rust 54 / automation 58
- Qwen 3.6 35B-A3B uncensored (automation): 46
- Gemma 4 26B-A4B (data): 86 [tests pass]
All "coder" models scored Critical Bugs across prompts — plausible-looking
async code with fatal bugs (broken LFU eviction, in-flight cancel no-op,
submit() raising instead of backpressuring, un-awaited async read-through).

grade_run.py: switch from OpenAI-compat /v1/chat/completions (empty stats)
to native /api/v1/chat — returns full stats incl. time_to_first_token_seconds.
Verified on Gemma-26B (53.5 t/s, ttft 1.13). Two native-API gotchas handled:
input (string) not messages; max_output_tokens not max_tokens (that 400s).

New capture: outputs/gemma-4-26b-a4b-data.py (native-API run).

Co-Authored-By: Claude <noreply@anthropic.com>

2026-07-28 21:17:08 -07:00