7 entries (30→37 total). Muse Glimmer 28B (GGUF) avg 80.7 — the strongest
model in the benchmark, 6/7 prompts Minor Logic Flaws:
lfu 76 | webhook 81 | automation 89 | rust 85 | data 88 | tts 58 | mcp 88
Standout results:
- rust 85 (KAT 36, Qwen3-Coder 54) — real tokio channels (mpsc::channel, not
hallucinated mpsc::bounded), two-tier CancellationToken, zero clippy lints;
one-line E0507 compile fix.
- automation 89 — first model to print a correct summary (98/2/0/100);
atomic temp+fsync+rename checkpointing.
- data 88 edges out Gemma-26B's 86; mcp 88 sets the bar on a new prompt.
Only weak spot: tts 58 (backpressure raises instead of awaits, like Qwen3-Coder).
Captured via the native /api/v1/chat fix (real tok/sec + TTFT). Slow
deep-thinker: ~17-19 t/s, 5-9 min/prompt, ~5-9k tokens incl. reasoning.
Also gitignore checkpoint.json (automation test runtime artifact).
Co-Authored-By: Claude <noreply@anthropic.com>
The uncensored Qwen 35B hangs/timeouts on 4/5 prompts via the API
(only automation saved). qwen3.6-27b and glm-4.7-flash also time out
via API (27B genuinely slow ~12min; GLM loops without stopping — needs
stop-token handling, noted for next session). Capture batch paused:
3 models fully captured (Gemma26B, kat-coder, qwen3-coder-30b),
grading deferred.
Co-Authored-By: Claude <noreply@anthropic.com>
Via tools/grade_run.py against LM Studio (load -> all prompts -> unload -> verify).
kat-coder: lfu/tts/webhook/automation/rust (lfu has a real SyntaxError: 'self._ bump_freq')
qwen3-coder-30b: lfu/tts/webhook/automation/rust (all 4 py parse clean; terse ~2k tok outputs)
Grading deferred to a later session (capture-only mode).
Script hardening:
- --resident / --unload-all commands (read loaded_instances, POST unload, verify)
- MEMORY GUARD: aborts if a different model is resident (never stack 30GB models)
- TTFT self-discovery dump (LM Studio returns empty stats non-streaming;
fix next session: switch to /api/v1/chat stream:true -> chat.end result.stats)
Co-Authored-By: Claude <noreply@anthropic.com>
adminandClaude