7 entries (30→37 total). Muse Glimmer 28B (GGUF) avg 80.7 — the strongest
model in the benchmark, 6/7 prompts Minor Logic Flaws:
lfu 76 | webhook 81 | automation 89 | rust 85 | data 88 | tts 58 | mcp 88
Standout results:
- rust 85 (KAT 36, Qwen3-Coder 54) — real tokio channels (mpsc::channel, not
hallucinated mpsc::bounded), two-tier CancellationToken, zero clippy lints;
one-line E0507 compile fix.
- automation 89 — first model to print a correct summary (98/2/0/100);
atomic temp+fsync+rename checkpointing.
- data 88 edges out Gemma-26B's 86; mcp 88 sets the bar on a new prompt.
Only weak spot: tts 58 (backpressure raises instead of awaits, like Qwen3-Coder).
Captured via the native /api/v1/chat fix (real tok/sec + TTFT). Slow
deep-thinker: ~17-19 t/s, 5-9 min/prompt, ~5-9k tokens incl. reasoning.
Also gitignore checkpoint.json (automation test runtime artifact).
Co-Authored-By: Claude <noreply@anthropic.com>
Tried Qwen-6bit automation (the one missing prompt) + qwen3.6-27b,
glm-4.7-flash, uncensored-qwen-35b via API — ALL timed out. Likely
cause: Mac screensaver/sleep throttles LM Studio (even fast models
hit the 240s cap with zero output). Documented in memory + CLAUDE.md:
runs need the Mac display AWAKE / sleep disabled.
Captured-but-ungraded outputs in outputs/ (Gemma26B, kat-coder,
qwen3-coder-30b all 5/5) await a grading session. Memory MCP DB was
down during handoff (will retry next session) — file-based memory +
CLAUDE.md carry the context either way.
Co-Authored-By: Claude <noreply@anthropic.com>
The uncensored Qwen 35B hangs/timeouts on 4/5 prompts via the API
(only automation saved). qwen3.6-27b and glm-4.7-flash also time out
via API (27B genuinely slow ~12min; GLM loops without stopping — needs
stop-token handling, noted for next session). Capture batch paused:
3 models fully captured (Gemma26B, kat-coder, qwen3-coder-30b),
grading deferred.
Co-Authored-By: Claude <noreply@anthropic.com>
Via tools/grade_run.py against LM Studio (load -> all prompts -> unload -> verify).
kat-coder: lfu/tts/webhook/automation/rust (lfu has a real SyntaxError: 'self._ bump_freq')
qwen3-coder-30b: lfu/tts/webhook/automation/rust (all 4 py parse clean; terse ~2k tok outputs)
Grading deferred to a later session (capture-only mode).
Script hardening:
- --resident / --unload-all commands (read loaded_instances, POST unload, verify)
- MEMORY GUARD: aborts if a different model is resident (never stack 30GB models)
- TTFT self-discovery dump (LM Studio returns empty stats non-streaming;
fix next session: switch to /api/v1/chat stream:true -> chat.end result.stats)
Co-Authored-By: Claude <noreply@anthropic.com>
Ran via tools/grade_run.py against LM Studio (no clipboard). Results:
TTS: 80 Minor Flaws PASSES (real N-worker concurrency) <- Qwen 49, didn't parse
Rust: 72 Minor Flaws COMPILES CLEAN (0 errs w/ deps) <- Qwen 50, 7 real errors
Webhook: 55 Critical uses forbidden aiohttp (won't run) <- Qwen 75, passed
Automation: 48 Critical SyntaxError (global-after-assign) <- first run for both
DECISIVE head-to-head: Gemma generalizes where Qwen fails (TTS, Rust),
but Qwen beats it on stdlib-discipline prompts (webhook). The two are
COMPLEMENTARY local offloads, not redundant.
Fixed: grade_run.py extractor (markdown/prose wrapping, multi-fence lang
selection), TTFT-null handling in generator. TTFT capture from LM Studio
API still needs the right stats key (left null + noted).
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces clipboard-based submission (which was mangling outputs).
Run from any LAN host that can reach LM Studio:
python3 tools/grade_run.py --lmstudio http://10.0.0.31:1234 --list
python3 tools/grade_run.py --model <id> --prompt tts --name <slug> --append
Saves the raw completion to outputs/<name>.<ext> (no clipboard),
captures tok/sec + tokens + TTFT from the API response (no manual
entry), and drafts a pending JSON entry. The 5-pillar audit still
done by hand (subjective). TODO_submission_tool.md updated.
Co-Authored-By: Claude <noreply@anthropic.com>
Fourth data point on Qwen 3.6 35B-A3B 6-bit MLX:
LFU cache 82 (runs)
Webhook 75 (runs, all tests pass) <- best non-LFU result
TTS pipeline 49 (doesn't parse)
Rust service 50 (7 compile errors)
Profile sharpens: Qwen 6-bit handles SINGLE-HANDLER logic well
(webhook HMAC/idempotency/rate-limit/429-backoff all correct) but fails
on multi-task orchestration (TTS) and typed/compiled langs (Rust).
Safe offload for HTTP/bridge/verification work; not for pipelines or Rust.
Added webhook prompt_id + pillar set. (Initial paste was mangled -
stripped '=' signs - re-pasted clean.)
Co-Authored-By: Claude <noreply@anthropic.com>
Third data point on the same model:
Qwen 3.6 35B-A3B 6-bit MLX:
LFU cache 82 (runs clean)
TTS pipeline 49 (doesn't parse)
Rust service 50 (7 compile errors)
Profile is now sharp: strong on single-file Python data-structure/ACID
work; repeatedly ships non-compiling/non-parsing code on multi-task async
and typed-language prompts. Keep it on Python ACID tasks; do NOT offload
Rust or async-pipeline work.
Verified with cargo 1.94 (mpsc::bounded hallucination, ownership moves,
dead test override, broken remove_watcher). Lang field added for non-Python.
Co-Authored-By: Claude <noreply@anthropic.com>
TTS grade for qwen3.6-35b-a3b-6bit-mlx: 49/100 Critical (same model that
scored 82 on LFU). File doesn't parse + bounded-concurrency is fake
(1 worker + inner semaphore = real concurrency 1). Per-task signal:
strong on data-structures, weak on async-pipeline work.
Schema: prompt_id + PILLARS_BY_PROMPT so each entry uses its own 5 pillars.
TODO_submission_tool.md sketches the grade-as-a-tool idea for later.
Co-Authored-By: Claude <noreply@anthropic.com>
Runnability-first split: TRUSTED/VERIFY (runs clean) vs AVOID (crashes).
Only 5/10 local models produce runnable code.
Co-Authored-By: Claude <noreply@anthropic.com>
Findings panel: live stats from the data (5/10 run tests, quant dominates
quality, concurrency is the killer pillar, 2/11 __slots__, 4/11 monotonic).
Aygea Test (prompts/aygea_test_battery.md): 5-prompt battery derived from
~/dev + jirachi project shapes. Notes prompt_id schema for multi-prompt.
Co-Authored-By: Claude <noreply@anthropic.com>
New graded (11 total now):
gemma4-26b-a4b-8bit-mlx 82 Minor Flaws (tied top local; delta-based tx freq)
qwen3.6-27b-8bit-mlx 78 Minor Flaws (clean; anom. slow generation flagged)
qwen3-coder-30b-6bit-mlx 50 Critical (asyncio.Lock used with sync with -> crash)
Dashboard redesign:
- Bar chart is now the full-width hero row (was cramped half-width)
- 4 stat tiles squished 2x2 beside the radar up top
- Quant + Format are dedicated columns in the leaderboard (MLX/GGUF/CLOUD chips)
- New 'Format & Quant Showdown' panel: groups same-family variants so
GGUF-vs-MLX and quant-depth comparisons are side by side
- Bar-chart axis labels now include the quant so duplicate model names
are distinguishable, with rotation for readability
Co-Authored-By: Claude <noreply@anthropic.com>
webhook.py: HMAC-signed receiver (X-Gitea-Signature), validates ref==main,
one-concurrent-deploy lock, no request data reaches shell.
deploy-webhook.sh: installs llm-bench-webhook systemd service (runs as
aygea, in docker group), generates + stores secret in .webhook.secret.
deploy.sh: port read from compose (now 31415).
Installed on mewtwo: listening 0.0.0.0:41798, enabled for boot.
Gitea webhook target: http://10.0.0.22:41798/hook
Co-Authored-By: Claude <noreply@anthropic.com>
Run on mewtwo: bound 0.0.0.0:8081 -> nginx :80.
Rebuilds site from source on every 'deploy.sh up' so new grades
flow through without Coolify. Point Netbird at http://<host>:8081.
Co-Authored-By: Claude <noreply@anthropic.com>
adminandClaude