Third data point on the same model:
Qwen 3.6 35B-A3B 6-bit MLX:
LFU cache 82 (runs clean)
TTS pipeline 49 (doesn't parse)
Rust service 50 (7 compile errors)
Profile is now sharp: strong on single-file Python data-structure/ACID
work; repeatedly ships non-compiling/non-parsing code on multi-task async
and typed-language prompts. Keep it on Python ACID tasks; do NOT offload
Rust or async-pipeline work.
Verified with cargo 1.94 (mpsc::bounded hallucination, ownership moves,
dead test override, broken remove_watcher). Lang field added for non-Python.
Co-Authored-By: Claude <noreply@anthropic.com>
TTS grade for qwen3.6-35b-a3b-6bit-mlx: 49/100 Critical (same model that
scored 82 on LFU). File doesn't parse + bounded-concurrency is fake
(1 worker + inner semaphore = real concurrency 1). Per-task signal:
strong on data-structures, weak on async-pipeline work.
Schema: prompt_id + PILLARS_BY_PROMPT so each entry uses its own 5 pillars.
TODO_submission_tool.md sketches the grade-as-a-tool idea for later.
Co-Authored-By: Claude <noreply@anthropic.com>
Runnability-first split: TRUSTED/VERIFY (runs clean) vs AVOID (crashes).
Only 5/10 local models produce runnable code.
Co-Authored-By: Claude <noreply@anthropic.com>
Findings panel: live stats from the data (5/10 run tests, quant dominates
quality, concurrency is the killer pillar, 2/11 __slots__, 4/11 monotonic).
Aygea Test (prompts/aygea_test_battery.md): 5-prompt battery derived from
~/dev + jirachi project shapes. Notes prompt_id schema for multi-prompt.
Co-Authored-By: Claude <noreply@anthropic.com>
New graded (11 total now):
gemma4-26b-a4b-8bit-mlx 82 Minor Flaws (tied top local; delta-based tx freq)
qwen3.6-27b-8bit-mlx 78 Minor Flaws (clean; anom. slow generation flagged)
qwen3-coder-30b-6bit-mlx 50 Critical (asyncio.Lock used with sync with -> crash)
Dashboard redesign:
- Bar chart is now the full-width hero row (was cramped half-width)
- 4 stat tiles squished 2x2 beside the radar up top
- Quant + Format are dedicated columns in the leaderboard (MLX/GGUF/CLOUD chips)
- New 'Format & Quant Showdown' panel: groups same-family variants so
GGUF-vs-MLX and quant-depth comparisons are side by side
- Bar-chart axis labels now include the quant so duplicate model names
are distinguishable, with rotation for readability
Co-Authored-By: Claude <noreply@anthropic.com>
webhook.py: HMAC-signed receiver (X-Gitea-Signature), validates ref==main,
one-concurrent-deploy lock, no request data reaches shell.
deploy-webhook.sh: installs llm-bench-webhook systemd service (runs as
aygea, in docker group), generates + stores secret in .webhook.secret.
deploy.sh: port read from compose (now 31415).
Installed on mewtwo: listening 0.0.0.0:41798, enabled for boot.
Gitea webhook target: http://10.0.0.22:41798/hook
Co-Authored-By: Claude <noreply@anthropic.com>
Run on mewtwo: bound 0.0.0.0:8081 -> nginx :80.
Rebuilds site from source on every 'deploy.sh up' so new grades
flow through without Coolify. Point Netbird at http://<host>:8081.
Co-Authored-By: Claude <noreply@anthropic.com>
adminandClaude