AygeaandClaude
2f99dd1e35
7 entries (30→37 total). Muse Glimmer 28B (GGUF) avg 80.7 — the strongest model in the benchmark, 6/7 prompts Minor Logic Flaws: lfu 76 | webhook 81 | automation 89 | rust 85 | data 88 | tts 58 | mcp 88 Standout results: - rust 85 (KAT 36, Qwen3-Coder 54) — real tokio channels (mpsc::channel, not hallucinated mpsc::bounded), two-tier CancellationToken, zero clippy lints; one-line E0507 compile fix. - automation 89 — first model to print a correct summary (98/2/0/100); atomic temp+fsync+rename checkpointing. - data 88 edges out Gemma-26B's 86; mcp 88 sets the bar on a new prompt. Only weak spot: tts 58 (backpressure raises instead of awaits, like Qwen3-Coder). Captured via the native /api/v1/chat fix (real tok/sec + TTFT). Slow deep-thinker: ~17-19 t/s, 5-9 min/prompt, ~5-9k tokens incl. reasoning. Also gitignore checkpoint.json (automation test runtime artifact). Co-Authored-By: Claude <noreply@anthropic.com>
🧪 Local LLM Benchmark Suite
Grades local LLM models (run via LM Studio on an Apple M3 Max / 48 GB) on a strict systems-coding prompt — a pure-stdlib Python concurrent async LFU cache with TTL eviction and ACID transactions — and renders the results into a cyberpunk-terminal dashboard.
What's in here
prompts/
lfu_cache_prompt.txt # the exam prompt every model gets
grading.txt # the grader's rubric (5 pillars × 20 pts = 100)
outputs/ # raw model .py outputs (named <model>-<quant>.py)
data/
benchmark_history.json # persistent results store (source of truth)
generate_dashboard.py # reads the JSON → builds dashboard.html + pages/*.html
dashboard.html # generated — main leaderboard + charts (gitignored)
pages/ # generated — per-model detail pages (gitignored)
Workflow
-
Feed
prompts/lfu_cache_prompt.txtto a model in LM Studio. -
Save its output to
outputs/<model>-<quant>.py. -
Grade it (audit the 5 pillars, capture tok/sec + tokens + TTFT), and append its entry to
data/benchmark_history.json. Seeprompts/grading.txtfor the rubric. -
Regenerate the site:
python3 generate_dashboard.pyThis (re)writes
dashboard.htmland everypages/<model>.html.
Viewing locally
python3 -m http.server 8000
# open http://localhost:8000/dashboard.html
Deploying (Gitea + Coolify)
The repo holds source only (outputs/, data/, generate_dashboard.py, prompts/).
The generated dashboard.html and pages/ are gitignored — Coolify runs
python3 generate_dashboard.py as a build step, then serves the static files.
See DEPLOY.md for the exact Coolify service config.
Hardware
Apple M3 Max, 48 GB unified memory. Local inference via LM Studio. A cloud model (DeepSeek V4 Flash) is included as a quality baseline.