modelTesting/CLAUDE.md

CLAUDE.md — Local LLM Benchmark Testing Project

What this project is

A personal test harness for evaluating local LLM models running in LM Studio on an Apple M3 Max MacBook Pro (48 GB unified memory). A fixed coding prompt (LFU cache) is fed to each model, the model's Python output is graded against a strict rubric, and the results are accumulated into a persistent JSON store and rendered into a standalone HTML dashboard.

This is a git repo (remote: ssh://git@git.itsaygea.com:2222/admin/modelTesting.git, branch main) deployed to Coolify, but it has no runtime build/run cycle locally — it's a folder of prompt files, model outputs, a JSON results store, and a Python generator that emits static HTML.

Folder layout

prompts/
  lfu_cache_prompt.txt   # The fixed prompt given to every model (the "exam question")
  grading.txt            # The grader's instructions / rubric (read at session start)
outputs/                 # Raw model outputs (the .py files each model produced)
  <model-name>-<quant>.py
data/
  benchmark_history.json # Persistent results store (SOURCE OF TRUTH — committed)
generate_dashboard.py    # Reads the JSON → builds dashboard.html + pages/*.html
dashboard.html           # GENERATED — gitignored (built on deploy, not committed)
pages/                   # GENERATED — per-model detail pages, gitignored
Dockerfile               # Coolify: python build stage → nginx static serve
DEPLOY.md                # Gitea→Coolify deploy instructions
README.md

Git policy: source only is committed (prompts/, outputs/, data/benchmark_history.json, generate_dashboard.py, Dockerfile, docs). The generated dashboard.html + pages/ are gitignored — Coolify rebuilds them on each push via the Dockerfile's python3 generate_dashboard.py step. If you ever want to commit the built HTML instead, uncomment those lines in .gitignore.

Remote: ssh://git@git.itsaygea.com:2222/admin/modelTesting.git (SSH key auth as admin verified). Push to main → auto-deploys (see Deploy section below).

Deploy: docker compose + Gitea-push webhook (on mewtwo)

Coolify was abandoned — replaced by a self-contained Docker setup on the Proxmox host mewtwo (LAN 10.0.0.22).

• Dashboard: docker compose up builds the Dockerfile (Python stage runs generate_dashboard.py, nginx stage serves dashboard.html+pages/ and proxies /hook). Container llm-benchmark, bound 0.0.0.0:31415:80, restart: unless-stopped. Netbird exposes it as https://llmtesting.itsaygea.com.
• Auto-deploy webhook: systemd service llm-bench-webhook runs webhook.py, listening 0.0.0.0:41798. The dashboard nginx proxies /hook → host:41798 (via extra_hosts: host-gateway). Gitea posts push events to https://llmtesting.itsaygea.com/hook; on a valid push to main it runs git fetch && git reset --hard origin/main && ./deploy.sh up.
• Auth chain (in order): Authorization Bearer token (WEBHOOK_AUTH_TOKEN / .webhook.auth) → 401 if missing/wrong; then HMAC X-Gitea-Signature (WEBHOOK_SECRET / .webhook.secret) → 403; then ref==refs/heads/main.
• Secrets: .webhook.secret and .webhook.auth are gitignored — generated by ./deploy-webhook.sh install.
• Commands: ./deploy.sh [up|logs|down|status] (dashboard), ./deploy-webhook.sh [install|status|secret|auth|uninstall] (webhook).

⚠ Self-revert trap (learned the hard way): the webhook's redeploy does git reset --hard origin/main. If you edit webhook.py/Dockerfile/docker-compose.yml/deploy-webhook.sh locally but DON'T commit+push first, any push (incl. your own test POSTs to /hook) resets those files back to the committed origin/main version, wiping your edits. Rule: edit → commit → push → then restart/rebuild. Never leave deploy-critical files uncommitted while the webhook is live.

prompts/
  lfu_cache_prompt.txt   # The fixed prompt given to every model (the "exam question")
  grading.txt            # The grader's instructions / rubric (read at session start)
outputs/                 # Raw model outputs (the .py files each model produced)
  <model-name>-<quant>.py
data/
  benchmark_history.json # Persistent results store (created on first grading run)
dashboard.html           # Generated standalone dashboard (dark-mode, Chart.js via CDN)
*.py                     # Loose model outputs in root (e.g. deepseekv4flash.py) — legacy/unsorted

The workflow (what to do when the user submits a model's output)

Follow the 4 steps in prompts/grading.txt. Captured metrics per model: tok_sec, total_tokens, ttft_sec (ask the user for these when grading — can't be inferred from files), plus optional speed_caveat text (e.g. the Gemma4 GPU-offload note) and tests_pass (bool — always run the model's .py and record whether it crashes).

1. Audit the .py file against the 5 pillars (each 0–20 → 0–100 total):
   • Complexity violations (true O(1) — no heaps, sorted(), linear scans)
   • Async race conditions / deadlocks (locks held across sleep/IO, unsynced shared state)
   • Transactional isolation leaks (read-your-own-writes, commit/rollback correctness)
   • Memory leaks & edge cases (DLL unlink, min_freq updates, time.monotonic(), __slots__)
   • Test coverage integrity (real edge cases vs trivial always-pass asserts)
2. Extract metrics: model name, quant, tok/sec, verdict, archetype/best-for, critical bugs, patch code.
3. Update data/benchmark_history.json — append/update the model's entry (schema in grading.txt). Create the file if missing.
4. Regenerate the site by running python3 generate_dashboard.py (reads data/benchmark_history.json, overwrites dashboard.html + every pages/<id>.html). Style is locked: cyberpunk-terminal (dark #0a0a0f bg, neon cyan/magenta/lime accents, Fira Code/Sans, scanline+grid texture, glow-on-hover score bars). Dashboard = summary stats + Chart.js bar (score vs tok/sec) + radar (top-3 pillars) + leaderboard table linking to per-model detail pages. Each detail page = verdict/score header, metric tiles, pillar bars, "went right/wrong" columns, critical bugs, recommended use, and a refactored patch code block. tests_pass, cloud models (no tok/sec), and speed-caveats are all handled in the UI.

The exam prompt (lfu_cache_prompt.txt) requires a pure-stdlib Python 3.11+ async LFU cache with frequency-bucket O(1), dual-layer TTL eviction, ACID-like transactions, and an async def main() test suite. Use it as the spec when judging correctness.

Model output — naming convention

Model output filenames encode the model + quant/format so a model is identifiable from the filename alone. The convention the user uses:

<model-family><version>-<param-size>-<variant>-<quant-or-format>.py

Lowercase, hyphen-separated, no spaces. Examples seen so far:

Filename	Reads as
qwen3.6-35b-a3b-6bit-mlx.py	Qwen 3.6, 35B-A3B (MoE), 6-bit, MLX format
qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf.py	Qwen 3.6 35B-A3B, uncensored fine-tune, GGUF
gemma-4-31b-qat-gguf.py	Gemma 4 31B, QAT, GGUF
gemma-4-12b-coder-fable5-composer2.5-v1-uncensored-heretic-mxfp8-mlx.py	merged/model-card-style name, MLX
kat-coder-v2.5-dev-xl-mlx.py	KAT Coder v2.5 Dev XL, MLX

Tokens: mlx = Apple MLX format; gguf = llama.cpp GGUF; quant suffixes like q6k, 6bit, 4bit, qat, mxfp8 go in the name. When in doubt about what a filename denotes, parse it loosely into model + quant rather than guessing a wrong label — the grading step pulls model/quant from "file name or user input," so either source is valid.

When the user pastes raw output instead of giving a file

If the user pastes a model's output text directly (no file), save it as a .py file in outputs/ using the naming convention above before grading. Ask the user for the model name + quant only if it cannot be reasonably inferred from context. Match the existing lowercase-hyphen style. Note: clipboard paste corrupts long outputs (strips = signs / indentation). Prefer the API capture workflow below.

Capturing outputs via the LM Studio API (preferred — no clipboard)

tools/grade_run.py runs a prompt against LM Studio's local server, saves the raw completion to outputs/, and grabs metrics from the API response. This is the preferred path — it avoids clipboard corruption and auto-captures tok/sec + token counts. It also needs the full context of prompts/aygea_test_battery.md (the 6-prompt battery) and PILLARS_BY_PROMPT in generate_dashboard.py (each prompt grades on its own 5 pillars, tagged via the entry's prompt_id).

Topology: LM Studio runs on the Mac (LAN 10.0.0.31:1234); the repo/dashboard run on mewtwo (10.0.0.22); mewtwo calls the Mac over LAN.

python3 tools/grade_run.py --resident            # what's actually in RAM (read-only, loads nothing)
python3 tools/grade_run.py --unload-all          # unload every resident model + verify clean
python3 tools/grade_run.py --model <id> --prompt <p> --name <slug> [--max-tokens N]
  prompts: lfu tts webhook automation rust data mcp

⚠ HARD MEMORY RULE: the Mac has 48 GB; models are ~28–30 GB. Never have two big models resident. Per model: confirm --resident is clean → run ALL its prompts while it's warm (models load from the NAS, so the first call is slow ~60 s) → --unload-all → verify clean → only then touch the next model. The script has a memory guard that aborts if a different model is resident when you ask to run a new one. --resident reads GET /api/v1/models → loaded_instances[].id (the only reliable resident list; /v1/models lies). Unload is POST /api/v1/models/unload {"instance_id": <id>}.

Known model behavior — capture the FAST ones (≥40 t/s); slow/looping models time out via the API and aren't worth batching:

• FAST & safe: qwen/qwen3-coder-30b, qwen/qwen3.6-35b-a3b, kat-coder-v2.5-dev-xl-mlx, google/gemma-4-26b-a4b.
• Hangs via API on most prompts: qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive.
• Slow / avoid: qwen/qwen3.6-27b (~12 min), the gemma4-31b* (10–15 t/s).
• Loops without a stop token (endlessly "improving"): zai-org/glm-4.7-flash — needs stop tokens, not a longer timeout.

Reasoning models (kat-coder, qwen3-coder) burn tokens thinking — use --max-tokens 16000+ or the output comes back empty. TTFT capture is currently broken (LM Studio returns empty stats on the OpenAI endpoint); the fix is switching to the native streaming POST /api/v1/chat {"stream":true} and parsing the chat.end event's result.stats.time_to_first_token_seconds.

Hardware / runtime context

• Machine: Apple M3 Max, 48 GB unified memory. Local inference only.
• Inference server: LM Studio (OpenAI-compatible local endpoint).
• Quants/formats that fit 48 GB comfortably: ~35B dense at 4-bit/6-bit, larger MoE models (only active params loaded). MLX is preferred for Apple Silicon; GGUF via llama.cpp also works.

Notes / gotchas

• data/ starts empty — benchmark_history.json does not exist until the first grading run creates it. Don't assume it's there; read defensively, create on write.
• dashboard.html is regenerated (overwritten), not appended to. The source of truth is the JSON file.
• No build step, no tests to run, no dependencies to install — pure stdlib Python outputs + a static HTML file.
• Treat the loose .py files in the project root (like deepseekv4flash.py) as unsorted outputs that belong in outputs/ per the convention. Don't move them unless asked.