# CLAUDE.md — Local LLM Benchmark Testing Project

## What this project is

A personal test harness for evaluating **local LLM models** running in **LM Studio** on an **Apple M3 Max MacBook Pro (48 GB unified memory)**. A fixed coding prompt (LFU cache) is fed to each model, the model's Python output is graded against a strict rubric, and the results are accumulated into a persistent JSON store and rendered into a standalone HTML dashboard.

This is **a git repo** (remote: `ssh://git@git.itsaygea.com:2222/admin/modelTesting.git`, branch `main`) deployed to Coolify, but it has **no runtime build/run cycle locally** — it's a folder of prompt files, model outputs, a JSON results store, and a Python generator that emits static HTML.

## Folder layout

```
prompts/
  lfu_cache_prompt.txt   # The fixed prompt given to every model (the "exam question")
  grading.txt            # The grader's instructions / rubric (read at session start)
outputs/                 # Raw model outputs (the .py files each model produced)
  <model-name>-<quant>.py
data/
  benchmark_history.json # Persistent results store (SOURCE OF TRUTH — committed)
generate_dashboard.py    # Reads the JSON → builds dashboard.html + pages/*.html
dashboard.html           # GENERATED — gitignored (built on deploy, not committed)
pages/                   # GENERATED — per-model detail pages, gitignored
Dockerfile               # Coolify: python build stage → nginx static serve
DEPLOY.md                # Gitea→Coolify deploy instructions
README.md
```

**Git policy:** source only is committed (`prompts/`, `outputs/`, `data/benchmark_history.json`, `generate_dashboard.py`, `Dockerfile`, docs). The generated `dashboard.html` + `pages/` are gitignored — Coolify rebuilds them on each push via the Dockerfile's `python3 generate_dashboard.py` step. If you ever want to commit the built HTML instead, uncomment those lines in `.gitignore`.

**Remote:** `ssh://git@git.itsaygea.com:2222/admin/modelTesting.git` (SSH key auth as `admin` verified). Push to `main` → auto-deploys (see Deploy section below).

## Deploy: docker compose + Gitea-push webhook (on mewtwo)

Coolify was abandoned — replaced by a self-contained Docker setup on the Proxmox host `mewtwo` (LAN `10.0.0.22`).

- **Dashboard:** `docker compose up` builds the `Dockerfile` (Python stage runs `generate_dashboard.py`, nginx stage serves `dashboard.html`+`pages/` and proxies `/hook`). Container `llm-benchmark`, bound `0.0.0.0:31415:80`, `restart: unless-stopped`. Netbird exposes it as `https://llmtesting.itsaygea.com`.
- **Auto-deploy webhook:** systemd service `llm-bench-webhook` runs `webhook.py`, listening `0.0.0.0:41798`. The dashboard nginx proxies `/hook` → `host:41798` (via `extra_hosts: host-gateway`). Gitea posts push events to `https://llmtesting.itsaygea.com/hook`; on a valid push to `main` it runs `git fetch && git reset --hard origin/main && ./deploy.sh up`.
- **Auth chain (in order):** Authorization Bearer token (`WEBHOOK_AUTH_TOKEN` / `.webhook.auth`) → 401 if missing/wrong; then HMAC `X-Gitea-Signature` (`WEBHOOK_SECRET` / `.webhook.secret`) → 403; then ref==`refs/heads/main`.
- **Secrets:** `.webhook.secret` and `.webhook.auth` are gitignored — generated by `./deploy-webhook.sh install`.
- **Commands:** `./deploy.sh [up|logs|down|status]` (dashboard), `./deploy-webhook.sh [install|status|secret|auth|uninstall]` (webhook).

**⚠ Self-revert trap (learned the hard way):** the webhook's redeploy does `git reset --hard origin/main`. If you edit `webhook.py`/`Dockerfile`/`docker-compose.yml`/`deploy-webhook.sh` locally but DON'T commit+push first, any push (incl. your own test POSTs to `/hook`) resets those files back to the committed `origin/main` version, wiping your edits. Rule: **edit → commit → push → then restart/rebuild.** Never leave deploy-critical files uncommitted while the webhook is live.

```
prompts/
  lfu_cache_prompt.txt   # The fixed prompt given to every model (the "exam question")
  grading.txt            # The grader's instructions / rubric (read at session start)
outputs/                 # Raw model outputs (the .py files each model produced)
  <model-name>-<quant>.py
data/
  benchmark_history.json # Persistent results store (created on first grading run)
dashboard.html           # Generated standalone dashboard (dark-mode, Chart.js via CDN)
*.py                     # Loose model outputs in root (e.g. deepseekv4flash.py) — legacy/unsorted
```

## The workflow (what to do when the user submits a model's output)

Follow the 4 steps in `prompts/grading.txt`. **Captured metrics per model:** `tok_sec`, `total_tokens`, `ttft_sec` (ask the user for these when grading — can't be inferred from files), plus optional `speed_caveat` text (e.g. the Gemma4 GPU-offload note) and `tests_pass` (bool — always run the model's `.py` and record whether it crashes).

1. **Audit** the `.py` file against the 5 pillars (each 0–20 → 0–100 total):
   - Complexity violations (true O(1) — no heaps, `sorted()`, linear scans)
   - Async race conditions / deadlocks (locks held across `sleep`/IO, unsynced shared state)
   - Transactional isolation leaks (read-your-own-writes, commit/rollback correctness)
   - Memory leaks & edge cases (DLL unlink, `min_freq` updates, `time.monotonic()`, `__slots__`)
   - Test coverage integrity (real edge cases vs trivial always-pass asserts)
2. **Extract metrics**: model name, quant, tok/sec, verdict, archetype/best-for, critical bugs, patch code.
3. **Update `data/benchmark_history.json`** — append/update the model's entry (schema in `grading.txt`). Create the file if missing.
4. **Regenerate the site** by running `python3 generate_dashboard.py` (reads `data/benchmark_history.json`, overwrites `dashboard.html` + every `pages/<id>.html`). Style is locked: **cyberpunk-terminal** (dark `#0a0a0f` bg, neon cyan/magenta/lime accents, Fira Code/Sans, scanline+grid texture, glow-on-hover score bars). Dashboard = summary stats + Chart.js bar (score vs tok/sec) + radar (top-3 pillars) + leaderboard table linking to per-model detail pages. Each detail page = verdict/score header, metric tiles, pillar bars, "went right/wrong" columns, critical bugs, recommended use, and a refactored patch code block. `tests_pass`, cloud models (no tok/sec), and speed-caveats are all handled in the UI.

The exam prompt (`lfu_cache_prompt.txt`) requires a **pure-stdlib Python 3.11+ async LFU cache** with frequency-bucket O(1), dual-layer TTL eviction, ACID-like transactions, and an `async def main()` test suite. Use it as the spec when judging correctness.

## Model output — naming convention

Model output filenames encode the **model + quant/format** so a model is identifiable from the filename alone. The convention the user uses:

`<model-family><version>-<param-size>-<variant>-<quant-or-format>.py`

Lowercase, hyphen-separated, no spaces. Examples seen so far:

| Filename | Reads as |
|---|---|
| `qwen3.6-35b-a3b-6bit-mlx.py` | Qwen 3.6, 35B-A3B (MoE), 6-bit, MLX format |
| `qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf.py` | Qwen 3.6 35B-A3B, uncensored fine-tune, GGUF |
| `gemma-4-31b-qat-gguf.py` | Gemma 4 31B, QAT, GGUF |
| `gemma-4-12b-coder-fable5-composer2.5-v1-uncensored-heretic-mxfp8-mlx.py` | merged/model-card-style name, MLX |
| `kat-coder-v2.5-dev-xl-mlx.py` | KAT Coder v2.5 Dev XL, MLX |

Tokens: `mlx` = Apple MLX format; `gguf` = llama.cpp GGUF; quant suffixes like `q6k`, `6bit`, `4bit`, `qat`, `mxfp8` go in the name. When in doubt about what a filename denotes, **parse it loosely into model + quant** rather than guessing a wrong label — the grading step pulls model/quant from "file name or user input," so either source is valid.

### When the user pastes raw output instead of giving a file

If the user pastes a model's output text directly (no file), **save it as a `.py` file in `outputs/`** using the naming convention above before grading. Ask the user for the model name + quant only if it cannot be reasonably inferred from context. Match the existing lowercase-hyphen style. **Note:** clipboard paste corrupts long outputs (strips `=` signs / indentation). Prefer the API capture workflow below.

## Capturing outputs via the LM Studio API (preferred — no clipboard)

`tools/grade_run.py` runs a prompt against LM Studio's local server, saves the raw completion to `outputs/`, and grabs metrics from the API response. **This is the preferred path** — it avoids clipboard corruption and auto-captures tok/sec + token counts. It also needs the full context of `prompts/aygea_test_battery.md` (the 6-prompt battery) and `PILLARS_BY_PROMPT` in `generate_dashboard.py` (each prompt grades on its own 5 pillars, tagged via the entry's `prompt_id`).

**Topology:** LM Studio runs on the Mac (LAN `10.0.0.31:1234`); the repo/dashboard run on mewtwo (`10.0.0.22`); mewtwo calls the Mac over LAN.

```
python3 tools/grade_run.py --resident            # what's actually in RAM (read-only, loads nothing)
python3 tools/grade_run.py --unload-all          # unload every resident model + verify clean
python3 tools/grade_run.py --model <id> --prompt <p> --name <slug> [--max-tokens N]
  prompts: lfu tts webhook automation rust data mcp
```

**⚠ HARD MEMORY RULE:** the Mac has 48 GB; models are ~28–30 GB. **Never have two big models resident.** Per model: confirm `--resident` is clean → run ALL its prompts while it's warm (models load from the NAS, so the first call is slow ~60 s) → `--unload-all` → verify clean → only then touch the next model. The script has a **memory guard** that aborts if a *different* model is resident when you ask to run a new one. `--resident` reads `GET /api/v1/models → loaded_instances[].id` (the only reliable resident list; `/v1/models` lies). Unload is `POST /api/v1/models/unload {"instance_id": <id>}`.

**Known model behavior** — capture the FAST ones (≥40 t/s); slow/looping models time out via the API and aren't worth batching:
- FAST & safe: `qwen/qwen3-coder-30b`, `qwen/qwen3.6-35b-a3b`, `kat-coder-v2.5-dev-xl-mlx`, `google/gemma-4-26b-a4b`.
- Hangs via API on most prompts: `qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive`.
- Slow / avoid: `qwen/qwen3.6-27b` (~12 min), the `gemma4-31b*` (10–15 t/s).
- Loops without a stop token (endlessly "improving"): `zai-org/glm-4.7-flash` — needs `stop` tokens, not a longer timeout.

**Reasoning models** (kat-coder, qwen3-coder) burn tokens *thinking* — use `--max-tokens 16000+` or the output comes back empty. **TTFT capture is currently broken** (LM Studio returns empty `stats` on the OpenAI endpoint); the fix is switching to the native streaming `POST /api/v1/chat {"stream":true}` and parsing the `chat.end` event's `result.stats.time_to_first_token_seconds`.

## Hardware / runtime context

- **Machine:** Apple M3 Max, 48 GB unified memory. Local inference only.
- **Inference server:** LM Studio (OpenAI-compatible local endpoint).
- Quants/formats that fit 48 GB comfortably: ~35B dense at 4-bit/6-bit, larger MoE models (only active params loaded). MLX is preferred for Apple Silicon; GGUF via llama.cpp also works.

## Notes / gotchas

- `data/` starts **empty** — `benchmark_history.json` does not exist until the first grading run creates it. Don't assume it's there; read defensively, create on write.
- `dashboard.html` is **regenerated** (overwritten), not appended to. The source of truth is the JSON file.
- No build step, no tests to run, no dependencies to install — pure stdlib Python outputs + a static HTML file.
- Treat the loose `.py` files in the project root (like `deepseekv4flash.py`) as unsorted outputs that belong in `outputs/` per the convention. Don't move them unless asked.