Initial benchmark suite: 8 graded models + cyberpunk dashboard generator
- prompts/: LFU cache exam + 5-pillar grading rubric - outputs/: 8 model .py outputs (local + cloud baseline) - data/benchmark_history.json: graded results (scores, metrics, bugs, patches) - generate_dashboard.py: builds dashboard.html + pages/*.html from JSON - Dockerfile + DEPLOY.md: Gitea→Coolify deploy (build-step, nginx static) - .gitignore: generated HTML excluded (built on deploy) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Claude
@@ -0,0 +1,72 @@
|
||||
# CLAUDE.md — Local LLM Benchmark Testing Project
|
||||
|
||||
## What this project is
|
||||
|
||||
A personal test harness for evaluating **local LLM models** running in **LM Studio** on an **Apple M3 Max MacBook Pro (48 GB unified memory)**. A fixed coding prompt (LFU cache) is fed to each model, the model's Python output is graded against a strict rubric, and the results are accumulated into a persistent JSON store and rendered into a standalone HTML dashboard.
|
||||
|
||||
This is **not** a git repo and is **not** an application with a build/run cycle. It is a folder of prompt files, model outputs, and generated artifacts.
|
||||
|
||||
## Folder layout
|
||||
|
||||
```
|
||||
prompts/
|
||||
lfu_cache_prompt.txt # The fixed prompt given to every model (the "exam question")
|
||||
grading.txt # The grader's instructions / rubric (read at session start)
|
||||
outputs/ # Raw model outputs (the .py files each model produced)
|
||||
<model-name>-<quant>.py
|
||||
data/
|
||||
benchmark_history.json # Persistent results store (created on first grading run)
|
||||
dashboard.html # Generated standalone dashboard (dark-mode, Chart.js via CDN)
|
||||
*.py # Loose model outputs in root (e.g. deepseekv4flash.py) — legacy/unsorted
|
||||
```
|
||||
|
||||
## The workflow (what to do when the user submits a model's output)
|
||||
|
||||
Follow the 4 steps in `prompts/grading.txt`:
|
||||
|
||||
1. **Audit** the `.py` file against the 5 pillars (each 0–20 → 0–100 total):
|
||||
- Complexity violations (true O(1) — no heaps, `sorted()`, linear scans)
|
||||
- Async race conditions / deadlocks (locks held across `sleep`/IO, unsynced shared state)
|
||||
- Transactional isolation leaks (read-your-own-writes, commit/rollback correctness)
|
||||
- Memory leaks & edge cases (DLL unlink, `min_freq` updates, `time.monotonic()`, `__slots__`)
|
||||
- Test coverage integrity (real edge cases vs trivial always-pass asserts)
|
||||
2. **Extract metrics**: model name, quant, tok/sec, verdict, archetype/best-for, critical bugs, patch code.
|
||||
3. **Update `data/benchmark_history.json`** — append/update the model's entry (schema in `grading.txt`). Create the file if missing.
|
||||
4. **Regenerate `dashboard.html`** — full overwrite, standalone single file, Chart.js (bar + radar), leaderboard table, collapsible audit cards, color-coded verdict badges (green 90+, blue/yellow 75–89, red <75).
|
||||
|
||||
The exam prompt (`lfu_cache_prompt.txt`) requires a **pure-stdlib Python 3.11+ async LFU cache** with frequency-bucket O(1), dual-layer TTL eviction, ACID-like transactions, and an `async def main()` test suite. Use it as the spec when judging correctness.
|
||||
|
||||
## Model output — naming convention
|
||||
|
||||
Model output filenames encode the **model + quant/format** so a model is identifiable from the filename alone. The convention the user uses:
|
||||
|
||||
`<model-family><version>-<param-size>-<variant>-<quant-or-format>.py`
|
||||
|
||||
Lowercase, hyphen-separated, no spaces. Examples seen so far:
|
||||
|
||||
| Filename | Reads as |
|
||||
|---|---|
|
||||
| `qwen3.6-35b-a3b-6bit-mlx.py` | Qwen 3.6, 35B-A3B (MoE), 6-bit, MLX format |
|
||||
| `qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf.py` | Qwen 3.6 35B-A3B, uncensored fine-tune, GGUF |
|
||||
| `gemma-4-31b-qat-gguf.py` | Gemma 4 31B, QAT, GGUF |
|
||||
| `gemma-4-12b-coder-fable5-composer2.5-v1-uncensored-heretic-mxfp8-mlx.py` | merged/model-card-style name, MLX |
|
||||
| `kat-coder-v2.5-dev-xl-mlx.py` | KAT Coder v2.5 Dev XL, MLX |
|
||||
|
||||
Tokens: `mlx` = Apple MLX format; `gguf` = llama.cpp GGUF; quant suffixes like `q6k`, `6bit`, `4bit`, `qat`, `mxfp8` go in the name. When in doubt about what a filename denotes, **parse it loosely into model + quant** rather than guessing a wrong label — the grading step pulls model/quant from "file name or user input," so either source is valid.
|
||||
|
||||
### When the user pastes raw output instead of giving a file
|
||||
|
||||
If the user pastes a model's output text directly (no file), **save it as a `.py` file in `outputs/`** using the naming convention above before grading. Ask the user for the model name + quant only if it cannot be reasonably inferred from context. Match the existing lowercase-hyphen style.
|
||||
|
||||
## Hardware / runtime context
|
||||
|
||||
- **Machine:** Apple M3 Max, 48 GB unified memory. Local inference only.
|
||||
- **Inference server:** LM Studio (OpenAI-compatible local endpoint).
|
||||
- Quants/formats that fit 48 GB comfortably: ~35B dense at 4-bit/6-bit, larger MoE models (only active params loaded). MLX is preferred for Apple Silicon; GGUF via llama.cpp also works.
|
||||
|
||||
## Notes / gotchas
|
||||
|
||||
- `data/` starts **empty** — `benchmark_history.json` does not exist until the first grading run creates it. Don't assume it's there; read defensively, create on write.
|
||||
- `dashboard.html` is **regenerated** (overwritten), not appended to. The source of truth is the JSON file.
|
||||
- No build step, no tests to run, no dependencies to install — pure stdlib Python outputs + a static HTML file.
|
||||
- Treat the loose `.py` files in the project root (like `deepseekv4flash.py`) as unsorted outputs that belong in `outputs/` per the convention. Don't move them unless asked.
|
||||
Reference in New Issue
Block a user