modelTesting/CLAUDE.md

CLAUDE.md — Local LLM Benchmark Testing Project

What this project is

A personal test harness for evaluating local LLM models running in LM Studio on an Apple M3 Max MacBook Pro (48 GB unified memory). A fixed coding prompt (LFU cache) is fed to each model, the model's Python output is graded against a strict rubric, and the results are accumulated into a persistent JSON store and rendered into a standalone HTML dashboard.

This is not a git repo and is not an application with a build/run cycle. It is a folder of prompt files, model outputs, and generated artifacts.

Folder layout

prompts/
  lfu_cache_prompt.txt   # The fixed prompt given to every model (the "exam question")
  grading.txt            # The grader's instructions / rubric (read at session start)
outputs/                 # Raw model outputs (the .py files each model produced)
  <model-name>-<quant>.py
data/
  benchmark_history.json # Persistent results store (created on first grading run)
dashboard.html           # Generated standalone dashboard (dark-mode, Chart.js via CDN)
*.py                     # Loose model outputs in root (e.g. deepseekv4flash.py) — legacy/unsorted

The workflow (what to do when the user submits a model's output)

Follow the 4 steps in prompts/grading.txt:

1. Audit the .py file against the 5 pillars (each 0–20 → 0–100 total):
   • Complexity violations (true O(1) — no heaps, sorted(), linear scans)
   • Async race conditions / deadlocks (locks held across sleep/IO, unsynced shared state)
   • Transactional isolation leaks (read-your-own-writes, commit/rollback correctness)
   • Memory leaks & edge cases (DLL unlink, min_freq updates, time.monotonic(), __slots__)
   • Test coverage integrity (real edge cases vs trivial always-pass asserts)
2. Extract metrics: model name, quant, tok/sec, verdict, archetype/best-for, critical bugs, patch code.
3. Update data/benchmark_history.json — append/update the model's entry (schema in grading.txt). Create the file if missing.
4. Regenerate dashboard.html — full overwrite, standalone single file, Chart.js (bar + radar), leaderboard table, collapsible audit cards, color-coded verdict badges (green 90+, blue/yellow 75–89, red <75).

The exam prompt (lfu_cache_prompt.txt) requires a pure-stdlib Python 3.11+ async LFU cache with frequency-bucket O(1), dual-layer TTL eviction, ACID-like transactions, and an async def main() test suite. Use it as the spec when judging correctness.

Model output — naming convention

Model output filenames encode the model + quant/format so a model is identifiable from the filename alone. The convention the user uses:

<model-family><version>-<param-size>-<variant>-<quant-or-format>.py

Lowercase, hyphen-separated, no spaces. Examples seen so far:

Filename	Reads as
qwen3.6-35b-a3b-6bit-mlx.py	Qwen 3.6, 35B-A3B (MoE), 6-bit, MLX format
qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf.py	Qwen 3.6 35B-A3B, uncensored fine-tune, GGUF
gemma-4-31b-qat-gguf.py	Gemma 4 31B, QAT, GGUF
gemma-4-12b-coder-fable5-composer2.5-v1-uncensored-heretic-mxfp8-mlx.py	merged/model-card-style name, MLX
kat-coder-v2.5-dev-xl-mlx.py	KAT Coder v2.5 Dev XL, MLX

Tokens: mlx = Apple MLX format; gguf = llama.cpp GGUF; quant suffixes like q6k, 6bit, 4bit, qat, mxfp8 go in the name. When in doubt about what a filename denotes, parse it loosely into model + quant rather than guessing a wrong label — the grading step pulls model/quant from "file name or user input," so either source is valid.

When the user pastes raw output instead of giving a file

If the user pastes a model's output text directly (no file), save it as a .py file in outputs/ using the naming convention above before grading. Ask the user for the model name + quant only if it cannot be reasonably inferred from context. Match the existing lowercase-hyphen style.

Hardware / runtime context

• Machine: Apple M3 Max, 48 GB unified memory. Local inference only.
• Inference server: LM Studio (OpenAI-compatible local endpoint).
• Quants/formats that fit 48 GB comfortably: ~35B dense at 4-bit/6-bit, larger MoE models (only active params loaded). MLX is preferred for Apple Silicon; GGUF via llama.cpp also works.

Notes / gotchas

• data/ starts empty — benchmark_history.json does not exist until the first grading run creates it. Don't assume it's there; read defensively, create on write.
• dashboard.html is regenerated (overwritten), not appended to. The source of truth is the JSON file.
• No build step, no tests to run, no dependencies to install — pure stdlib Python outputs + a static HTML file.
• Treat the loose .py files in the project root (like deepseekv4flash.py) as unsorted outputs that belong in outputs/ per the convention. Don't move them unless asked.