diff --git a/CLAUDE.md b/CLAUDE.md index 1206c89..aa05580 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -88,7 +88,30 @@ Tokens: `mlx` = Apple MLX format; `gguf` = llama.cpp GGUF; quant suffixes like ` ### When the user pastes raw output instead of giving a file -If the user pastes a model's output text directly (no file), **save it as a `.py` file in `outputs/`** using the naming convention above before grading. Ask the user for the model name + quant only if it cannot be reasonably inferred from context. Match the existing lowercase-hyphen style. +If the user pastes a model's output text directly (no file), **save it as a `.py` file in `outputs/`** using the naming convention above before grading. Ask the user for the model name + quant only if it cannot be reasonably inferred from context. Match the existing lowercase-hyphen style. **Note:** clipboard paste corrupts long outputs (strips `=` signs / indentation). Prefer the API capture workflow below. + +## Capturing outputs via the LM Studio API (preferred — no clipboard) + +`tools/grade_run.py` runs a prompt against LM Studio's local server, saves the raw completion to `outputs/`, and grabs metrics from the API response. **This is the preferred path** — it avoids clipboard corruption and auto-captures tok/sec + token counts. It also needs the full context of `prompts/aygea_test_battery.md` (the 6-prompt battery) and `PILLARS_BY_PROMPT` in `generate_dashboard.py` (each prompt grades on its own 5 pillars, tagged via the entry's `prompt_id`). + +**Topology:** LM Studio runs on the Mac (LAN `10.0.0.31:1234`); the repo/dashboard run on mewtwo (`10.0.0.22`); mewtwo calls the Mac over LAN. + +``` +python3 tools/grade_run.py --resident # what's actually in RAM (read-only, loads nothing) +python3 tools/grade_run.py --unload-all # unload every resident model + verify clean +python3 tools/grade_run.py --model --prompt

--name [--max-tokens N] + prompts: lfu tts webhook automation rust data mcp +``` + +**⚠ HARD MEMORY RULE:** the Mac has 48 GB; models are ~28–30 GB. **Never have two big models resident.** Per model: confirm `--resident` is clean → run ALL its prompts while it's warm (models load from the NAS, so the first call is slow ~60 s) → `--unload-all` → verify clean → only then touch the next model. The script has a **memory guard** that aborts if a *different* model is resident when you ask to run a new one. `--resident` reads `GET /api/v1/models → loaded_instances[].id` (the only reliable resident list; `/v1/models` lies). Unload is `POST /api/v1/models/unload {"instance_id": }`. + +**Known model behavior** — capture the FAST ones (≥40 t/s); slow/looping models time out via the API and aren't worth batching: +- FAST & safe: `qwen/qwen3-coder-30b`, `qwen/qwen3.6-35b-a3b`, `kat-coder-v2.5-dev-xl-mlx`, `google/gemma-4-26b-a4b`. +- Hangs via API on most prompts: `qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive`. +- Slow / avoid: `qwen/qwen3.6-27b` (~12 min), the `gemma4-31b*` (10–15 t/s). +- Loops without a stop token (endlessly "improving"): `zai-org/glm-4.7-flash` — needs `stop` tokens, not a longer timeout. + +**Reasoning models** (kat-coder, qwen3-coder) burn tokens *thinking* — use `--max-tokens 16000+` or the output comes back empty. **TTFT capture is currently broken** (LM Studio returns empty `stats` on the OpenAI endpoint); the fix is switching to the native streaming `POST /api/v1/chat {"stream":true}` and parsing the `chat.end` event's `result.stats.time_to_first_token_seconds`. ## Hardware / runtime context