Handoff: capture workflow in CLAUDE.md + per-session memory files

Tried Qwen-6bit automation (the one missing prompt) + qwen3.6-27b,
glm-4.7-flash, uncensored-qwen-35b via API — ALL timed out. Likely
cause: Mac screensaver/sleep throttles LM Studio (even fast models
hit the 240s cap with zero output). Documented in memory + CLAUDE.md:
runs need the Mac display AWAKE / sleep disabled.

Captured-but-ungraded outputs in outputs/ (Gemma26B, kat-coder,
qwen3-coder-30b all 5/5) await a grading session. Memory MCP DB was
down during handoff (will retry next session) — file-based memory +
CLAUDE.md carry the context either way.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-28 20:33:21 -07:00
co-authored by Claude
parent c98b6bb257
commit 8806c88a71
+24 -1
View File
@@ -88,7 +88,30 @@ Tokens: `mlx` = Apple MLX format; `gguf` = llama.cpp GGUF; quant suffixes like `
### When the user pastes raw output instead of giving a file
If the user pastes a model's output text directly (no file), **save it as a `.py` file in `outputs/`** using the naming convention above before grading. Ask the user for the model name + quant only if it cannot be reasonably inferred from context. Match the existing lowercase-hyphen style.
If the user pastes a model's output text directly (no file), **save it as a `.py` file in `outputs/`** using the naming convention above before grading. Ask the user for the model name + quant only if it cannot be reasonably inferred from context. Match the existing lowercase-hyphen style. **Note:** clipboard paste corrupts long outputs (strips `=` signs / indentation). Prefer the API capture workflow below.
## Capturing outputs via the LM Studio API (preferred — no clipboard)
`tools/grade_run.py` runs a prompt against LM Studio's local server, saves the raw completion to `outputs/`, and grabs metrics from the API response. **This is the preferred path** — it avoids clipboard corruption and auto-captures tok/sec + token counts. It also needs the full context of `prompts/aygea_test_battery.md` (the 6-prompt battery) and `PILLARS_BY_PROMPT` in `generate_dashboard.py` (each prompt grades on its own 5 pillars, tagged via the entry's `prompt_id`).
**Topology:** LM Studio runs on the Mac (LAN `10.0.0.31:1234`); the repo/dashboard run on mewtwo (`10.0.0.22`); mewtwo calls the Mac over LAN.
```
python3 tools/grade_run.py --resident # what's actually in RAM (read-only, loads nothing)
python3 tools/grade_run.py --unload-all # unload every resident model + verify clean
python3 tools/grade_run.py --model <id> --prompt <p> --name <slug> [--max-tokens N]
prompts: lfu tts webhook automation rust data mcp
```
**⚠ HARD MEMORY RULE:** the Mac has 48 GB; models are ~2830 GB. **Never have two big models resident.** Per model: confirm `--resident` is clean → run ALL its prompts while it's warm (models load from the NAS, so the first call is slow ~60 s) → `--unload-all` → verify clean → only then touch the next model. The script has a **memory guard** that aborts if a *different* model is resident when you ask to run a new one. `--resident` reads `GET /api/v1/models → loaded_instances[].id` (the only reliable resident list; `/v1/models` lies). Unload is `POST /api/v1/models/unload {"instance_id": <id>}`.
**Known model behavior** — capture the FAST ones (≥40 t/s); slow/looping models time out via the API and aren't worth batching:
- FAST & safe: `qwen/qwen3-coder-30b`, `qwen/qwen3.6-35b-a3b`, `kat-coder-v2.5-dev-xl-mlx`, `google/gemma-4-26b-a4b`.
- Hangs via API on most prompts: `qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive`.
- Slow / avoid: `qwen/qwen3.6-27b` (~12 min), the `gemma4-31b*` (1015 t/s).
- Loops without a stop token (endlessly "improving"): `zai-org/glm-4.7-flash` — needs `stop` tokens, not a longer timeout.
**Reasoning models** (kat-coder, qwen3-coder) burn tokens *thinking* — use `--max-tokens 16000+` or the output comes back empty. **TTFT capture is currently broken** (LM Studio returns empty `stats` on the OpenAI endpoint); the fix is switching to the native streaming `POST /api/v1/chat {"stream":true}` and parsing the `chat.end` event's `result.stats.time_to_first_token_seconds`.
## Hardware / runtime context