Via tools/grade_run.py against LM Studio (load -> all prompts -> unload -> verify).
kat-coder: lfu/tts/webhook/automation/rust (lfu has a real SyntaxError: 'self._ bump_freq')
qwen3-coder-30b: lfu/tts/webhook/automation/rust (all 4 py parse clean; terse ~2k tok outputs)
Grading deferred to a later session (capture-only mode).
Script hardening:
- --resident / --unload-all commands (read loaded_instances, POST unload, verify)
- MEMORY GUARD: aborts if a different model is resident (never stack 30GB models)
- TTFT self-discovery dump (LM Studio returns empty stats non-streaming;
fix next session: switch to /api/v1/chat stream:true -> chat.end result.stats)
Co-Authored-By: Claude <noreply@anthropic.com>
Ran via tools/grade_run.py against LM Studio (no clipboard). Results:
TTS: 80 Minor Flaws PASSES (real N-worker concurrency) <- Qwen 49, didn't parse
Rust: 72 Minor Flaws COMPILES CLEAN (0 errs w/ deps) <- Qwen 50, 7 real errors
Webhook: 55 Critical uses forbidden aiohttp (won't run) <- Qwen 75, passed
Automation: 48 Critical SyntaxError (global-after-assign) <- first run for both
DECISIVE head-to-head: Gemma generalizes where Qwen fails (TTS, Rust),
but Qwen beats it on stdlib-discipline prompts (webhook). The two are
COMPLEMENTARY local offloads, not redundant.
Fixed: grade_run.py extractor (markdown/prose wrapping, multi-fence lang
selection), TTFT-null handling in generator. TTFT capture from LM Studio
API still needs the right stats key (left null + noted).
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces clipboard-based submission (which was mangling outputs).
Run from any LAN host that can reach LM Studio:
python3 tools/grade_run.py --lmstudio http://10.0.0.31:1234 --list
python3 tools/grade_run.py --model <id> --prompt tts --name <slug> --append
Saves the raw completion to outputs/<name>.<ext> (no clipboard),
captures tok/sec + tokens + TTFT from the API response (no manual
entry), and drafts a pending JSON entry. The 5-pillar audit still
done by hand (subjective). TODO_submission_tool.md updated.
Co-Authored-By: Claude <noreply@anthropic.com>
adminandClaude