Ran via tools/grade_run.py against LM Studio (no clipboard). Results:
TTS: 80 Minor Flaws PASSES (real N-worker concurrency) <- Qwen 49, didn't parse
Rust: 72 Minor Flaws COMPILES CLEAN (0 errs w/ deps) <- Qwen 50, 7 real errors
Webhook: 55 Critical uses forbidden aiohttp (won't run) <- Qwen 75, passed
Automation: 48 Critical SyntaxError (global-after-assign) <- first run for both
DECISIVE head-to-head: Gemma generalizes where Qwen fails (TTS, Rust),
but Qwen beats it on stdlib-discipline prompts (webhook). The two are
COMPLEMENTARY local offloads, not redundant.
Fixed: grade_run.py extractor (markdown/prose wrapping, multi-fence lang
selection), TTFT-null handling in generator. TTFT capture from LM Studio
API still needs the right stats key (left null + noted).
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces clipboard-based submission (which was mangling outputs).
Run from any LAN host that can reach LM Studio:
python3 tools/grade_run.py --lmstudio http://10.0.0.31:1234 --list
python3 tools/grade_run.py --model <id> --prompt tts --name <slug> --append
Saves the raw completion to outputs/<name>.<ext> (no clipboard),
captures tok/sec + tokens + TTFT from the API response (no manual
entry), and drafts a pending JSON entry. The 5-pillar audit still
done by hand (subjective). TODO_submission_tool.md updated.
Co-Authored-By: Claude <noreply@anthropic.com>
adminandClaude