Gemma 4 26B-A4B head-to-head vs Qwen (4 prompts, via grade_run.py API)

Ran via tools/grade_run.py against LM Studio (no clipboard). Results:
  TTS:        80 Minor Flaws  PASSES (real N-worker concurrency) <- Qwen 49, didn't parse
  Rust:       72 Minor Flaws  COMPILES CLEAN (0 errs w/ deps)     <- Qwen 50, 7 real errors
  Webhook:    55 Critical     uses forbidden aiohttp (won't run)  <- Qwen 75, passed
  Automation: 48 Critical     SyntaxError (global-after-assign)   <- first run for both

DECISIVE head-to-head: Gemma generalizes where Qwen fails (TTS, Rust),
but Qwen beats it on stdlib-discipline prompts (webhook). The two are
COMPLEMENTARY local offloads, not redundant.

Fixed: grade_run.py extractor (markdown/prose wrapping, multi-fence lang
selection), TTFT-null handling in generator. TTFT capture from LM Studio
API still needs the right stats key (left null + noted).

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-28 18:42:58 -07:00
co-authored by Claude
parent b82c665ec6
commit b9f45a7c46
7 changed files with 1310 additions and 8 deletions
+3 -1
View File
@@ -582,10 +582,12 @@ def render_detail(m, data):
patch = m.get("patch_code", "")
metrics_block = ""
if m.get("tok_sec") is not None:
ttft = m.get("ttft_sec")
ttft_str = f"{ttft:.2f}s" if isinstance(ttft, (int, float)) else ""
metrics_block = f"""
<div class="mini"><span class="mlbl">tok/sec</span><span class="mval mono" style="color:var(--mag)">{m['tok_sec']:.2f}</span></div>
<div class="mini"><span class="mlbl">tokens</span><span class="mval mono">{m.get('total_tokens') or ''}</span></div>
<div class="mini"><span class="mlbl">TTFT</span><span class="mval mono">{m.get('ttft_sec'):.2f}s</span> </div>"""
<div class="mini"><span class="mlbl">TTFT</span><span class="mval mono">{ttft_str}</span> </div>"""
else:
metrics_block = '<div class="mini" style="grid-column:1/-1"><span class="mlbl">runtime</span><span class="mval mono" style="color:var(--blue)">CLOUD — not measured</span></div>'