Gemma 4 26B-A4B head-to-head vs Qwen (4 prompts, via grade_run.py API)
Ran via tools/grade_run.py against LM Studio (no clipboard). Results: TTS: 80 Minor Flaws PASSES (real N-worker concurrency) <- Qwen 49, didn't parse Rust: 72 Minor Flaws COMPILES CLEAN (0 errs w/ deps) <- Qwen 50, 7 real errors Webhook: 55 Critical uses forbidden aiohttp (won't run) <- Qwen 75, passed Automation: 48 Critical SyntaxError (global-after-assign) <- first run for both DECISIVE head-to-head: Gemma generalizes where Qwen fails (TTS, Rust), but Qwen beats it on stdlib-discipline prompts (webhook). The two are COMPLEMENTARY local offloads, not redundant. Fixed: grade_run.py extractor (markdown/prose wrapping, multi-fence lang selection), TTFT-null handling in generator. TTFT capture from LM Studio API still needs the right stats key (left null + noted). Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -582,10 +582,12 @@ def render_detail(m, data):
|
||||
patch = m.get("patch_code", "")
|
||||
metrics_block = ""
|
||||
if m.get("tok_sec") is not None:
|
||||
ttft = m.get("ttft_sec")
|
||||
ttft_str = f"{ttft:.2f}s" if isinstance(ttft, (int, float)) else "—"
|
||||
metrics_block = f"""
|
||||
<div class="mini"><span class="mlbl">tok/sec</span><span class="mval mono" style="color:var(--mag)">{m['tok_sec']:.2f}</span></div>
|
||||
<div class="mini"><span class="mlbl">tokens</span><span class="mval mono">{m.get('total_tokens') or '—'}</span></div>
|
||||
<div class="mini"><span class="mlbl">TTFT</span><span class="mval mono">{m.get('ttft_sec'):.2f}s</span> </div>"""
|
||||
<div class="mini"><span class="mlbl">TTFT</span><span class="mval mono">{ttft_str}</span> </div>"""
|
||||
else:
|
||||
metrics_block = '<div class="mini" style="grid-column:1/-1"><span class="mlbl">runtime</span><span class="mval mono" style="color:var(--blue)">CLOUD — not measured</span></div>'
|
||||
|
||||
|
||||
Reference in New Issue
Block a user