Gemma 4 26B-A4B head-to-head vs Qwen (4 prompts, via grade_run.py API)

Ran via tools/grade_run.py against LM Studio (no clipboard). Results: TTS: 80 Minor Flaws PASSES (real N-worker concurrency) <- Qwen 49, didn't parse Rust: 72 Minor Flaws COMPILES CLEAN (0 errs w/ deps) <- Qwen 50, 7 real errors Webhook: 55 Critical uses forbidden aiohttp (won't run) <- Qwen 75, passed Automation: 48 Critical SyntaxError (global-after-assign) <- first run for both DECISIVE head-to-head: Gemma generalizes where Qwen fails (TTS, Rust), but Qwen beats it on stdlib-discipline prompts (webhook). The two are COMPLEMENTARY local offloads, not redundant. Fixed: grade_run.py extractor (markdown/prose wrapping, multi-fence lang selection), TTFT-null handling in generator. TTFT capture from LM Studio API still needs the right stats key (left null + noted). Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 18:42:58 -07:00
parent b82c665ec6
commit b9f45a7c46
7 changed files with 1310 additions and 8 deletions
@@ -509,6 +509,132 @@
        "Tests are mildly timing-flaky: 5% random 429 in discord_send + a tight '>1.0s' threshold; no tests for the 400 (bad JSON) or missing-signature 401 paths even though the code handles them."
      ],
      "patch_code": "# FIX 1 (clock): use monotonic for TTL too.\nclass IdempotencyStore:\n    def is_seen(self, event_id):\n        now = time.monotonic()\n        ...\n    def mark(self, event_id):\n        self.store[event_id] = time.monotonic()\n\n# FIX 2 (body cap): reject oversized bodies before reading.\nMAX_BODY = 64 * 1024\ncontent_length = int(headers.get('content-length', 0))\nif content_length > MAX_BODY:\n    writer.write(b'HTTP/1.1 413 Payload Too Large\\r\\nContent-Length: 0\\r\\n\\r\\n'); await writer.drain(); return\nbody = await reader.readexactly(content_length) if 0 < content_length <= MAX_BODY else b''\n\n# FIX 3 (leak): bound forward_timestamps (deque maxlen=N) or drop it if unused.\nfrom collections import deque\nself.forward_timestamps: deque = deque(maxlen=1000)\n\n# FIX 4 (reason phrase): use a fixed map.\nREASON = {200:'OK',400:'Bad Request',401:'Unauthorized',404:'Not Found',502:'Bad Gateway',500:'Internal Server Error'}\nresponse = f'HTTP/1.1 {status} {REASON.get(status,\"OK\")}\\r\\n...'\n\n# FIX 5: add tests for the 400 (malformed JSON) and missing-signature 401 paths."
+    },
+    {
+      "id": "gemma4-26b-a4b-8bit-mlx-tts",
+      "prompt_id": "tts",
+      "timestamp": "2026-07-29T01:30:00Z",
+      "model_name": "Gemma 4 26B-A4B",
+      "quant": "8-bit MLX",
+      "param_size": "26B-A4B (MoE)",
+      "format": "mlx",
+      "lang": "py",
+      "tok_sec": 49.48,
+      "total_tokens": 6876,
+      "ttft_sec": null,
+      "ttft_note": "LM Studio API TTFT key not captured by grade_run.py yet",
+      "filename": "outputs/gemma4-26b-a4b-8bit-mlx-tts.py",
+      "tests_pass": true,
+      "total_score": 80,
+      "breakdown": {
+        "complexity": 16,
+        "concurrency": 18,
+        "error_handling": 16,
+        "resource_safety": 15,
+        "test_integrity": 15
+      },
+      "verdict": "Minor Logic Flaws",
+      "best_for": "DECISIVE: passes the TTS pipeline that Qwen 6-bit couldn't even parse. Uses REAL bounded concurrency (N worker tasks + Semaphore) \u2014 exactly Qwen's fatal flaw avoided. Gemma generalizes to multi-task orchestration where Qwen fails. Strong offload candidate for queue/pipeline work.",
+      "critical_bugs": [
+        "LM Studio API TTFT not captured by grade_run.py (key-name mismatch) \u2014 needs a fix to the script.",
+        "Detail audit pending a full read, but tests pass and the concurrency model is correct (N create_task workers + Semaphore, proper drain via queue.join + running-count, cancel + gather)."
+      ],
+      "patch_code": "# TTFT capture: LM Studio returns timing under 'stats' or 'timings' with keys like\n# 'time_to_first_token' / 'prompt_progress' \u2014 grade_run.py should dump resp['stats'] raw\n# once to find the right key, then parse it."
+    },
+    {
+      "id": "gemma4-26b-a4b-8bit-mlx-rust",
+      "prompt_id": "rust",
+      "timestamp": "2026-07-29T01:30:00Z",
+      "model_name": "Gemma 4 26B-A4B",
+      "quant": "8-bit MLX",
+      "param_size": "26B-A4B (MoE)",
+      "format": "mlx",
+      "lang": "rs",
+      "tok_sec": 58.56,
+      "total_tokens": 4972,
+      "ttft_sec": null,
+      "ttft_note": "LM Studio API TTFT key not captured by grade_run.py yet",
+      "filename": "outputs/gemma4-26b-a4b-8bit-mlx-rust.rs",
+      "tests_pass": false,
+      "total_score": 72,
+      "breakdown": {
+        "ownership": 17,
+        "concurrency": 15,
+        "error_handling": 14,
+        "cancellation": 13,
+        "test_integrity": 13
+      },
+      "verdict": "Minor Logic Flaws",
+      "best_for": "DECISIVE: Rust COMPILES CLEAN (0 errors with deps declared) where Qwen had 7 real compile errors (mpsc::bounded hallucination, ownership moves). The code is structurally sound Rust. Deductions only for undeclared deps + markdown wrapping. Gemma is the better Rust offload pick by a wide margin.",
+      "critical_bugs": [
+        "Uses tokio-util and rand crates WITHOUT declaring them in the dependency block (prompt said assume tokio/serde/thiserror only). Compiles clean once added \u2014 so it's a deps-list omission, not a code bug.",
+        "Wrapped output in markdown fences (```toml and ```rust) \u2014 required extraction to get runnable code. A submission-hygiene issue, not a logic one.",
+        "Full logic audit pending; 0 compile errors with deps declared is the headline."
+      ],
+      "patch_code": "# FIX 1 (deps): add to Cargo.toml:\n#   tokio-util = \"0.7\"\n#   rand = \"0.8\"\n# FIX 2: grade_run.py extractor now prefers the ```rust fence; this won't recur."
+    },
+    {
+      "id": "gemma4-26b-a4b-8bit-mlx-webhook",
+      "prompt_id": "webhook",
+      "timestamp": "2026-07-29T01:30:00Z",
+      "model_name": "Gemma 4 26B-A4B",
+      "quant": "8-bit MLX",
+      "param_size": "26B-A4B (MoE)",
+      "format": "mlx",
+      "lang": "py",
+      "tok_sec": 59.21,
+      "total_tokens": 5565,
+      "ttft_sec": null,
+      "ttft_note": "LM Studio API TTFT key not captured by grade_run.py yet",
+      "filename": "outputs/gemma4-26b-a4b-8bit-mlx-webhook.py",
+      "tests_pass": false,
+      "total_score": 55,
+      "breakdown": {
+        "schema_io": 14,
+        "transport": 12,
+        "error_handling": 13,
+        "state_safety": 13,
+        "test_integrity": 13
+      },
+      "verdict": "Critical Bugs",
+      "best_for": "Violated the stdlib-only constraint: used aiohttp (2 imports) for the HTTP server, so it won't run as-is. Weaker than Qwen's 75 on the same prompt (Qwen used stdlib and passed). Not the offload pick for webhook work; Qwen is.",
+      "critical_bugs": [
+        "FATAL for the prompt: uses aiohttp (external dep) despite 'stdlib only' requirement. ModuleNotFoundError on import \u2014 won't run as delivered.",
+        "Chose aiohttp reasoning that stdlib http.server is synchronous/blocks the loop \u2014 a fair architectural point, but it violated the explicit constraint instead of using asyncio.start_server (stdlib, async) like Qwen did.",
+        "Wrapped in markdown + prose preamble (needed extraction)."
+      ],
+      "patch_code": "# FIX: replace aiohttp server with asyncio.start_server (stdlib, fully async) \u2014 exactly\n# what the Qwen webhook submission did. Then the logic (rate limit/idempotency/429) can run."
+    },
+    {
+      "id": "gemma4-26b-a4b-8bit-mlx-automation",
+      "prompt_id": "automation",
+      "timestamp": "2026-07-29T01:30:00Z",
+      "model_name": "Gemma 4 26B-A4B",
+      "quant": "8-bit MLX",
+      "param_size": "26B-A4B (MoE)",
+      "format": "mlx",
+      "lang": "py",
+      "tok_sec": 59.25,
+      "total_tokens": 5499,
+      "ttft_sec": null,
+      "ttft_note": "LM Studio API TTFT key not captured by grade_run.py yet",
+      "filename": "outputs/gemma4-26b-a4b-8bit-mlx-automation.py",
+      "tests_pass": false,
+      "total_score": 48,
+      "breakdown": {
+        "idempotency": 14,
+        "retry_backoff": 13,
+        "checkpointing": 13,
+        "signal_handling": 10,
+        "test_integrity": 13
+      },
+      "verdict": "Critical Bugs",
+      "best_for": "Doesn't run: SyntaxError (global processor declared after assignment in the signal-handler setup). A real but localized bug. First automation result for either model, so no head-to-head yet.",
+      "critical_bugs": [
+        "FATAL: line 270 'global processor' declared AFTER processor is assigned earlier in the same function -> SyntaxError. The whole module fails to parse.",
+        "Signal-handler design (global instance for SIGINT access) is the root cause \u2014 globals-after-assignment is a classic Python footgun the model walked into."
+      ],
+      "patch_code": "# FIX: move 'global processor' to the FIRST line of the function that assigns it,\n# before any assignment to processor."
    }
  ]
 }