Grade Qwen 6-bit on Rust prompt: 50/100 Critical (does not compile, 7 errors)

Third data point on the same model:
  Qwen 3.6 35B-A3B 6-bit MLX:
    LFU cache    82  (runs clean)
    TTS pipeline 49  (doesn't parse)
    Rust service 50  (7 compile errors)

Profile is now sharp: strong on single-file Python data-structure/ACID
work; repeatedly ships non-compiling/non-parsing code on multi-task async
and typed-language prompts. Keep it on Python ACID tasks; do NOT offload
Rust or async-pipeline work.

Verified with cargo 1.94 (mpsc::bounded hallucination, ownership moves,
dead test override, broken remove_watcher). Lang field added for non-Python.

Co-Authored-By: Claude <noreply@anthropic.com>

data/benchmark_history.json

@@ -435,6 +435,41 @@
         "Callback errors are silently swallowed (broad try/except Exception) \u2014 a silent-failure pattern; debug visibility lost."
       ],
       "patch_code": "# FIX 1 (the parse error): make on_event async (or don't await inside it).\n# Simplest correct version:\nasync def on_event(self, callback):\n    async with self._callbacks_lock:\n        self._callbacks.append(callback)\n\n# FIX 2 (real bounded concurrency): spawn N workers OR create_task per job\n# gated by the semaphore. Option B (concurrency from the semaphore itself):\nasync def _worker_loop(self):\n    while self._running:\n        job = await self._queue.get()\n        # do NOT hold the semaphore in the single worker; instead launch\n        # each job as its own task, gated so at most max_concurrency run:\n        async def _run(j):\n            async with self._semaphore:\n                await self._process_job(j)\n                self._queue.task_done()\n        asyncio.create_task(_run(job))\n# (and add a test that submits >max_concurrency long jobs and asserts\n# exactly max_concurrency run at once.)\n\n# FIX 3: reject submit() after stop() (guard on self._running).\n# FIX 4: bound _final_states (e.g. keep last N, or evict terminal >TTL).\n# FIX 5: log callback errors instead of swallowing them silently."
+    },
+    {
+      "id": "qwen3.6-35b-a3b-6bit-mlx-rust",
+      "prompt_id": "rust",
+      "timestamp": "2026-07-29T00:50:00Z",
+      "model_name": "Qwen 3.6 35B-A3B",
+      "quant": "6-bit MLX",
+      "param_size": "35B-A3B (MoE)",
+      "format": "mlx",
+      "lang": "rust",
+      "tok_sec": 68.99,
+      "total_tokens": 13310,
+      "ttft_sec": 1.07,
+      "filename": "outputs/qwen3.6-35b-a3b-6bit-mlx-rust.rs",
+      "tests_pass": false,
+      "total_score": 50,
+      "breakdown": {
+        "ownership": 8,
+        "concurrency": 13,
+        "error_handling": 12,
+        "cancellation": 11,
+        "test_integrity": 6
+      },
+      "verdict": "Critical Bugs",
+      "best_for": "NOT usable for Rust as-is \u2014 7 compile errors. Same model that scored 82 on LFU now: TTS 49, Rust 50. The profile is now clear: Qwen 6-bit is strong on single-file Python data-structure work but repeatedly ships non-compiling/non-parsing code on multi-task async + typed-language prompts. Keep it on Python ACID/data-structure tasks; do NOT offload Rust or async-pipeline work to it.",
+      "critical_bugs": [
+        "FATAL: does not compile \u2014 7 errors (verified with cargo 1.94). Key ones: mpsc::bounded(32) (tokio has no bounded(); should be mpsc::channel(32) \u2014 an async-std/flume API hallucination); no `main` fn (lib-style file, won't 'run directly' as the prompt required); u32/u64 type mismatch in Duration::from_millis(20 + id % 30).",
+        "Ownership errors (would not compile): shutdown() tries to move e.join_handle out of &WatcherEntry, and drop(self.item_tx)/drop(self.health_tx) out of &self. Needs Option<Sender> + &mut self or mem::take.",
+        "remove_watcher is broken: removes the HashMap entry but NEVER cancels that watcher's task. The orphaned task keeps polling until global shutdown() \u2014 contradicting the 'clean per-watcher removal' requirement.",
+        "Test (b) failure-injection is DEAD CODE: a global TEST_ALWAYS_FAIL flag is toggled in the test, but watcher_loop calls mock_fetch directly, not the test_fetch override that reads the flag. So the unhealthy-marking test relies on the natural ~15% failure rate over 400ms \u2014 flaky, may never reach 5 consecutive failures.",
+        "Test (a) asserts nothing real \u2014 the comment admits 'we trust the join'; prompt required asserting >0 items received and no hang.",
+        "mock_fetch randomness uses SystemTime nanos + id; across watchers polled in the same tick the high bits are shared, so failure/item counts cluster (poor randomness, not truly independent).",
+        "Several `let _ = tx.send(...)` silently swallow channel-closed errors."
+      ],
+      "patch_code": "// FIX 1 (the API hallucination): tokio mpsc has no bounded().\n//   let (item_tx, item_rx) = mpsc::bounded(32);\nlet (item_tx, item_rx) = mpsc::channel(32);\n\n// FIX 2 (type mismatch):\ntokio::time::sleep(Duration::from_millis(20 + (id as u64 % 30))).await;\n\n// FIX 3 (ownership in shutdown): store JoinHandles in Option + take them,\n// and make shutdown take &mut self (or hold senders in Option):\nstruct WatcherEntry { status: WatcherStatus, consecutive_failures: u32, join_handle: Option<JoinHandle<()>> }\n// in shutdown: let handles: Vec<_> = inner.watchers.values_mut().map(|e| e.join_handle.take()).flatten().collect();\n//   drop(self.item_tx.take()) etc. with Option<Sender> fields.\n\n// FIX 4 (remove_watcher must actually stop the task): either send on a per-watcher\n// oneshot/CancellationToken, or broadcast shutdown to that watcher's sub-channel.\n// Simplest: give each watcher a CancellationToken; remove_watcher cancels it, then awaits the handle.\n\n// FIX 5 (test isolation): inject the fetch fn into watcher_loop as a parameter so tests\n// can pass a failing mock; drop the dead global flag.\n// FIX 6: add `fn main() { ... }` or make it `cargo test`-only and document that."
     }
   ]
 }