{
  "meta": {
    "project": "Local LLM Benchmark Suite \u2014 LFU Cache & ACID Audit",
    "machine": "Apple M3 Max, 48GB unified memory, LM Studio",
    "exam_prompt": "prompts/lfu_cache_prompt.txt",
    "grading_rubric": "prompts/grading.txt",
    "pillars": [
      "complexity",
      "concurrency",
      "isolation",
      "memory_edge_cases",
      "test_integrity"
    ],
    "max_per_pillar": 20,
    "schema_version": 1,
    "prompts": {
      "lfu": {
        "label": "LFU Cache + ACID (systems/async)",
        "file": "prompts/lfu_cache_prompt.txt"
      },
      "tts": {
        "label": "TTS Pipeline (queues/backpressure)",
        "file": "prompts/tts_pipeline.txt"
      },
      "mcp": {
        "label": "MCP Tool Server",
        "file": "prompts/mcp_server.txt"
      },
      "rust": {
        "label": "Rust tokio Service",
        "file": "prompts/rust_service.txt"
      },
      "data": {
        "label": "Data Service (pooling/SQL)",
        "file": "prompts/data_service.txt"
      },
      "automation": {
        "label": "Automation Glue (idempotent batch)",
        "file": "prompts/automation_glue.txt"
      },
      "webhook": {
        "label": "Webhook Bridge (HMAC/idempotency/rate-limit)",
        "file": "prompts/webhook_bridge.txt"
      }
    }
  },
  "models": [
    {
      "id": "qwen3.6-35b-a3b-6bit-mlx",
      "timestamp": "2026-07-28T13:00:00Z",
      "model_name": "Qwen 3.6 35B-A3B",
      "quant": "6-bit MLX",
      "format": "mlx",
      "tok_sec": 68.86,
      "total_tokens": 14883,
      "ttft_sec": 0.94,
      "filename": "outputs/qwen3.6-35b-a3b-6bit-mlx.py",
      "tests_pass": true,
      "total_score": 82,
      "breakdown": {
        "complexity": 16,
        "concurrency": 16,
        "isolation": 17,
        "memory_edge_cases": 16,
        "test_integrity": 17
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "Solid daily-driver scaffolding for ACID/async patterns \u2014 produces runnable, well-structured code, but needs a human pass for __slots__, monotonic clocks, and lock granularity before production.",
      "critical_bugs": [
        "No __slots__ declared on Node/Transaction/LFUCache \u2014 rubric explicitly required it for memory efficiency.",
        "Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments can cause premature/incorrect TTL eviction.",
        "_cleanup_freq_lists() performs a hidden O(F) scan (iterates all freq tiers + min()), called after every eviction, background batch, AND inside commit() \u2014 violates the strict O(1) requirement.",
        "Transaction commit holds the single cache lock across all write/delete/bump loops + cleanup \u2014 coarse-grained, blocks all readers for the whole commit window; no fine-grained locking.",
        "Lost-update risk: commit applies tx-local writes without any MVCC/version check, so a key modified by the background evictor or another committer between tx.get() and commit() is overwritten blindly.",
        "Tests dodge hard cases: 50-task stress uses unique keys with capacity 100, so no eviction-under-contention ever happens; no test for rollback-after-partial-application or mid-commit read isolation."
      ],
      "patch_code": "# FIX 1: Add __slots__ for memory efficiency\n@dataclass\nclass Node:\n    __slots__ = ('key', 'value', 'freq', 'expires_at', 'prev', 'next')\n    key: Any\n    value: Any\n    freq: int\n    expires_at: Optional[float]\n    prev: Optional['Node'] = None\n    next: Optional['Node'] = None\n\n# FIX 2: Use monotonic clock everywhere (get/put/_add_node/commit)\n#   time.time()  ->  time.monotonic()\n# e.g.\nexpires_at = time.monotonic() + ttl_seconds if ttl_seconds else None\n\n# FIX 3: Make _cleanup_freq_lists O(1) \u2014 bump min_freq incrementally\n# instead of recomputing min() across all tiers:\n# In _update_freq, when emptying the min_freq bucket, only bump min_freq\n# if you're evicting from it; otherwise leave it. Delete the global\n# min(self.freq_map.keys()) scan. For background sweeps, prune empty\n# buckets lazily on next _evict() rather than scanning proactively.\n\n# FIX 4: Shrink commit critical section \u2014 apply writes into a staging\n# structure under the lock, then release; or use per-bucket locks so\n# readers on unrelated keys aren't blocked.\n\n# FIX 5: Add MVCC version to Node; in commit, raise/abort if the\n# stored version != the version seen at tx.get() time (lost-update detect).",
      "prompt_id": "lfu"
    },
    {
      "id": "qwen3.6-35b-a3b-4bit-mlx",
      "timestamp": "2026-07-28T13:05:00Z",
      "model_name": "Qwen 3.6 35B-A3B",
      "quant": "4-bit MLX",
      "format": "mlx",
      "tok_sec": 83.31,
      "total_tokens": 13278,
      "ttft_sec": 0.73,
      "filename": "outputs/qwen3.6-35b-a3b-4bit-mlx.py",
      "tests_pass": false,
      "total_score": 57,
      "breakdown": {
        "complexity": 15,
        "concurrency": 15,
        "isolation": 13,
        "memory_edge_cases": 10,
        "test_integrity": 4
      },
      "verdict": "Critical Bugs",
      "best_for": "Not recommended for systems code as-is. The 4-bit quant degrades logic sharply vs the 6-bit sibling (82->57) \u2014 usable only for boilerplate/scaffolding drafts that a human will heavily rewrite.",
      "critical_bugs": [
        "FATAL: _evict() double-removes nodes \u2014 pop() already calls remove() (nulling node.prev/next), then _remove_node() calls remove() AGAIN -> AttributeError: 'NoneType' on first eviction. The cache cannot survive reaching capacity.",
        "Test suite never executes: test_lfu_eviction crashes at the first eviction, so the 'All tests passed' message is unreachable and assertions are effectively unverified.",
        "Background _eviction_loop materializes list(self.key_to_node.keys())[:50] every sweep \u2014 an O(N) linear scan, forbidden by the strict O(1) requirement.",
        "_FreqList.pop() has no empty-guard \u2014 calling pop() on an empty list dereferences self.head.next (the dummy tail) and corrupts the DLL.",
        "Transaction _apply_put applies the buffered value into the EXACT original_node captured at tx.put() time; if the global cache evicted/relocated that node between put and commit, you mutate a stale/dangling node (no MVCC/version check).",
        "Commit is not atomic across exceptions: a crash mid-_apply loop leaves half-applied global state with no rollback.",
        "Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP jumps corrupt TTL eviction."
      ],
      "patch_code": "# FIX 1 (the crash): _evict double-removes. pop() already unlinks,\n# so do NOT call _remove_node on a popped node. Either:\n#   (a) pop and then only delete the key_map entry + min_freq bookkeeping:\ndef _evict(self):\n    if not self.freq_to_list:\n        return\n    evict_list = self.freq_to_list[self.min_freq]\n    if evict_list.size == 0:        # guard against empty\n        del self.freq_to_list[self.min_freq]\n        return\n    node = evict_list.pop()         # pop() unlinks + nulls prev/next\n    del self.key_to_node[node.key]  # DON'T call _remove_node again\n    if self.freq_to_list[self.min_freq].size == 0:\n        del self.freq_to_list[self.min_freq]\n        self.min_freq += 1\n\n# FIX 2: _FreqList.pop empty-guard\ndef pop(self) -> _Node:\n    if self.size == 0:\n        raise IndexError('pop from empty _FreqList')\n    node = self.head.next\n    self.remove(node)\n    return node\n\n# FIX 3: kill the O(N) scan in background sweep \u2014 maintain a separate\n# set of keys that have a TTL, and iterate that set in batches:\nasync with self.lock:\n    batch = list(self._ttl_keys)[:50]\n    for k in batch:\n        node = self.key_to_node.get(k)\n        if node and 0 < node.expires_at <= time.monotonic():\n            self._remove_node(node)\n\n# FIX 4: time.time() -> time.monotonic() everywhere.\n# FIX 5: add node.version; in tx._apply_put, abort/refresh if\n# cache.key_to_node[key] is a different node than original_node.",
      "prompt_id": "lfu"
    },
    {
      "id": "qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf",
      "timestamp": "2026-07-28T13:10:00Z",
      "model_name": "Qwen 3.6 35B-A3B (uncensored hauhaucs aggressive)",
      "quant": "GGUF",
      "format": "gguf",
      "tok_sec": 62.54,
      "total_tokens": 13897,
      "ttft_sec": 1.09,
      "filename": "outputs/qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf.py",
      "tests_pass": false,
      "total_score": 49,
      "breakdown": {
        "complexity": 12,
        "concurrency": 10,
        "isolation": 12,
        "memory_edge_cases": 11,
        "test_integrity": 4
      },
      "verdict": "Critical Bugs",
      "best_for": "Not recommended for production code. Reasonable API shape and correctly used time.monotonic(), but the module does not parse (syntax error), contains an infinite while:pass loop, and has data races. Avoid for systems/concurrency work.",
      "critical_bugs": [
        "SyntaxError: line 305 'assert val := await cache.get(...)' is invalid Python \u2014 walrus operator cannot appear in an assert statement. The ENTIRE module fails to compile, so nothing runs and no test can execute.",
        "Infinite busy-loop: _evict_lfu lines 159-160 \u2014 'while self.min_freq in self.freq_map and self.min_freq < max(...): pass' has an empty body that never updates min_freq, recomputes max() (O(F)) each iteration, and can never terminate.",
        "Hidden O(F) scan: min(self.freq_map.keys()) / max(self.freq_map.keys()) appears at 6 call sites (every eviction and removal) \u2014 violates the strict O(1) requirement.",
        "Race condition: get() and put() perform lazy-TTL _remove_key() BEFORE acquiring the lock (lines 71-73, 107-108), mutating shared state unlocked while other coroutines read/write.",
        "Deadlock risk: background_loop holds self._lock, then calls await self._remove_key() which is itself a lock-acquiring method \u2014 asyncio.Lock is NOT reentrant -> deadlock when the evictor runs.",
        "No __slots__ on _Node despite using a dataclass (rubric required it for memory efficiency).",
        "_remove_key will KeyError on self.freq_map[freq] if a concurrent operation already deleted that bucket."
      ],
      "patch_code": "# FIX 1 (the parse error): assign first, then assert.\nval_d = await cache.get(\"D\")\nassert val_d, \"D should exist\"\nval_a = await cache.get(\"A\")\nassert val_a, \"A should exist (highest freq)\"\n\n# FIX 2: delete the broken while:pass loop. Bump min_freq incrementally:\n# only when the min_freq bucket empties, and only ever UP by 1 (a key\n# whose freq increased must land at min_freq+1). Never call min()/max().\nif not old_bucket:\n    del self.freq_map[old_freq]\n    if self.min_freq == old_freq:\n        self.min_freq += 1   # next tier up; never scan\n\n# FIX 3: do ALL lazy eviction INSIDE the lock, not before it:\nasync def get(self, key):\n    async with self._lock:\n        if self.ttl_map.get(key, inf) <= time.monotonic():\n            await self._remove_key(key)   # now locked\n            return None\n        ...\n\n# FIX 4: make _remove_key a non-locking private helper, called from\n# inside already-locked public methods, so background_loop doesn't try\n# to re-acquire the non-reentrant asyncio.Lock.\n\n# FIX 5: add __slots__ = ('key','value','ttl_expiry') to _Node.",
      "prompt_id": "lfu"
    },
    {
      "id": "kat-coder-v2.5-dev-xl-mlx",
      "timestamp": "2026-07-28T13:15:00Z",
      "model_name": "KAT-Coder v2.5 Dev XL",
      "quant": "MLX",
      "format": "mlx",
      "tok_sec": 65.26,
      "total_tokens": 6172,
      "ttft_sec": 7.42,
      "filename": "outputs/kat-coder-v2.5-dev-xl-mlx.py",
      "tests_pass": false,
      "total_score": 65,
      "breakdown": {
        "complexity": 15,
        "concurrency": 16,
        "isolation": 16,
        "memory_edge_cases": 13,
        "test_integrity": 5
      },
      "verdict": "Critical Bugs",
      "best_for": "Promising code-design instincts (cleanest abstractions and best transaction isolation design in the set) but undone by a single fatal one-line bug that stops it running. With the bug fixed it would likely score 80+; as-is, only useful as a structural reference.",
      "critical_bugs": [
        "FATAL: _put_internal line 305 inserts a NEW key with 'self._freq_map[1].push_front(...)' but never ensures the freq-1 bucket exists \u2014 KeyError: 1 on the very first put. The cache cannot store a single key. The _ensure_freq_list(1) helper it should use exists and is used correctly everywhere else (lines 294, 354).",
        "Transaction.commit() calls _put_internal for buffered writes, so it hits the same KeyError: 1 \u2014 committed transactions crash too.",
        "Test suite cannot execute: crashes at the first cache.put() in main(); the well-built test harness (pass/fail counter, 4 real scenarios) validates nothing.",
        "No __slots__ on _DLLNode/_CacheNode/_DoublyLinkedList despite the rubric requiring it for memory efficiency.",
        "_evict_node uses min(self._freq_map) (line 325) when the min-tier empties \u2014 a hidden O(F) scan, violating strict O(1).",
        "No MVCC/version check on transaction commit (lost-update possible if the global key is modified between tx.get and commit); commit is not exception-safe across the writes-vs-deletes loops."
      ],
      "patch_code": "# FIX 1 (the fatal one-liner): use the helper that already exists.\n# line 305, in _put_internal, new-key branch:\n- self._freq_map[1].push_front(dll_node)\n+ self._ensure_freq_list(1).push_front(dll_node)\n# (This single change makes the cache and transactions functional.)\n\n# FIX 2: replace the O(F) min() scan with an incremental bump:\n# in _evict_node, when the min-tier bucket empties, min_freq is the\n# lowest remaining tier. Since freq only ever increments by 1, the\n# next min is almost always min_freq+1; track it incrementally rather\n# than scanning. Or, since this only happens on full eviction, accept\n# O(F) but only on the empty-cache edge \u2014 document it.\n\n# FIX 3: add __slots__ to all internal classes:\nclass _CacheNode:\n    __slots__ = ('key','value','ttl_seconds','expiry_time','freq','dll_node')\n    ...\n\n# FIX 4: wrap commit applies in try/except so a mid-commit exception\n# does not leave a half-applied global state; consider abort semantics.\n# FIX 5: add node.version; in tx commit, abort if the global node for\n# a key is not the one seen at tx.get() time.",
      "prompt_id": "lfu"
    },
    {
      "id": "gemma4-31b-gguf",
      "timestamp": "2026-07-28T13:20:00Z",
      "model_name": "Gemma 4 31B",
      "quant": "GGUF",
      "format": "gguf",
      "tok_sec": 10.09,
      "total_tokens": 4536,
      "ttft_sec": 4.39,
      "speed_caveat": "All Gemma 4 models ran abnormally slow (GPU offload appeared inactive despite being set), so tok/sec and TTFT are NOT representative of the model itself \u2014 likely an LM Studio/GGUF config issue. Treat speed numbers for the Gemma 4 batch as suspect.",
      "filename": "outputs/gemma4-31b-gguf.py",
      "tests_pass": true,
      "total_score": 78,
      "breakdown": {
        "complexity": 17,
        "concurrency": 16,
        "isolation": 14,
        "memory_edge_cases": 15,
        "test_integrity": 16
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "Clean, correct, runnable code with solid O(1) structure and good concurrency granularity. A reliable pick for everyday caching/async work after a monotonic-clock + __slots__ pass.",
      "critical_bugs": [
        "Isolation leak: Transaction.get falls back to the PUBLIC cache.get, which calls _update_frequency \u2014 so reading a key inside a transaction mutates GLOBAL frequency state before commit, leaking uncommitted access patterns into global eviction order. Spec requires tx reads not to alter global freq.",
        "Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments corrupt TTL eviction.",
        "No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction \u2014 rubric required it for memory efficiency.",
        "_delete_internal deliberately leaves empty frequency buckets in freq_map (documented but a minor memory leak: empty DoublyLinkedList objects accumulate).",
        "Background evictor does list(self.cache.keys()) = O(N) snapshot every interval \u2014 a linear scan, forbidden by strict O(1).",
        "No MVCC/version check on commit (lost-update possible); commit is not exception-safe across the deletes-vs-puts loops.",
        "Tests pass but don't probe mid-commit read isolation or eviction-under-real-contention (capacity sized so all keys fit), so the isolation leak above goes undetected."
      ],
      "patch_code": "# FIX 1 (the isolation leak): tx reads must NOT mutate global freq.\n# Add a read-only global lookup (no _update_frequency) and use it in tx.get:\nasync def _read_raw(self, key):           # no freq bump\n    node = self.cache.get(key)\n    if node is None: return None\n    if time.monotonic() > node.expiry:\n        await self._delete_internal(key)\n        return None\n    return node.value\n# then in Transaction.get fallback:\n    return await self._cache._read_raw(key)   # NOT cache.get\n\n# FIX 2: time.time() -> time.monotonic() everywhere (get/put/_put_internal/bg loop).\n# FIX 3: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 4: prune empty freq buckets on delete, or have _evict_lfu drop them.\n# FIX 5: iterate a dedicated _ttl_keys set (batched) in the bg evictor\n# instead of list(self.cache.keys()) to stay O(batch), not O(N).",
      "prompt_id": "lfu"
    },
    {
      "id": "gemma-4-31b-qat-gguf",
      "timestamp": "2026-07-28T13:25:00Z",
      "model_name": "Gemma 4 31B QAT",
      "quant": "QAT GGUF",
      "format": "gguf",
      "tok_sec": 15.0,
      "total_tokens": 4552,
      "ttft_sec": 4.01,
      "speed_caveat": "Same as the Gemma 4 batch: GPU offload appeared inactive so tok/sec/TTFT are NOT representative of the model. Suspected LM Studio/GGUF config issue.",
      "filename": "outputs/gemma-4-31b-qat-gguf.py",
      "tests_pass": true,
      "total_score": 70,
      "breakdown": {
        "complexity": 16,
        "concurrency": 16,
        "isolation": 11,
        "memory_edge_cases": 13,
        "test_integrity": 14
      },
      "verdict": "Critical Bugs",
      "best_for": "Runnable and structurally sound, but the LFU eviction has a stale-min_freq capacity-breach path and the transaction API doesn't match the spec (no commit/rollback). Usable for prototypes if you fix eviction and re-skin transactions.",
      "critical_bugs": [
        "Capacity breach: eviction does self.freq_map[self.min_freq].pop_tail() with NO guard that the bucket exists or is non-empty, and never prunes emptied freq buckets. After manual deletes empty the min-tier, pop_tail returns None silently -> eviction fails -> cache grows PAST capacity. This is the exact stale-min_freq capacity-breach the rubric flags.",
        "Non-conformant transaction API: Transaction has NO commit() or rollback() method (both required by spec). Commit happens via a separate cache.apply_transaction_changes(tx._state) \u2014 wrong surface; the test only passes because it uses this internal path.",
        "Isolation leak: Transaction.get falls back to the public cache.get, which bumps global frequency before commit \u2014 uncommitted tx reads alter global eviction order.",
        "Duplicated eviction logic in apply_transaction_changes re-introduces the stale-min_freq bug at line 211.",
        "Uses time.time() (system clock) via _get_now() instead of time.monotonic() \u2014 NTP jumps corrupt TTL (though _get_now is a clean single fix point).",
        "No __slots__ on Node/DoublyLinkedList/LFUCache/TransactionState/Transaction.",
        "Background sweep does list(self.cache.keys()) = O(N) per interval.",
        "Tests pass but use the non-spec commit path and don't probe capacity breach or isolation leak."
      ],
      "patch_code": "# FIX 1 (capacity breach): guard + prune empty buckets on eviction:\nwhile self.min_freq in self.freq_map and self.freq_map[self.min_freq].size == 0:\n    del self.freq_map[self.min_freq]\n    self.min_freq += 1\n    if not self.freq_map:\n        break\nif self.min_freq not in self.freq_map:\n    return  # nothing to evict\nevicted = self.freq_map[self.min_freq].pop_tail()\nif evicted and self.freq_map[self.min_freq].size == 0:\n    del self.freq_map[self.min_freq]\n\n# FIX 2 (conformant API): add commit/rollback to Transaction:\nasync def commit(self):\n    await self._cache.apply_transaction_changes(self._state)\n    self._committed = True\ndef rollback(self):\n    self._state.writes.clear()\n    self._committed = True\n\n# FIX 3 (isolation leak): tx.get should use a read-only global lookup\n# (no _update_freq), not the public cache.get.\n# FIX 4: _get_now returns time.monotonic().\n# FIX 5: add __slots__ to all node/list classes.",
      "prompt_id": "lfu"
    },
    {
      "id": "gemma-4-12b-coder-heretic-mxfp8-mlx",
      "timestamp": "2026-07-28T13:30:00Z",
      "model_name": "Gemma 4 12B Coder (fable5-composer2.5-v1-uncensored-heretic merge)",
      "quant": "mxfp8 MLX",
      "format": "mlx",
      "tok_sec": 25.28,
      "total_tokens": 2625,
      "ttft_sec": 3.21,
      "filename": "outputs/gemma-4-12b-coder-fable5-composer2.5-v1-uncensored-heretic-mxfp8-mlx.py",
      "tests_pass": false,
      "total_score": 43,
      "breakdown": {
        "complexity": 13,
        "concurrency": 7,
        "isolation": 10,
        "memory_edge_cases": 9,
        "test_integrity": 4
      },
      "verdict": "Critical Bugs",
      "best_for": "Not usable as-is \u2014 the cache cannot store its first key and the background evictor would crash the event loop. The smallest model in the set (12B) and lowest-quality output. Avoid for systems work.",
      "critical_bugs": [
        "FATAL: put() line 112 does 'bucket = self.freq_buckets[self.min_freq]' after setting min_freq=1 but NEVER creates freq_buckets[1] -> KeyError: 1 on the very first put. Cache is unusable.",
        "Background evictor is fundamentally broken: start_evictor defines a SYNC 'def evict_loop' and passes it to create_task; inside it calls blocking time.sleep(interval) (freezes the event loop) AND asyncio.run(...) from within a running loop -> RuntimeError. Would crash hard if ever reached.",
        "Eviction-by-re-put: expired keys are 'evicted' by re-inserting them with TTL 0 (line 122) instead of deleting them \u2014 wrong semantics and triggers immediate re-eviction.",
        "Transaction duplicates the entire LFU machinery (local_cache + local_freq + local_min_freq) for snapshot isolation, but _update_local_freq has the same missing-bucket KeyError (line 140).",
        "Class name typo 'DoublyLinkedListList' (doubled word).",
        "No __slots__; time.time() (not monotonic) throughout.",
        "Tests cannot run \u2014 crash at first put."
      ],
      "patch_code": "# FIX 1 (the fatal KeyError): create the bucket before use.\n# In put(), new-key branch:\n- bucket = self.freq_buckets[self.min_freq]\n+ bucket = self.freq_buckets.setdefault(self.min_freq, DoublyLinkedListList())\n# Same fix in _update_freq and Transaction._update_local_freq (use setdefault).\n\n# FIX 2 (the broken evictor): make it a real async task that deletes:\nasync def _evict_loop(self, interval):\n    while True:\n        await asyncio.sleep(interval)          # async, non-blocking\n        now = time.monotonic()\n        async with self.global_lock:\n            expired = [k for k, n in list(self.cache.items()) if now > n.ttl_expiry]\n            for k in expired:\n                node = self.cache.pop(k, None)\n                if node:\n                    self.freq_buckets[node.freq].remove(node)   # DELETE, not re-put\n\nasync def start_evictor(self, interval=1.0):\n    self.evictor_task = asyncio.create_task(self._evict_loop(interval))\n\n# FIX 3: delete expired keys; do NOT re-insert with TTL 0.\n# FIX 4: time.time() -> time.monotonic().\n# FIX 5: add __slots__ to Node / DoublyLinkedListList / ConcurrentLFUCache / Transaction.",
      "prompt_id": "lfu"
    },
    {
      "id": "deepseekv4flash-cloud",
      "timestamp": "2026-07-28T13:35:00Z",
      "model_name": "DeepSeek V4 Flash (CLOUD baseline)",
      "quant": "n/a (cloud)",
      "format": "cloud",
      "tok_sec": null,
      "total_tokens": null,
      "ttft_sec": null,
      "speed_caveat": "Cloud model (run via opencode, not LM Studio) \u2014 tok_sec/tokens/TTFT are N/A (not measured for cloud). Included as a quality baseline against the local models. NOTE: it took 3 attempts to produce any output and ~12 minutes of thinking before succeeding \u2014 so it is a QUALITY benchmark, not a speed/usability one.",
      "filename": "deepseekv4flash.py",
      "tests_pass": true,
      "total_score": 91,
      "breakdown": {
        "complexity": 18,
        "concurrency": 18,
        "isolation": 19,
        "memory_edge_cases": 18,
        "test_integrity": 18
      },
      "verdict": "Production Ready",
      "best_for": "Reference-quality baseline (91/100) \u2014 the bar the local models are measured against. Only submission with __slots__ + time.monotonic() + delta-based transactional frequency accounting. Sets the ceiling for correctness, though its unreliability (3 attempts, 12-min think time) makes it a poor *local* daily-driver.",
      "critical_bugs": [
        "Two min(self._freq_to_list) linear scans in _evict_one's defensive recovery path (lines 415, 429) \u2014 only triggered when min_freq desyncs, not per-operation, but still a non-O(1) path. Could be replaced with incremental tracking.",
        "Single coarse lock held across the whole commit-apply loop \u2014 not the fine-grained locking the prompt asked for.",
        "__slots__ present on _Node and _DLL but not extended to Transaction / LFUCache.",
        "No explicit lost-update/conflict abort on commit (delta-based freq is applied unconditionally).",
        "Tests pass 20/20 but don't include a mid-commit read-isolation probe or adversarial eviction-under-contention stress."
      ],
      "patch_code": "# These are minor refinements on an already production-ready file.\n\n# FIX 1: eliminate the recovery min() scans by keeping min_freq\n# strictly in sync on every insert/bump/remove (it already does on\n# the hot path), so the _evict_one recovery branch is unreachable and\n# can assert instead of scanning:\nassert self._min_freq in self._freq_to_list or not self._freq_to_list\n\n# FIX 2: extend __slots__ to Transaction and LFUCache.\nclass LFUCache(Generic[KT, VT]):\n    __slots__ = ('_capacity','_key_to_node','_freq_to_list','_min_freq',\n                 '_lock','_ttl_index','_evictor_task','_closed')\n\n# FIX 3 (optional): on commit, if a key's global node changed since the\n# tx snapshot, raise LFUCacheError('lost update') instead of overwriting.",
      "prompt_id": "lfu"
    },
    {
      "id": "gemma4-26b-a4b-8bit-mlx",
      "timestamp": "2026-07-28T16:25:00Z",
      "model_name": "Gemma 4 26B-A4B",
      "quant": "8-bit",
      "param_size": "26B-A4B (MoE)",
      "format": "mlx",
      "tok_sec": 58.3,
      "total_tokens": 7390,
      "ttft_sec": 0.93,
      "filename": "outputs/gemma4-26b-a4b-8bit-mlx.py",
      "tests_pass": true,
      "total_score": 82,
      "breakdown": {
        "complexity": 17,
        "concurrency": 16,
        "isolation": 18,
        "memory_edge_cases": 15,
        "test_integrity": 16
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "Tied top local scorer (82). The only local model to use delta-based transactional frequency accounting (freq bumps deferred to commit), matching the cloud baseline's isolation approach. Reliable for async/ACID-pattern work after a __slots__ + min_freq-edge pass.",
      "critical_bugs": [
        "Stale min_freq on manual delete: _remove_node_from_structures empties the min-freq bucket but does `pass` instead of recomputing min_freq (lines 199-202, documented as 'a simplification'). Correct only because eviction has a min_freq-in-freq_map guard + arbitrary-key fallback (lines 237-244) \u2014 latent fragility under concurrent deletes.",
        "No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction despite Generic dataclasses (rubric required it for memory efficiency).",
        "Background evictor does list(self.cache_data.keys()) = O(N) snapshot every interval \u2014 a linear scan, forbidden by strict O(1).",
        "No MVCC/version check on commit (lost-update possible if the global key changes between tx.get and commit); commit applies deletes->reads->puts without try/except, so a mid-commit exception leaves partial state."
      ],
      "patch_code": "# FIX 1 (stale min_freq): recompute or invalidate when the min bucket empties.\n# In _remove_node_from_structures, replace the `pass`:\nif dll.size == 0:\n    del self.freq_map[node.freq]\n    if self.min_freq == node.freq:\n        # bump to next existing tier (frequencies are contiguous under normal use)\n        self.min_freq = self.min_freq + 1 if (self.min_freq + 1) in self.freq_map else min(self.freq_map, default=1)\n\n# FIX 2: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 3: background evictor \u2014 maintain a _ttl_keys set and iterate IT in\n# batches instead of list(self.cache_data.keys()) to stay O(batch).\n# FIX 4: wrap commit's three loops in try/except with rollback semantics on failure.",
      "prompt_id": "lfu"
    },
    {
      "id": "qwen3.6-27b-8bit-mlx",
      "timestamp": "2026-07-28T16:30:00Z",
      "model_name": "Qwen 3.6 27B",
      "quant": "8-bit",
      "param_size": "27B dense",
      "format": "mlx",
      "tok_sec": 12.29,
      "total_tokens": 12790,
      "ttft_sec": 2.9,
      "speed_caveat": "This model 'thought' for 12m48s before producing output and ran at 12.29 tok/sec \u2014 anomalously slow for an 8-bit MLX on M3 Max. Likely an inference/quant issue worth investigating; the slow generation is NOT representative of normal 27B-8bit throughput.",
      "filename": "outputs/qwen3.6-27b-8bit-mlx.py",
      "tests_pass": true,
      "total_score": 78,
      "breakdown": {
        "complexity": 17,
        "concurrency": 16,
        "isolation": 14,
        "memory_edge_cases": 15,
        "test_integrity": 16
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "Clean, correct, runnable \u2014 same tier as Gemma 4 31B (78). Good O(1) structure and concurrency granularity. Reliable for everyday async/caching work after a monotonic-clock + __slots__ + isolation pass. Caveat: was anomalously slow to generate.",
      "critical_bugs": [
        "Isolation leak: Transaction.get falls back to the PUBLIC cache.get (line 78), which calls _update_freq \u2014 so reading a key inside a transaction mutates GLOBAL frequency state before commit, leaking uncommitted access patterns into global eviction order.",
        "Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments corrupt TTL eviction.",
        "No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction \u2014 rubric required it for memory efficiency.",
        "Background _evict_loop materializes list(self.nodes.keys()) (O(N)) before checking only batch_size keys \u2014 the break caps work but not the snapshot cost, a hidden O(N) per sweep.",
        "Commit re-implements put+evict inline (lines 102-118) duplicating the public path = duplicated bug surface; not wrapped in try/except so a mid-commit exception leaves partial state. No MVCC version check (lost-update possible)."
      ],
      "patch_code": "# FIX 1 (isolation leak): add a read-only global lookup (no freq bump)\n# and use it in tx.get instead of the public cache.get:\nasync def _read_raw(self, key):\n    async with self.lock:\n        node = self.nodes.get(key)\n        if node is None: return None\n        if time.monotonic() > node.expires_at:\n            self._remove_node(key); return None\n        return node.value\n# then:  return await self._cache._read_raw(key)\n# FIX 2: time.time() -> time.monotonic() everywhere.\n# FIX 3: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 4: maintain a _ttl_keys set; iterate IT (batched) in the bg loop\n# instead of list(self.nodes.keys()).\n# FIX 5: factor commit's put/evict to reuse the internal helpers; wrap\n# the commit loop in try/except with rollback-on-failure.",
      "prompt_id": "lfu"
    },
    {
      "id": "qwen3-coder-30b-6bit-mlx",
      "timestamp": "2026-07-28T16:35:00Z",
      "model_name": "Qwen3 Coder 30B",
      "quant": "6-bit",
      "param_size": "30B",
      "format": "mlx",
      "tok_sec": 72.7,
      "total_tokens": 2779,
      "ttft_sec": 0.9,
      "filename": "outputs/qwen3-coder-30b-6bit-mlx.py",
      "tests_pass": false,
      "total_score": 50,
      "breakdown": {
        "complexity": 15,
        "concurrency": 9,
        "isolation": 11,
        "memory_edge_cases": 11,
        "test_integrity": 4
      },
      "verdict": "Critical Bugs",
      "best_for": "Not usable as-is \u2014 transactions crash immediately due to an async/sync lock mismatch. The coder-specialist produced terse, fast output (2779 tok, 72.7 t/s) with competent freq-bucket structure, but fumbled the async primitive. Fix the one lock bug and it would likely score 70+.",
      "critical_bugs": [
        "FATAL: _transaction_lock is an asyncio.Lock() (line 101) but begin_transaction() is a SYNC def that uses synchronous `with self._transaction_lock:` (line 199). asyncio.Lock does not support the sync context-manager protocol -> TypeError on the first transaction, crashing the entire test suite.",
        "Test suite cannot run: crashes at cache.begin_transaction() in main(); the transaction and concurrency assertions never execute.",
        "Unused `import threading` and `import weakref` \u2014 vestigial confusion between threading and asyncio primitives.",
        "Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP jumps corrupt TTL eviction.",
        "No __slots__ on CacheNode/FrequencyBucket/InMemoryLFUCache/Transaction \u2014 rubric required it.",
        "FrequencyBucket stores nodes in BOTH a DLL and a parallel `nodes` dict \u2014 redundant memory per bucket."
      ],
      "patch_code": "# FIX 1 (the fatal crash): make begin_transaction async and use async with.\nasync def begin_transaction(self) -> 'Transaction':\n    async with self._transaction_lock:\n        self._transaction_counter += 1\n        tx = Transaction(self)\n        self._transactions[self._transaction_counter] = tx\n    return tx\n# (and update callers: `tx = await cache.begin_transaction()`)\n# Alternative if sync creation is required: use threading.Lock for the\n# counter, but that is wrong in an asyncio codebase \u2014 go async.\n\n# FIX 2: remove unused `import threading` and `import weakref`.\n# FIX 3: time.time() -> time.monotonic() everywhere.\n# FIX 4: add __slots__ to all classes.\n# FIX 5: drop the redundant FrequencyBucket.nodes dict; the DLL already\n# tracks membership, so the dict is duplicate storage.",
      "prompt_id": "lfu"
    },
    {
      "id": "qwen3.6-35b-a3b-6bit-mlx-tts",
      "prompt_id": "tts",
      "timestamp": "2026-07-29T00:25:00Z",
      "model_name": "Qwen 3.6 35B-A3B",
      "quant": "6-bit MLX",
      "param_size": "35B-A3B (MoE)",
      "format": "mlx",
      "tok_sec": 69.87,
      "total_tokens": 11996,
      "ttft_sec": 0.84,
      "filename": "outputs/qwen3.6-35b-a3b-6bit-mlx-tts.py",
      "tests_pass": false,
      "total_score": 49,
      "breakdown": {
        "complexity": 10,
        "concurrency": 8,
        "error_handling": 13,
        "resource_safety": 12,
        "test_integrity": 6
      },
      "verdict": "Critical Bugs",
      "best_for": "NOT a safe offload for async-pipeline / queue work despite scoring 82 on the LFU exam. Same model+quant, different prompt: 82 -> 49. Use it for data-structure/ACID tasks; route queue/backpressure/retry work elsewhere (or to the cloud model).",
      "critical_bugs": [
        "FATAL: file does not parse. 'async with self._callbacks_lock' (L103) and 'await coro' (L98) are inside on_event, a SYNC def. SyntaxError before any test runs.",
        "Bounded-concurrency design is broken: only ONE worker task is created (start() does a single create_task), and the Semaphore(4) is acquired inside that single loop. Effective concurrency is 1, not 4 \u2014 the semaphore is decorative.",
        "Test A asserts max_concurrency <= 4, but since real concurrency is ~1 it passes trivially and proves NOTHING about the cap actually holding under saturation.",
        "Cooperative cancel only sets a flag; it cannot interrupt an in-flight mock_synthesize mid-sleep (acceptable, but not 'cancel at next safe checkpoint' for a long synth).",
        "submit() after stop() silently enqueues to a dead worker \u2014 no rejection, jobs leak (never processed, never failed).",
        "_final_states dict grows unbounded (no eviction) \u2014 memory leak for a long-running service.",
        "Callback errors are silently swallowed (broad try/except Exception) \u2014 a silent-failure pattern; debug visibility lost."
      ],
      "patch_code": "# FIX 1 (the parse error): make on_event async (or don't await inside it).\n# Simplest correct version:\nasync def on_event(self, callback):\n    async with self._callbacks_lock:\n        self._callbacks.append(callback)\n\n# FIX 2 (real bounded concurrency): spawn N workers OR create_task per job\n# gated by the semaphore. Option B (concurrency from the semaphore itself):\nasync def _worker_loop(self):\n    while self._running:\n        job = await self._queue.get()\n        # do NOT hold the semaphore in the single worker; instead launch\n        # each job as its own task, gated so at most max_concurrency run:\n        async def _run(j):\n            async with self._semaphore:\n                await self._process_job(j)\n                self._queue.task_done()\n        asyncio.create_task(_run(job))\n# (and add a test that submits >max_concurrency long jobs and asserts\n# exactly max_concurrency run at once.)\n\n# FIX 3: reject submit() after stop() (guard on self._running).\n# FIX 4: bound _final_states (e.g. keep last N, or evict terminal >TTL).\n# FIX 5: log callback errors instead of swallowing them silently."
    },
    {
      "id": "qwen3.6-35b-a3b-6bit-mlx-rust",
      "prompt_id": "rust",
      "timestamp": "2026-07-29T00:50:00Z",
      "model_name": "Qwen 3.6 35B-A3B",
      "quant": "6-bit MLX",
      "param_size": "35B-A3B (MoE)",
      "format": "mlx",
      "lang": "rust",
      "tok_sec": 68.99,
      "total_tokens": 13310,
      "ttft_sec": 1.07,
      "filename": "outputs/qwen3.6-35b-a3b-6bit-mlx-rust.rs",
      "tests_pass": false,
      "total_score": 50,
      "breakdown": {
        "ownership": 8,
        "concurrency": 13,
        "error_handling": 12,
        "cancellation": 11,
        "test_integrity": 6
      },
      "verdict": "Critical Bugs",
      "best_for": "NOT usable for Rust as-is \u2014 7 compile errors. Same model that scored 82 on LFU now: TTS 49, Rust 50. The profile is now clear: Qwen 6-bit is strong on single-file Python data-structure work but repeatedly ships non-compiling/non-parsing code on multi-task async + typed-language prompts. Keep it on Python ACID/data-structure tasks; do NOT offload Rust or async-pipeline work to it.",
      "critical_bugs": [
        "FATAL: does not compile \u2014 7 errors (verified with cargo 1.94). Key ones: mpsc::bounded(32) (tokio has no bounded(); should be mpsc::channel(32) \u2014 an async-std/flume API hallucination); no `main` fn (lib-style file, won't 'run directly' as the prompt required); u32/u64 type mismatch in Duration::from_millis(20 + id % 30).",
        "Ownership errors (would not compile): shutdown() tries to move e.join_handle out of &WatcherEntry, and drop(self.item_tx)/drop(self.health_tx) out of &self. Needs Option<Sender> + &mut self or mem::take.",
        "remove_watcher is broken: removes the HashMap entry but NEVER cancels that watcher's task. The orphaned task keeps polling until global shutdown() \u2014 contradicting the 'clean per-watcher removal' requirement.",
        "Test (b) failure-injection is DEAD CODE: a global TEST_ALWAYS_FAIL flag is toggled in the test, but watcher_loop calls mock_fetch directly, not the test_fetch override that reads the flag. So the unhealthy-marking test relies on the natural ~15% failure rate over 400ms \u2014 flaky, may never reach 5 consecutive failures.",
        "Test (a) asserts nothing real \u2014 the comment admits 'we trust the join'; prompt required asserting >0 items received and no hang.",
        "mock_fetch randomness uses SystemTime nanos + id; across watchers polled in the same tick the high bits are shared, so failure/item counts cluster (poor randomness, not truly independent).",
        "Several `let _ = tx.send(...)` silently swallow channel-closed errors."
      ],
      "patch_code": "// FIX 1 (the API hallucination): tokio mpsc has no bounded().\n//   let (item_tx, item_rx) = mpsc::bounded(32);\nlet (item_tx, item_rx) = mpsc::channel(32);\n\n// FIX 2 (type mismatch):\ntokio::time::sleep(Duration::from_millis(20 + (id as u64 % 30))).await;\n\n// FIX 3 (ownership in shutdown): store JoinHandles in Option + take them,\n// and make shutdown take &mut self (or hold senders in Option):\nstruct WatcherEntry { status: WatcherStatus, consecutive_failures: u32, join_handle: Option<JoinHandle<()>> }\n// in shutdown: let handles: Vec<_> = inner.watchers.values_mut().map(|e| e.join_handle.take()).flatten().collect();\n//   drop(self.item_tx.take()) etc. with Option<Sender> fields.\n\n// FIX 4 (remove_watcher must actually stop the task): either send on a per-watcher\n// oneshot/CancellationToken, or broadcast shutdown to that watcher's sub-channel.\n// Simplest: give each watcher a CancellationToken; remove_watcher cancels it, then awaits the handle.\n\n// FIX 5 (test isolation): inject the fetch fn into watcher_loop as a parameter so tests\n// can pass a failing mock; drop the dead global flag.\n// FIX 6: add `fn main() { ... }` or make it `cargo test`-only and document that."
    },
    {
      "id": "qwen3.6-35b-a3b-6bit-mlx-webhook",
      "prompt_id": "webhook",
      "timestamp": "2026-07-29T01:05:00Z",
      "model_name": "Qwen 3.6 35B-A3B",
      "quant": "6-bit MLX",
      "param_size": "35B-A3B (MoE)",
      "format": "mlx",
      "lang": "python",
      "tok_sec": 69.24,
      "total_tokens": 12595,
      "ttft_sec": 0.95,
      "filename": "outputs/qwen3.6-35b-a3b-6bit-mlx-webhook.py",
      "tests_pass": true,
      "total_score": 75,
      "breakdown": {
        "schema_io": 16,
        "transport": 14,
        "error_handling": 16,
        "state_safety": 14,
        "test_integrity": 15
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "Best non-LFU result for this model (75 vs TTS 49, Rust 50). Runs clean, passes all 4 tests, implements HMAC + idempotency + token-bucket rate-limit + 429 backoff correctly. Safe to offload single-handler HTTP/bridge logic (webhooks, signature verification, rate-limited forwarding). AVOID for multi-task orchestration (TTS) and typed/compiled languages (Rust).",
      "critical_bugs": [
        "Clock inconsistency: IdempotencyStore uses time.time() (system clock) while TokenBucketLimiter uses time.monotonic() \u2014 an NTP jump could wrongly expire/replay events. Should be monotonic everywhere.",
        "No max-body cap: handle_client does reader.readexactly(content_length) with no limit \u2014 a hostile Content-Length could force a huge allocation (DoS). The rate limiter doesn't protect pre-parse.",
        "forward_timestamps list grows unbounded (append-only, only cleared in tests) \u2014 memory leak for a long-running service.",
        "HTTP reason phrase is the raw message string (HTTP/1.1 200 Forwarded) \u2014 works for the bundled test client but is not valid HTTP for real clients/proxies.",
        "No Content-Type validation on incoming requests (accepts any).",
        "Idempotency eviction is lazy (only on is_seen) \u2014 a quiet store retains stale entries until next access; not a leak in steady state but imperfect.",
        "Tests are mildly timing-flaky: 5% random 429 in discord_send + a tight '>1.0s' threshold; no tests for the 400 (bad JSON) or missing-signature 401 paths even though the code handles them."
      ],
      "patch_code": "# FIX 1 (clock): use monotonic for TTL too.\nclass IdempotencyStore:\n    def is_seen(self, event_id):\n        now = time.monotonic()\n        ...\n    def mark(self, event_id):\n        self.store[event_id] = time.monotonic()\n\n# FIX 2 (body cap): reject oversized bodies before reading.\nMAX_BODY = 64 * 1024\ncontent_length = int(headers.get('content-length', 0))\nif content_length > MAX_BODY:\n    writer.write(b'HTTP/1.1 413 Payload Too Large\\r\\nContent-Length: 0\\r\\n\\r\\n'); await writer.drain(); return\nbody = await reader.readexactly(content_length) if 0 < content_length <= MAX_BODY else b''\n\n# FIX 3 (leak): bound forward_timestamps (deque maxlen=N) or drop it if unused.\nfrom collections import deque\nself.forward_timestamps: deque = deque(maxlen=1000)\n\n# FIX 4 (reason phrase): use a fixed map.\nREASON = {200:'OK',400:'Bad Request',401:'Unauthorized',404:'Not Found',502:'Bad Gateway',500:'Internal Server Error'}\nresponse = f'HTTP/1.1 {status} {REASON.get(status,\"OK\")}\\r\\n...'\n\n# FIX 5: add tests for the 400 (malformed JSON) and missing-signature 401 paths."
    },
    {
      "id": "gemma4-26b-a4b-8bit-mlx-tts",
      "prompt_id": "tts",
      "timestamp": "2026-07-29T01:30:00Z",
      "model_name": "Gemma 4 26B-A4B",
      "quant": "8-bit MLX",
      "param_size": "26B-A4B (MoE)",
      "format": "mlx",
      "lang": "py",
      "tok_sec": 49.48,
      "total_tokens": 6876,
      "ttft_sec": null,
      "ttft_note": "LM Studio API TTFT key not captured by grade_run.py yet",
      "filename": "outputs/gemma4-26b-a4b-8bit-mlx-tts.py",
      "tests_pass": true,
      "total_score": 80,
      "breakdown": {
        "complexity": 16,
        "concurrency": 18,
        "error_handling": 16,
        "resource_safety": 15,
        "test_integrity": 15
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "DECISIVE: passes the TTS pipeline that Qwen 6-bit couldn't even parse. Uses REAL bounded concurrency (N worker tasks + Semaphore) \u2014 exactly Qwen's fatal flaw avoided. Gemma generalizes to multi-task orchestration where Qwen fails. Strong offload candidate for queue/pipeline work.",
      "critical_bugs": [
        "LM Studio API TTFT not captured by grade_run.py (key-name mismatch) \u2014 needs a fix to the script.",
        "Detail audit pending a full read, but tests pass and the concurrency model is correct (N create_task workers + Semaphore, proper drain via queue.join + running-count, cancel + gather)."
      ],
      "patch_code": "# TTFT capture: LM Studio returns timing under 'stats' or 'timings' with keys like\n# 'time_to_first_token' / 'prompt_progress' \u2014 grade_run.py should dump resp['stats'] raw\n# once to find the right key, then parse it."
    },
    {
      "id": "gemma4-26b-a4b-8bit-mlx-rust",
      "prompt_id": "rust",
      "timestamp": "2026-07-29T01:30:00Z",
      "model_name": "Gemma 4 26B-A4B",
      "quant": "8-bit MLX",
      "param_size": "26B-A4B (MoE)",
      "format": "mlx",
      "lang": "rs",
      "tok_sec": 58.56,
      "total_tokens": 4972,
      "ttft_sec": null,
      "ttft_note": "LM Studio API TTFT key not captured by grade_run.py yet",
      "filename": "outputs/gemma4-26b-a4b-8bit-mlx-rust.rs",
      "tests_pass": false,
      "total_score": 72,
      "breakdown": {
        "ownership": 17,
        "concurrency": 15,
        "error_handling": 14,
        "cancellation": 13,
        "test_integrity": 13
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "DECISIVE: Rust COMPILES CLEAN (0 errors with deps declared) where Qwen had 7 real compile errors (mpsc::bounded hallucination, ownership moves). The code is structurally sound Rust. Deductions only for undeclared deps + markdown wrapping. Gemma is the better Rust offload pick by a wide margin.",
      "critical_bugs": [
        "Uses tokio-util and rand crates WITHOUT declaring them in the dependency block (prompt said assume tokio/serde/thiserror only). Compiles clean once added \u2014 so it's a deps-list omission, not a code bug.",
        "Wrapped output in markdown fences (```toml and ```rust) \u2014 required extraction to get runnable code. A submission-hygiene issue, not a logic one.",
        "Full logic audit pending; 0 compile errors with deps declared is the headline."
      ],
      "patch_code": "# FIX 1 (deps): add to Cargo.toml:\n#   tokio-util = \"0.7\"\n#   rand = \"0.8\"\n# FIX 2: grade_run.py extractor now prefers the ```rust fence; this won't recur."
    },
    {
      "id": "gemma4-26b-a4b-8bit-mlx-webhook",
      "prompt_id": "webhook",
      "timestamp": "2026-07-29T01:30:00Z",
      "model_name": "Gemma 4 26B-A4B",
      "quant": "8-bit MLX",
      "param_size": "26B-A4B (MoE)",
      "format": "mlx",
      "lang": "py",
      "tok_sec": 59.21,
      "total_tokens": 5565,
      "ttft_sec": null,
      "ttft_note": "LM Studio API TTFT key not captured by grade_run.py yet",
      "filename": "outputs/gemma4-26b-a4b-8bit-mlx-webhook.py",
      "tests_pass": false,
      "total_score": 55,
      "breakdown": {
        "schema_io": 14,
        "transport": 12,
        "error_handling": 13,
        "state_safety": 13,
        "test_integrity": 13
      },
      "verdict": "Critical Bugs",
      "best_for": "Violated the stdlib-only constraint: used aiohttp (2 imports) for the HTTP server, so it won't run as-is. Weaker than Qwen's 75 on the same prompt (Qwen used stdlib and passed). Not the offload pick for webhook work; Qwen is.",
      "critical_bugs": [
        "FATAL for the prompt: uses aiohttp (external dep) despite 'stdlib only' requirement. ModuleNotFoundError on import \u2014 won't run as delivered.",
        "Chose aiohttp reasoning that stdlib http.server is synchronous/blocks the loop \u2014 a fair architectural point, but it violated the explicit constraint instead of using asyncio.start_server (stdlib, async) like Qwen did.",
        "Wrapped in markdown + prose preamble (needed extraction)."
      ],
      "patch_code": "# FIX: replace aiohttp server with asyncio.start_server (stdlib, fully async) \u2014 exactly\n# what the Qwen webhook submission did. Then the logic (rate limit/idempotency/429) can run."
    },
    {
      "id": "gemma4-26b-a4b-8bit-mlx-automation",
      "prompt_id": "automation",
      "timestamp": "2026-07-29T01:30:00Z",
      "model_name": "Gemma 4 26B-A4B",
      "quant": "8-bit MLX",
      "param_size": "26B-A4B (MoE)",
      "format": "mlx",
      "lang": "py",
      "tok_sec": 59.25,
      "total_tokens": 5499,
      "ttft_sec": null,
      "ttft_note": "LM Studio API TTFT key not captured by grade_run.py yet",
      "filename": "outputs/gemma4-26b-a4b-8bit-mlx-automation.py",
      "tests_pass": false,
      "total_score": 48,
      "breakdown": {
        "idempotency": 14,
        "retry_backoff": 13,
        "checkpointing": 13,
        "signal_handling": 10,
        "test_integrity": 13
      },
      "verdict": "Critical Bugs",
      "best_for": "Doesn't run: SyntaxError (global processor declared after assignment in the signal-handler setup). A real but localized bug. First automation result for either model, so no head-to-head yet.",
      "critical_bugs": [
        "FATAL: line 270 'global processor' declared AFTER processor is assigned earlier in the same function -> SyntaxError. The whole module fails to parse.",
        "Signal-handler design (global instance for SIGINT access) is the root cause \u2014 globals-after-assignment is a classic Python footgun the model walked into."
      ],
      "patch_code": "# FIX: move 'global processor' to the FIRST line of the function that assigns it,\n# before any assignment to processor."
    }
  ]
}