modelTesting/data/benchmark_history.json

{
  "meta": {
    "project": "Local LLM Benchmark Suite \u2014 LFU Cache & ACID Audit",
    "machine": "Apple M3 Max, 48GB unified memory, LM Studio",
    "exam_prompt": "prompts/lfu_cache_prompt.txt",
    "grading_rubric": "prompts/grading.txt",
    "pillars": [
      "complexity",
      "concurrency",
      "isolation",
      "memory_edge_cases",
      "test_integrity"
    ],
    "max_per_pillar": 20,
    "schema_version": 1
  },
  "models": [
    {
      "id": "qwen3.6-35b-a3b-6bit-mlx",
      "timestamp": "2026-07-28T13:00:00Z",
      "model_name": "Qwen 3.6 35B-A3B",
      "quant": "6-bit MLX",
      "format": "mlx",
      "tok_sec": 68.86,
      "total_tokens": 14883,
      "ttft_sec": 0.94,
      "filename": "outputs/qwen3.6-35b-a3b-6bit-mlx.py",
      "tests_pass": true,
      "total_score": 82,
      "breakdown": {
        "complexity": 16,
        "concurrency": 16,
        "isolation": 17,
        "memory_edge_cases": 16,
        "test_integrity": 17
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "Solid daily-driver scaffolding for ACID/async patterns \u2014 produces runnable, well-structured code, but needs a human pass for __slots__, monotonic clocks, and lock granularity before production.",
      "critical_bugs": [
        "No __slots__ declared on Node/Transaction/LFUCache \u2014 rubric explicitly required it for memory efficiency.",
        "Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments can cause premature/incorrect TTL eviction.",
        "_cleanup_freq_lists() performs a hidden O(F) scan (iterates all freq tiers + min()), called after every eviction, background batch, AND inside commit() \u2014 violates the strict O(1) requirement.",
        "Transaction commit holds the single cache lock across all write/delete/bump loops + cleanup \u2014 coarse-grained, blocks all readers for the whole commit window; no fine-grained locking.",
        "Lost-update risk: commit applies tx-local writes without any MVCC/version check, so a key modified by the background evictor or another committer between tx.get() and commit() is overwritten blindly.",
        "Tests dodge hard cases: 50-task stress uses unique keys with capacity 100, so no eviction-under-contention ever happens; no test for rollback-after-partial-application or mid-commit read isolation."
      ],
      "patch_code": "# FIX 1: Add __slots__ for memory efficiency\n@dataclass\nclass Node:\n    __slots__ = ('key', 'value', 'freq', 'expires_at', 'prev', 'next')\n    key: Any\n    value: Any\n    freq: int\n    expires_at: Optional[float]\n    prev: Optional['Node'] = None\n    next: Optional['Node'] = None\n\n# FIX 2: Use monotonic clock everywhere (get/put/_add_node/commit)\n#   time.time()  ->  time.monotonic()\n# e.g.\nexpires_at = time.monotonic() + ttl_seconds if ttl_seconds else None\n\n# FIX 3: Make _cleanup_freq_lists O(1) \u2014 bump min_freq incrementally\n# instead of recomputing min() across all tiers:\n# In _update_freq, when emptying the min_freq bucket, only bump min_freq\n# if you're evicting from it; otherwise leave it. Delete the global\n# min(self.freq_map.keys()) scan. For background sweeps, prune empty\n# buckets lazily on next _evict() rather than scanning proactively.\n\n# FIX 4: Shrink commit critical section \u2014 apply writes into a staging\n# structure under the lock, then release; or use per-bucket locks so\n# readers on unrelated keys aren't blocked.\n\n# FIX 5: Add MVCC version to Node; in commit, raise/abort if the\n# stored version != the version seen at tx.get() time (lost-update detect)."
    },
    {
      "id": "qwen3.6-35b-a3b-4bit-mlx",
      "timestamp": "2026-07-28T13:05:00Z",
      "model_name": "Qwen 3.6 35B-A3B",
      "quant": "4-bit MLX",
      "format": "mlx",
      "tok_sec": 83.31,
      "total_tokens": 13278,
      "ttft_sec": 0.73,
      "filename": "outputs/qwen3.6-35b-a3b-4bit-mlx.py",
      "tests_pass": false,
      "total_score": 57,
      "breakdown": {
        "complexity": 15,
        "concurrency": 15,
        "isolation": 13,
        "memory_edge_cases": 10,
        "test_integrity": 4
      },
      "verdict": "Critical Bugs",
      "best_for": "Not recommended for systems code as-is. The 4-bit quant degrades logic sharply vs the 6-bit sibling (82->57) \u2014 usable only for boilerplate/scaffolding drafts that a human will heavily rewrite.",
      "critical_bugs": [
        "FATAL: _evict() double-removes nodes \u2014 pop() already calls remove() (nulling node.prev/next), then _remove_node() calls remove() AGAIN -> AttributeError: 'NoneType' on first eviction. The cache cannot survive reaching capacity.",
        "Test suite never executes: test_lfu_eviction crashes at the first eviction, so the 'All tests passed' message is unreachable and assertions are effectively unverified.",
        "Background _eviction_loop materializes list(self.key_to_node.keys())[:50] every sweep \u2014 an O(N) linear scan, forbidden by the strict O(1) requirement.",
        "_FreqList.pop() has no empty-guard \u2014 calling pop() on an empty list dereferences self.head.next (the dummy tail) and corrupts the DLL.",
        "Transaction _apply_put applies the buffered value into the EXACT original_node captured at tx.put() time; if the global cache evicted/relocated that node between put and commit, you mutate a stale/dangling node (no MVCC/version check).",
        "Commit is not atomic across exceptions: a crash mid-_apply loop leaves half-applied global state with no rollback.",
        "Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP jumps corrupt TTL eviction."
      ],
      "patch_code": "# FIX 1 (the crash): _evict double-removes. pop() already unlinks,\n# so do NOT call _remove_node on a popped node. Either:\n#   (a) pop and then only delete the key_map entry + min_freq bookkeeping:\ndef _evict(self):\n    if not self.freq_to_list:\n        return\n    evict_list = self.freq_to_list[self.min_freq]\n    if evict_list.size == 0:        # guard against empty\n        del self.freq_to_list[self.min_freq]\n        return\n    node = evict_list.pop()         # pop() unlinks + nulls prev/next\n    del self.key_to_node[node.key]  # DON'T call _remove_node again\n    if self.freq_to_list[self.min_freq].size == 0:\n        del self.freq_to_list[self.min_freq]\n        self.min_freq += 1\n\n# FIX 2: _FreqList.pop empty-guard\ndef pop(self) -> _Node:\n    if self.size == 0:\n        raise IndexError('pop from empty _FreqList')\n    node = self.head.next\n    self.remove(node)\n    return node\n\n# FIX 3: kill the O(N) scan in background sweep \u2014 maintain a separate\n# set of keys that have a TTL, and iterate that set in batches:\nasync with self.lock:\n    batch = list(self._ttl_keys)[:50]\n    for k in batch:\n        node = self.key_to_node.get(k)\n        if node and 0 < node.expires_at <= time.monotonic():\n            self._remove_node(node)\n\n# FIX 4: time.time() -> time.monotonic() everywhere.\n# FIX 5: add node.version; in tx._apply_put, abort/refresh if\n# cache.key_to_node[key] is a different node than original_node."
    },
    {
      "id": "qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf",
      "timestamp": "2026-07-28T13:10:00Z",
      "model_name": "Qwen 3.6 35B-A3B (uncensored hauhaucs aggressive)",
      "quant": "GGUF",
      "format": "gguf",
      "tok_sec": 62.54,
      "total_tokens": 13897,
      "ttft_sec": 1.09,
      "filename": "outputs/qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf.py",
      "tests_pass": false,
      "total_score": 49,
      "breakdown": {
        "complexity": 12,
        "concurrency": 10,
        "isolation": 12,
        "memory_edge_cases": 11,
        "test_integrity": 4
      },
      "verdict": "Critical Bugs",
      "best_for": "Not recommended for production code. Reasonable API shape and correctly used time.monotonic(), but the module does not parse (syntax error), contains an infinite while:pass loop, and has data races. Avoid for systems/concurrency work.",
      "critical_bugs": [
        "SyntaxError: line 305 'assert val := await cache.get(...)' is invalid Python \u2014 walrus operator cannot appear in an assert statement. The ENTIRE module fails to compile, so nothing runs and no test can execute.",
        "Infinite busy-loop: _evict_lfu lines 159-160 \u2014 'while self.min_freq in self.freq_map and self.min_freq < max(...): pass' has an empty body that never updates min_freq, recomputes max() (O(F)) each iteration, and can never terminate.",
        "Hidden O(F) scan: min(self.freq_map.keys()) / max(self.freq_map.keys()) appears at 6 call sites (every eviction and removal) \u2014 violates the strict O(1) requirement.",
        "Race condition: get() and put() perform lazy-TTL _remove_key() BEFORE acquiring the lock (lines 71-73, 107-108), mutating shared state unlocked while other coroutines read/write.",
        "Deadlock risk: background_loop holds self._lock, then calls await self._remove_key() which is itself a lock-acquiring method \u2014 asyncio.Lock is NOT reentrant -> deadlock when the evictor runs.",
        "No __slots__ on _Node despite using a dataclass (rubric required it for memory efficiency).",
        "_remove_key will KeyError on self.freq_map[freq] if a concurrent operation already deleted that bucket."
      ],
      "patch_code": "# FIX 1 (the parse error): assign first, then assert.\nval_d = await cache.get(\"D\")\nassert val_d, \"D should exist\"\nval_a = await cache.get(\"A\")\nassert val_a, \"A should exist (highest freq)\"\n\n# FIX 2: delete the broken while:pass loop. Bump min_freq incrementally:\n# only when the min_freq bucket empties, and only ever UP by 1 (a key\n# whose freq increased must land at min_freq+1). Never call min()/max().\nif not old_bucket:\n    del self.freq_map[old_freq]\n    if self.min_freq == old_freq:\n        self.min_freq += 1   # next tier up; never scan\n\n# FIX 3: do ALL lazy eviction INSIDE the lock, not before it:\nasync def get(self, key):\n    async with self._lock:\n        if self.ttl_map.get(key, inf) <= time.monotonic():\n            await self._remove_key(key)   # now locked\n            return None\n        ...\n\n# FIX 4: make _remove_key a non-locking private helper, called from\n# inside already-locked public methods, so background_loop doesn't try\n# to re-acquire the non-reentrant asyncio.Lock.\n\n# FIX 5: add __slots__ = ('key','value','ttl_expiry') to _Node."
    },
    {
      "id": "kat-coder-v2.5-dev-xl-mlx",
      "timestamp": "2026-07-28T13:15:00Z",
      "model_name": "KAT-Coder v2.5 Dev XL",
      "quant": "MLX",
      "format": "mlx",
      "tok_sec": 65.26,
      "total_tokens": 6172,
      "ttft_sec": 7.42,
      "filename": "outputs/kat-coder-v2.5-dev-xl-mlx.py",
      "tests_pass": false,
      "total_score": 65,
      "breakdown": {
        "complexity": 15,
        "concurrency": 16,
        "isolation": 16,
        "memory_edge_cases": 13,
        "test_integrity": 5
      },
      "verdict": "Critical Bugs",
      "best_for": "Promising code-design instincts (cleanest abstractions and best transaction isolation design in the set) but undone by a single fatal one-line bug that stops it running. With the bug fixed it would likely score 80+; as-is, only useful as a structural reference.",
      "critical_bugs": [
        "FATAL: _put_internal line 305 inserts a NEW key with 'self._freq_map[1].push_front(...)' but never ensures the freq-1 bucket exists \u2014 KeyError: 1 on the very first put. The cache cannot store a single key. The _ensure_freq_list(1) helper it should use exists and is used correctly everywhere else (lines 294, 354).",
        "Transaction.commit() calls _put_internal for buffered writes, so it hits the same KeyError: 1 \u2014 committed transactions crash too.",
        "Test suite cannot execute: crashes at the first cache.put() in main(); the well-built test harness (pass/fail counter, 4 real scenarios) validates nothing.",
        "No __slots__ on _DLLNode/_CacheNode/_DoublyLinkedList despite the rubric requiring it for memory efficiency.",
        "_evict_node uses min(self._freq_map) (line 325) when the min-tier empties \u2014 a hidden O(F) scan, violating strict O(1).",
        "No MVCC/version check on transaction commit (lost-update possible if the global key is modified between tx.get and commit); commit is not exception-safe across the writes-vs-deletes loops."
      ],
      "patch_code": "# FIX 1 (the fatal one-liner): use the helper that already exists.\n# line 305, in _put_internal, new-key branch:\n- self._freq_map[1].push_front(dll_node)\n+ self._ensure_freq_list(1).push_front(dll_node)\n# (This single change makes the cache and transactions functional.)\n\n# FIX 2: replace the O(F) min() scan with an incremental bump:\n# in _evict_node, when the min-tier bucket empties, min_freq is the\n# lowest remaining tier. Since freq only ever increments by 1, the\n# next min is almost always min_freq+1; track it incrementally rather\n# than scanning. Or, since this only happens on full eviction, accept\n# O(F) but only on the empty-cache edge \u2014 document it.\n\n# FIX 3: add __slots__ to all internal classes:\nclass _CacheNode:\n    __slots__ = ('key','value','ttl_seconds','expiry_time','freq','dll_node')\n    ...\n\n# FIX 4: wrap commit applies in try/except so a mid-commit exception\n# does not leave a half-applied global state; consider abort semantics.\n# FIX 5: add node.version; in tx commit, abort if the global node for\n# a key is not the one seen at tx.get() time."
    },
    {
      "id": "gemma4-31b-gguf",
      "timestamp": "2026-07-28T13:20:00Z",
      "model_name": "Gemma 4 31B",
      "quant": "GGUF",
      "format": "gguf",
      "tok_sec": 10.09,
      "total_tokens": 4536,
      "ttft_sec": 4.39,
      "speed_caveat": "All Gemma 4 models ran abnormally slow (GPU offload appeared inactive despite being set), so tok/sec and TTFT are NOT representative of the model itself \u2014 likely an LM Studio/GGUF config issue. Treat speed numbers for the Gemma 4 batch as suspect.",
      "filename": "outputs/gemma4-31b-gguf.py",
      "tests_pass": true,
      "total_score": 78,
      "breakdown": {
        "complexity": 17,
        "concurrency": 16,
        "isolation": 14,
        "memory_edge_cases": 15,
        "test_integrity": 16
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "Clean, correct, runnable code with solid O(1) structure and good concurrency granularity. A reliable pick for everyday caching/async work after a monotonic-clock + __slots__ pass.",
      "critical_bugs": [
        "Isolation leak: Transaction.get falls back to the PUBLIC cache.get, which calls _update_frequency \u2014 so reading a key inside a transaction mutates GLOBAL frequency state before commit, leaking uncommitted access patterns into global eviction order. Spec requires tx reads not to alter global freq.",
        "Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments corrupt TTL eviction.",
        "No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction \u2014 rubric required it for memory efficiency.",
        "_delete_internal deliberately leaves empty frequency buckets in freq_map (documented but a minor memory leak: empty DoublyLinkedList objects accumulate).",
        "Background evictor does list(self.cache.keys()) = O(N) snapshot every interval \u2014 a linear scan, forbidden by strict O(1).",
        "No MVCC/version check on commit (lost-update possible); commit is not exception-safe across the deletes-vs-puts loops.",
        "Tests pass but don't probe mid-commit read isolation or eviction-under-real-contention (capacity sized so all keys fit), so the isolation leak above goes undetected."
      ],
      "patch_code": "# FIX 1 (the isolation leak): tx reads must NOT mutate global freq.\n# Add a read-only global lookup (no _update_frequency) and use it in tx.get:\nasync def _read_raw(self, key):           # no freq bump\n    node = self.cache.get(key)\n    if node is None: return None\n    if time.monotonic() > node.expiry:\n        await self._delete_internal(key)\n        return None\n    return node.value\n# then in Transaction.get fallback:\n    return await self._cache._read_raw(key)   # NOT cache.get\n\n# FIX 2: time.time() -> time.monotonic() everywhere (get/put/_put_internal/bg loop).\n# FIX 3: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 4: prune empty freq buckets on delete, or have _evict_lfu drop them.\n# FIX 5: iterate a dedicated _ttl_keys set (batched) in the bg evictor\n# instead of list(self.cache.keys()) to stay O(batch), not O(N)."
    },
    {
      "id": "gemma-4-31b-qat-gguf",
      "timestamp": "2026-07-28T13:25:00Z",
      "model_name": "Gemma 4 31B QAT",
      "quant": "QAT GGUF",
      "format": "gguf",
      "tok_sec": 15.0,
      "total_tokens": 4552,
      "ttft_sec": 4.01,
      "speed_caveat": "Same as the Gemma 4 batch: GPU offload appeared inactive so tok/sec/TTFT are NOT representative of the model. Suspected LM Studio/GGUF config issue.",
      "filename": "outputs/gemma-4-31b-qat-gguf.py",
      "tests_pass": true,
      "total_score": 70,
      "breakdown": {
        "complexity": 16,
        "concurrency": 16,
        "isolation": 11,
        "memory_edge_cases": 13,
        "test_integrity": 14
      },
      "verdict": "Critical Bugs",
      "best_for": "Runnable and structurally sound, but the LFU eviction has a stale-min_freq capacity-breach path and the transaction API doesn't match the spec (no commit/rollback). Usable for prototypes if you fix eviction and re-skin transactions.",
      "critical_bugs": [
        "Capacity breach: eviction does self.freq_map[self.min_freq].pop_tail() with NO guard that the bucket exists or is non-empty, and never prunes emptied freq buckets. After manual deletes empty the min-tier, pop_tail returns None silently -> eviction fails -> cache grows PAST capacity. This is the exact stale-min_freq capacity-breach the rubric flags.",
        "Non-conformant transaction API: Transaction has NO commit() or rollback() method (both required by spec). Commit happens via a separate cache.apply_transaction_changes(tx._state) \u2014 wrong surface; the test only passes because it uses this internal path.",
        "Isolation leak: Transaction.get falls back to the public cache.get, which bumps global frequency before commit \u2014 uncommitted tx reads alter global eviction order.",
        "Duplicated eviction logic in apply_transaction_changes re-introduces the stale-min_freq bug at line 211.",
        "Uses time.time() (system clock) via _get_now() instead of time.monotonic() \u2014 NTP jumps corrupt TTL (though _get_now is a clean single fix point).",
        "No __slots__ on Node/DoublyLinkedList/LFUCache/TransactionState/Transaction.",
        "Background sweep does list(self.cache.keys()) = O(N) per interval.",
        "Tests pass but use the non-spec commit path and don't probe capacity breach or isolation leak."
      ],
      "patch_code": "# FIX 1 (capacity breach): guard + prune empty buckets on eviction:\nwhile self.min_freq in self.freq_map and self.freq_map[self.min_freq].size == 0:\n    del self.freq_map[self.min_freq]\n    self.min_freq += 1\n    if not self.freq_map:\n        break\nif self.min_freq not in self.freq_map:\n    return  # nothing to evict\nevicted = self.freq_map[self.min_freq].pop_tail()\nif evicted and self.freq_map[self.min_freq].size == 0:\n    del self.freq_map[self.min_freq]\n\n# FIX 2 (conformant API): add commit/rollback to Transaction:\nasync def commit(self):\n    await self._cache.apply_transaction_changes(self._state)\n    self._committed = True\ndef rollback(self):\n    self._state.writes.clear()\n    self._committed = True\n\n# FIX 3 (isolation leak): tx.get should use a read-only global lookup\n# (no _update_freq), not the public cache.get.\n# FIX 4: _get_now returns time.monotonic().\n# FIX 5: add __slots__ to all node/list classes."
    },
    {
      "id": "gemma-4-12b-coder-heretic-mxfp8-mlx",
      "timestamp": "2026-07-28T13:30:00Z",
      "model_name": "Gemma 4 12B Coder (fable5-composer2.5-v1-uncensored-heretic merge)",
      "quant": "mxfp8 MLX",
      "format": "mlx",
      "tok_sec": 25.28,
      "total_tokens": 2625,
      "ttft_sec": 3.21,
      "filename": "outputs/gemma-4-12b-coder-fable5-composer2.5-v1-uncensored-heretic-mxfp8-mlx.py",
      "tests_pass": false,
      "total_score": 43,
      "breakdown": {
        "complexity": 13,
        "concurrency": 7,
        "isolation": 10,
        "memory_edge_cases": 9,
        "test_integrity": 4
      },
      "verdict": "Critical Bugs",
      "best_for": "Not usable as-is \u2014 the cache cannot store its first key and the background evictor would crash the event loop. The smallest model in the set (12B) and lowest-quality output. Avoid for systems work.",
      "critical_bugs": [
        "FATAL: put() line 112 does 'bucket = self.freq_buckets[self.min_freq]' after setting min_freq=1 but NEVER creates freq_buckets[1] -> KeyError: 1 on the very first put. Cache is unusable.",
        "Background evictor is fundamentally broken: start_evictor defines a SYNC 'def evict_loop' and passes it to create_task; inside it calls blocking time.sleep(interval) (freezes the event loop) AND asyncio.run(...) from within a running loop -> RuntimeError. Would crash hard if ever reached.",
        "Eviction-by-re-put: expired keys are 'evicted' by re-inserting them with TTL 0 (line 122) instead of deleting them \u2014 wrong semantics and triggers immediate re-eviction.",
        "Transaction duplicates the entire LFU machinery (local_cache + local_freq + local_min_freq) for snapshot isolation, but _update_local_freq has the same missing-bucket KeyError (line 140).",
        "Class name typo 'DoublyLinkedListList' (doubled word).",
        "No __slots__; time.time() (not monotonic) throughout.",
        "Tests cannot run \u2014 crash at first put."
      ],
      "patch_code": "# FIX 1 (the fatal KeyError): create the bucket before use.\n# In put(), new-key branch:\n- bucket = self.freq_buckets[self.min_freq]\n+ bucket = self.freq_buckets.setdefault(self.min_freq, DoublyLinkedListList())\n# Same fix in _update_freq and Transaction._update_local_freq (use setdefault).\n\n# FIX 2 (the broken evictor): make it a real async task that deletes:\nasync def _evict_loop(self, interval):\n    while True:\n        await asyncio.sleep(interval)          # async, non-blocking\n        now = time.monotonic()\n        async with self.global_lock:\n            expired = [k for k, n in list(self.cache.items()) if now > n.ttl_expiry]\n            for k in expired:\n                node = self.cache.pop(k, None)\n                if node:\n                    self.freq_buckets[node.freq].remove(node)   # DELETE, not re-put\n\nasync def start_evictor(self, interval=1.0):\n    self.evictor_task = asyncio.create_task(self._evict_loop(interval))\n\n# FIX 3: delete expired keys; do NOT re-insert with TTL 0.\n# FIX 4: time.time() -> time.monotonic().\n# FIX 5: add __slots__ to Node / DoublyLinkedListList / ConcurrentLFUCache / Transaction."
    },
    {
      "id": "deepseekv4flash-cloud",
      "timestamp": "2026-07-28T13:35:00Z",
      "model_name": "DeepSeek V4 Flash (CLOUD baseline)",
      "quant": "n/a (cloud)",
      "format": "cloud",
      "tok_sec": null,
      "total_tokens": null,
      "ttft_sec": null,
      "speed_caveat": "Cloud model (run via opencode, not LM Studio) \u2014 tok_sec/tokens/TTFT are N/A (not measured for cloud). Included as a quality baseline against the local models. NOTE: it took 3 attempts to produce any output and ~12 minutes of thinking before succeeding \u2014 so it is a QUALITY benchmark, not a speed/usability one.",
      "filename": "deepseekv4flash.py",
      "tests_pass": true,
      "total_score": 91,
      "breakdown": {
        "complexity": 18,
        "concurrency": 18,
        "isolation": 19,
        "memory_edge_cases": 18,
        "test_integrity": 18
      },
      "verdict": "Production Ready",
      "best_for": "Reference-quality baseline (91/100) \u2014 the bar the local models are measured against. Only submission with __slots__ + time.monotonic() + delta-based transactional frequency accounting. Sets the ceiling for correctness, though its unreliability (3 attempts, 12-min think time) makes it a poor *local* daily-driver.",
      "critical_bugs": [
        "Two min(self._freq_to_list) linear scans in _evict_one's defensive recovery path (lines 415, 429) \u2014 only triggered when min_freq desyncs, not per-operation, but still a non-O(1) path. Could be replaced with incremental tracking.",
        "Single coarse lock held across the whole commit-apply loop \u2014 not the fine-grained locking the prompt asked for.",
        "__slots__ present on _Node and _DLL but not extended to Transaction / LFUCache.",
        "No explicit lost-update/conflict abort on commit (delta-based freq is applied unconditionally).",
        "Tests pass 20/20 but don't include a mid-commit read-isolation probe or adversarial eviction-under-contention stress."
      ],
      "patch_code": "# These are minor refinements on an already production-ready file.\n\n# FIX 1: eliminate the recovery min() scans by keeping min_freq\n# strictly in sync on every insert/bump/remove (it already does on\n# the hot path), so the _evict_one recovery branch is unreachable and\n# can assert instead of scanning:\nassert self._min_freq in self._freq_to_list or not self._freq_to_list\n\n# FIX 2: extend __slots__ to Transaction and LFUCache.\nclass LFUCache(Generic[KT, VT]):\n    __slots__ = ('_capacity','_key_to_node','_freq_to_list','_min_freq',\n                 '_lock','_ttl_index','_evictor_task','_closed')\n\n# FIX 3 (optional): on commit, if a key's global node changed since the\n# tx snapshot, raise LFUCacheError('lost update') instead of overwriting."
    },
    {
      "id": "gemma4-26b-a4b-8bit-mlx",
      "timestamp": "2026-07-28T16:25:00Z",
      "model_name": "Gemma 4 26B-A4B",
      "quant": "8-bit",
      "param_size": "26B-A4B (MoE)",
      "format": "mlx",
      "tok_sec": 58.3,
      "total_tokens": 7390,
      "ttft_sec": 0.93,
      "filename": "outputs/gemma4-26b-a4b-8bit-mlx.py",
      "tests_pass": true,
      "total_score": 82,
      "breakdown": {
        "complexity": 17,
        "concurrency": 16,
        "isolation": 18,
        "memory_edge_cases": 15,
        "test_integrity": 16
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "Tied top local scorer (82). The only local model to use delta-based transactional frequency accounting (freq bumps deferred to commit), matching the cloud baseline's isolation approach. Reliable for async/ACID-pattern work after a __slots__ + min_freq-edge pass.",
      "critical_bugs": [
        "Stale min_freq on manual delete: _remove_node_from_structures empties the min-freq bucket but does `pass` instead of recomputing min_freq (lines 199-202, documented as 'a simplification'). Correct only because eviction has a min_freq-in-freq_map guard + arbitrary-key fallback (lines 237-244) \u2014 latent fragility under concurrent deletes.",
        "No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction despite Generic dataclasses (rubric required it for memory efficiency).",
        "Background evictor does list(self.cache_data.keys()) = O(N) snapshot every interval \u2014 a linear scan, forbidden by strict O(1).",
        "No MVCC/version check on commit (lost-update possible if the global key changes between tx.get and commit); commit applies deletes->reads->puts without try/except, so a mid-commit exception leaves partial state."
      ],
      "patch_code": "# FIX 1 (stale min_freq): recompute or invalidate when the min bucket empties.\n# In _remove_node_from_structures, replace the `pass`:\nif dll.size == 0:\n    del self.freq_map[node.freq]\n    if self.min_freq == node.freq:\n        # bump to next existing tier (frequencies are contiguous under normal use)\n        self.min_freq = self.min_freq + 1 if (self.min_freq + 1) in self.freq_map else min(self.freq_map, default=1)\n\n# FIX 2: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 3: background evictor \u2014 maintain a _ttl_keys set and iterate IT in\n# batches instead of list(self.cache_data.keys()) to stay O(batch).\n# FIX 4: wrap commit's three loops in try/except with rollback semantics on failure."
    },
    {
      "id": "qwen3.6-27b-8bit-mlx",
      "timestamp": "2026-07-28T16:30:00Z",
      "model_name": "Qwen 3.6 27B",
      "quant": "8-bit",
      "param_size": "27B dense",
      "format": "mlx",
      "tok_sec": 12.29,
      "total_tokens": 12790,
      "ttft_sec": 2.9,
      "speed_caveat": "This model 'thought' for 12m48s before producing output and ran at 12.29 tok/sec \u2014 anomalously slow for an 8-bit MLX on M3 Max. Likely an inference/quant issue worth investigating; the slow generation is NOT representative of normal 27B-8bit throughput.",
      "filename": "outputs/qwen3.6-27b-8bit-mlx.py",
      "tests_pass": true,
      "total_score": 78,
      "breakdown": {
        "complexity": 17,
        "concurrency": 16,
        "isolation": 14,
        "memory_edge_cases": 15,
        "test_integrity": 16
      },
      "verdict": "Minor Logic Flaws",
      "best_for": "Clean, correct, runnable \u2014 same tier as Gemma 4 31B (78). Good O(1) structure and concurrency granularity. Reliable for everyday async/caching work after a monotonic-clock + __slots__ + isolation pass. Caveat: was anomalously slow to generate.",
      "critical_bugs": [
        "Isolation leak: Transaction.get falls back to the PUBLIC cache.get (line 78), which calls _update_freq \u2014 so reading a key inside a transaction mutates GLOBAL frequency state before commit, leaking uncommitted access patterns into global eviction order.",
        "Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments corrupt TTL eviction.",
        "No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction \u2014 rubric required it for memory efficiency.",
        "Background _evict_loop materializes list(self.nodes.keys()) (O(N)) before checking only batch_size keys \u2014 the break caps work but not the snapshot cost, a hidden O(N) per sweep.",
        "Commit re-implements put+evict inline (lines 102-118) duplicating the public path = duplicated bug surface; not wrapped in try/except so a mid-commit exception leaves partial state. No MVCC version check (lost-update possible)."
      ],
      "patch_code": "# FIX 1 (isolation leak): add a read-only global lookup (no freq bump)\n# and use it in tx.get instead of the public cache.get:\nasync def _read_raw(self, key):\n    async with self.lock:\n        node = self.nodes.get(key)\n        if node is None: return None\n        if time.monotonic() > node.expires_at:\n            self._remove_node(key); return None\n        return node.value\n# then:  return await self._cache._read_raw(key)\n# FIX 2: time.time() -> time.monotonic() everywhere.\n# FIX 3: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 4: maintain a _ttl_keys set; iterate IT (batched) in the bg loop\n# instead of list(self.nodes.keys()).\n# FIX 5: factor commit's put/evict to reuse the internal helpers; wrap\n# the commit loop in try/except with rollback-on-failure."
    },
    {
      "id": "qwen3-coder-30b-6bit-mlx",
      "timestamp": "2026-07-28T16:35:00Z",
      "model_name": "Qwen3 Coder 30B",
      "quant": "6-bit",
      "param_size": "30B",
      "format": "mlx",
      "tok_sec": 72.7,
      "total_tokens": 2779,
      "ttft_sec": 0.9,
      "filename": "outputs/qwen3-coder-30b-6bit-mlx.py",
      "tests_pass": false,
      "total_score": 50,
      "breakdown": {
        "complexity": 15,
        "concurrency": 9,
        "isolation": 11,
        "memory_edge_cases": 11,
        "test_integrity": 4
      },
      "verdict": "Critical Bugs",
      "best_for": "Not usable as-is \u2014 transactions crash immediately due to an async/sync lock mismatch. The coder-specialist produced terse, fast output (2779 tok, 72.7 t/s) with competent freq-bucket structure, but fumbled the async primitive. Fix the one lock bug and it would likely score 70+.",
      "critical_bugs": [
        "FATAL: _transaction_lock is an asyncio.Lock() (line 101) but begin_transaction() is a SYNC def that uses synchronous `with self._transaction_lock:` (line 199). asyncio.Lock does not support the sync context-manager protocol -> TypeError on the first transaction, crashing the entire test suite.",
        "Test suite cannot run: crashes at cache.begin_transaction() in main(); the transaction and concurrency assertions never execute.",
        "Unused `import threading` and `import weakref` \u2014 vestigial confusion between threading and asyncio primitives.",
        "Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP jumps corrupt TTL eviction.",
        "No __slots__ on CacheNode/FrequencyBucket/InMemoryLFUCache/Transaction \u2014 rubric required it.",
        "FrequencyBucket stores nodes in BOTH a DLL and a parallel `nodes` dict \u2014 redundant memory per bucket."
      ],
      "patch_code": "# FIX 1 (the fatal crash): make begin_transaction async and use async with.\nasync def begin_transaction(self) -> 'Transaction':\n    async with self._transaction_lock:\n        self._transaction_counter += 1\n        tx = Transaction(self)\n        self._transactions[self._transaction_counter] = tx\n    return tx\n# (and update callers: `tx = await cache.begin_transaction()`)\n# Alternative if sync creation is required: use threading.Lock for the\n# counter, but that is wrong in an asyncio codebase \u2014 go async.\n\n# FIX 2: remove unused `import threading` and `import weakref`.\n# FIX 3: time.time() -> time.monotonic() everywhere.\n# FIX 4: add __slots__ to all classes.\n# FIX 5: drop the redundant FrequencyBucket.nodes dict; the DLL already\n# tracks membership, so the dict is duplicate storage."
    }
  ]
}