adminandClaude
e3d73de36a
Third data point on the same model:
Qwen 3.6 35B-A3B 6-bit MLX:
LFU cache 82 (runs clean)
TTS pipeline 49 (doesn't parse)
Rust service 50 (7 compile errors)
Profile is now sharp: strong on single-file Python data-structure/ACID
work; repeatedly ships non-compiling/non-parsing code on multi-task async
and typed-language prompts. Keep it on Python ACID tasks; do NOT offload
Rust or async-pipeline work.
Verified with cargo 1.94 (mpsc::bounded hallucination, ownership moves,
dead test override, broken remove_watcher). Lang field added for non-Python.
Co-Authored-By: Claude <noreply@anthropic.com>
475 lines
41 KiB
JSON
475 lines
41 KiB
JSON
{
|
|
"meta": {
|
|
"project": "Local LLM Benchmark Suite \u2014 LFU Cache & ACID Audit",
|
|
"machine": "Apple M3 Max, 48GB unified memory, LM Studio",
|
|
"exam_prompt": "prompts/lfu_cache_prompt.txt",
|
|
"grading_rubric": "prompts/grading.txt",
|
|
"pillars": [
|
|
"complexity",
|
|
"concurrency",
|
|
"isolation",
|
|
"memory_edge_cases",
|
|
"test_integrity"
|
|
],
|
|
"max_per_pillar": 20,
|
|
"schema_version": 1,
|
|
"prompts": {
|
|
"lfu": {
|
|
"label": "LFU Cache + ACID (systems/async)",
|
|
"file": "prompts/lfu_cache_prompt.txt"
|
|
},
|
|
"tts": {
|
|
"label": "TTS Pipeline (queues/backpressure)",
|
|
"file": "prompts/tts_pipeline.txt"
|
|
},
|
|
"mcp": {
|
|
"label": "MCP Tool Server",
|
|
"file": "prompts/mcp_server.txt"
|
|
},
|
|
"rust": {
|
|
"label": "Rust tokio Service",
|
|
"file": "prompts/rust_service.txt"
|
|
},
|
|
"data": {
|
|
"label": "Data Service (pooling/SQL)",
|
|
"file": "prompts/data_service.txt"
|
|
},
|
|
"automation": {
|
|
"label": "Automation Glue (idempotent batch)",
|
|
"file": "prompts/automation_glue.txt"
|
|
}
|
|
}
|
|
},
|
|
"models": [
|
|
{
|
|
"id": "qwen3.6-35b-a3b-6bit-mlx",
|
|
"timestamp": "2026-07-28T13:00:00Z",
|
|
"model_name": "Qwen 3.6 35B-A3B",
|
|
"quant": "6-bit MLX",
|
|
"format": "mlx",
|
|
"tok_sec": 68.86,
|
|
"total_tokens": 14883,
|
|
"ttft_sec": 0.94,
|
|
"filename": "outputs/qwen3.6-35b-a3b-6bit-mlx.py",
|
|
"tests_pass": true,
|
|
"total_score": 82,
|
|
"breakdown": {
|
|
"complexity": 16,
|
|
"concurrency": 16,
|
|
"isolation": 17,
|
|
"memory_edge_cases": 16,
|
|
"test_integrity": 17
|
|
},
|
|
"verdict": "Minor Logic Flaws",
|
|
"best_for": "Solid daily-driver scaffolding for ACID/async patterns \u2014 produces runnable, well-structured code, but needs a human pass for __slots__, monotonic clocks, and lock granularity before production.",
|
|
"critical_bugs": [
|
|
"No __slots__ declared on Node/Transaction/LFUCache \u2014 rubric explicitly required it for memory efficiency.",
|
|
"Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments can cause premature/incorrect TTL eviction.",
|
|
"_cleanup_freq_lists() performs a hidden O(F) scan (iterates all freq tiers + min()), called after every eviction, background batch, AND inside commit() \u2014 violates the strict O(1) requirement.",
|
|
"Transaction commit holds the single cache lock across all write/delete/bump loops + cleanup \u2014 coarse-grained, blocks all readers for the whole commit window; no fine-grained locking.",
|
|
"Lost-update risk: commit applies tx-local writes without any MVCC/version check, so a key modified by the background evictor or another committer between tx.get() and commit() is overwritten blindly.",
|
|
"Tests dodge hard cases: 50-task stress uses unique keys with capacity 100, so no eviction-under-contention ever happens; no test for rollback-after-partial-application or mid-commit read isolation."
|
|
],
|
|
"patch_code": "# FIX 1: Add __slots__ for memory efficiency\n@dataclass\nclass Node:\n __slots__ = ('key', 'value', 'freq', 'expires_at', 'prev', 'next')\n key: Any\n value: Any\n freq: int\n expires_at: Optional[float]\n prev: Optional['Node'] = None\n next: Optional['Node'] = None\n\n# FIX 2: Use monotonic clock everywhere (get/put/_add_node/commit)\n# time.time() -> time.monotonic()\n# e.g.\nexpires_at = time.monotonic() + ttl_seconds if ttl_seconds else None\n\n# FIX 3: Make _cleanup_freq_lists O(1) \u2014 bump min_freq incrementally\n# instead of recomputing min() across all tiers:\n# In _update_freq, when emptying the min_freq bucket, only bump min_freq\n# if you're evicting from it; otherwise leave it. Delete the global\n# min(self.freq_map.keys()) scan. For background sweeps, prune empty\n# buckets lazily on next _evict() rather than scanning proactively.\n\n# FIX 4: Shrink commit critical section \u2014 apply writes into a staging\n# structure under the lock, then release; or use per-bucket locks so\n# readers on unrelated keys aren't blocked.\n\n# FIX 5: Add MVCC version to Node; in commit, raise/abort if the\n# stored version != the version seen at tx.get() time (lost-update detect).",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "qwen3.6-35b-a3b-4bit-mlx",
|
|
"timestamp": "2026-07-28T13:05:00Z",
|
|
"model_name": "Qwen 3.6 35B-A3B",
|
|
"quant": "4-bit MLX",
|
|
"format": "mlx",
|
|
"tok_sec": 83.31,
|
|
"total_tokens": 13278,
|
|
"ttft_sec": 0.73,
|
|
"filename": "outputs/qwen3.6-35b-a3b-4bit-mlx.py",
|
|
"tests_pass": false,
|
|
"total_score": 57,
|
|
"breakdown": {
|
|
"complexity": 15,
|
|
"concurrency": 15,
|
|
"isolation": 13,
|
|
"memory_edge_cases": 10,
|
|
"test_integrity": 4
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Not recommended for systems code as-is. The 4-bit quant degrades logic sharply vs the 6-bit sibling (82->57) \u2014 usable only for boilerplate/scaffolding drafts that a human will heavily rewrite.",
|
|
"critical_bugs": [
|
|
"FATAL: _evict() double-removes nodes \u2014 pop() already calls remove() (nulling node.prev/next), then _remove_node() calls remove() AGAIN -> AttributeError: 'NoneType' on first eviction. The cache cannot survive reaching capacity.",
|
|
"Test suite never executes: test_lfu_eviction crashes at the first eviction, so the 'All tests passed' message is unreachable and assertions are effectively unverified.",
|
|
"Background _eviction_loop materializes list(self.key_to_node.keys())[:50] every sweep \u2014 an O(N) linear scan, forbidden by the strict O(1) requirement.",
|
|
"_FreqList.pop() has no empty-guard \u2014 calling pop() on an empty list dereferences self.head.next (the dummy tail) and corrupts the DLL.",
|
|
"Transaction _apply_put applies the buffered value into the EXACT original_node captured at tx.put() time; if the global cache evicted/relocated that node between put and commit, you mutate a stale/dangling node (no MVCC/version check).",
|
|
"Commit is not atomic across exceptions: a crash mid-_apply loop leaves half-applied global state with no rollback.",
|
|
"Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP jumps corrupt TTL eviction."
|
|
],
|
|
"patch_code": "# FIX 1 (the crash): _evict double-removes. pop() already unlinks,\n# so do NOT call _remove_node on a popped node. Either:\n# (a) pop and then only delete the key_map entry + min_freq bookkeeping:\ndef _evict(self):\n if not self.freq_to_list:\n return\n evict_list = self.freq_to_list[self.min_freq]\n if evict_list.size == 0: # guard against empty\n del self.freq_to_list[self.min_freq]\n return\n node = evict_list.pop() # pop() unlinks + nulls prev/next\n del self.key_to_node[node.key] # DON'T call _remove_node again\n if self.freq_to_list[self.min_freq].size == 0:\n del self.freq_to_list[self.min_freq]\n self.min_freq += 1\n\n# FIX 2: _FreqList.pop empty-guard\ndef pop(self) -> _Node:\n if self.size == 0:\n raise IndexError('pop from empty _FreqList')\n node = self.head.next\n self.remove(node)\n return node\n\n# FIX 3: kill the O(N) scan in background sweep \u2014 maintain a separate\n# set of keys that have a TTL, and iterate that set in batches:\nasync with self.lock:\n batch = list(self._ttl_keys)[:50]\n for k in batch:\n node = self.key_to_node.get(k)\n if node and 0 < node.expires_at <= time.monotonic():\n self._remove_node(node)\n\n# FIX 4: time.time() -> time.monotonic() everywhere.\n# FIX 5: add node.version; in tx._apply_put, abort/refresh if\n# cache.key_to_node[key] is a different node than original_node.",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf",
|
|
"timestamp": "2026-07-28T13:10:00Z",
|
|
"model_name": "Qwen 3.6 35B-A3B (uncensored hauhaucs aggressive)",
|
|
"quant": "GGUF",
|
|
"format": "gguf",
|
|
"tok_sec": 62.54,
|
|
"total_tokens": 13897,
|
|
"ttft_sec": 1.09,
|
|
"filename": "outputs/qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf.py",
|
|
"tests_pass": false,
|
|
"total_score": 49,
|
|
"breakdown": {
|
|
"complexity": 12,
|
|
"concurrency": 10,
|
|
"isolation": 12,
|
|
"memory_edge_cases": 11,
|
|
"test_integrity": 4
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Not recommended for production code. Reasonable API shape and correctly used time.monotonic(), but the module does not parse (syntax error), contains an infinite while:pass loop, and has data races. Avoid for systems/concurrency work.",
|
|
"critical_bugs": [
|
|
"SyntaxError: line 305 'assert val := await cache.get(...)' is invalid Python \u2014 walrus operator cannot appear in an assert statement. The ENTIRE module fails to compile, so nothing runs and no test can execute.",
|
|
"Infinite busy-loop: _evict_lfu lines 159-160 \u2014 'while self.min_freq in self.freq_map and self.min_freq < max(...): pass' has an empty body that never updates min_freq, recomputes max() (O(F)) each iteration, and can never terminate.",
|
|
"Hidden O(F) scan: min(self.freq_map.keys()) / max(self.freq_map.keys()) appears at 6 call sites (every eviction and removal) \u2014 violates the strict O(1) requirement.",
|
|
"Race condition: get() and put() perform lazy-TTL _remove_key() BEFORE acquiring the lock (lines 71-73, 107-108), mutating shared state unlocked while other coroutines read/write.",
|
|
"Deadlock risk: background_loop holds self._lock, then calls await self._remove_key() which is itself a lock-acquiring method \u2014 asyncio.Lock is NOT reentrant -> deadlock when the evictor runs.",
|
|
"No __slots__ on _Node despite using a dataclass (rubric required it for memory efficiency).",
|
|
"_remove_key will KeyError on self.freq_map[freq] if a concurrent operation already deleted that bucket."
|
|
],
|
|
"patch_code": "# FIX 1 (the parse error): assign first, then assert.\nval_d = await cache.get(\"D\")\nassert val_d, \"D should exist\"\nval_a = await cache.get(\"A\")\nassert val_a, \"A should exist (highest freq)\"\n\n# FIX 2: delete the broken while:pass loop. Bump min_freq incrementally:\n# only when the min_freq bucket empties, and only ever UP by 1 (a key\n# whose freq increased must land at min_freq+1). Never call min()/max().\nif not old_bucket:\n del self.freq_map[old_freq]\n if self.min_freq == old_freq:\n self.min_freq += 1 # next tier up; never scan\n\n# FIX 3: do ALL lazy eviction INSIDE the lock, not before it:\nasync def get(self, key):\n async with self._lock:\n if self.ttl_map.get(key, inf) <= time.monotonic():\n await self._remove_key(key) # now locked\n return None\n ...\n\n# FIX 4: make _remove_key a non-locking private helper, called from\n# inside already-locked public methods, so background_loop doesn't try\n# to re-acquire the non-reentrant asyncio.Lock.\n\n# FIX 5: add __slots__ = ('key','value','ttl_expiry') to _Node.",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "kat-coder-v2.5-dev-xl-mlx",
|
|
"timestamp": "2026-07-28T13:15:00Z",
|
|
"model_name": "KAT-Coder v2.5 Dev XL",
|
|
"quant": "MLX",
|
|
"format": "mlx",
|
|
"tok_sec": 65.26,
|
|
"total_tokens": 6172,
|
|
"ttft_sec": 7.42,
|
|
"filename": "outputs/kat-coder-v2.5-dev-xl-mlx.py",
|
|
"tests_pass": false,
|
|
"total_score": 65,
|
|
"breakdown": {
|
|
"complexity": 15,
|
|
"concurrency": 16,
|
|
"isolation": 16,
|
|
"memory_edge_cases": 13,
|
|
"test_integrity": 5
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Promising code-design instincts (cleanest abstractions and best transaction isolation design in the set) but undone by a single fatal one-line bug that stops it running. With the bug fixed it would likely score 80+; as-is, only useful as a structural reference.",
|
|
"critical_bugs": [
|
|
"FATAL: _put_internal line 305 inserts a NEW key with 'self._freq_map[1].push_front(...)' but never ensures the freq-1 bucket exists \u2014 KeyError: 1 on the very first put. The cache cannot store a single key. The _ensure_freq_list(1) helper it should use exists and is used correctly everywhere else (lines 294, 354).",
|
|
"Transaction.commit() calls _put_internal for buffered writes, so it hits the same KeyError: 1 \u2014 committed transactions crash too.",
|
|
"Test suite cannot execute: crashes at the first cache.put() in main(); the well-built test harness (pass/fail counter, 4 real scenarios) validates nothing.",
|
|
"No __slots__ on _DLLNode/_CacheNode/_DoublyLinkedList despite the rubric requiring it for memory efficiency.",
|
|
"_evict_node uses min(self._freq_map) (line 325) when the min-tier empties \u2014 a hidden O(F) scan, violating strict O(1).",
|
|
"No MVCC/version check on transaction commit (lost-update possible if the global key is modified between tx.get and commit); commit is not exception-safe across the writes-vs-deletes loops."
|
|
],
|
|
"patch_code": "# FIX 1 (the fatal one-liner): use the helper that already exists.\n# line 305, in _put_internal, new-key branch:\n- self._freq_map[1].push_front(dll_node)\n+ self._ensure_freq_list(1).push_front(dll_node)\n# (This single change makes the cache and transactions functional.)\n\n# FIX 2: replace the O(F) min() scan with an incremental bump:\n# in _evict_node, when the min-tier bucket empties, min_freq is the\n# lowest remaining tier. Since freq only ever increments by 1, the\n# next min is almost always min_freq+1; track it incrementally rather\n# than scanning. Or, since this only happens on full eviction, accept\n# O(F) but only on the empty-cache edge \u2014 document it.\n\n# FIX 3: add __slots__ to all internal classes:\nclass _CacheNode:\n __slots__ = ('key','value','ttl_seconds','expiry_time','freq','dll_node')\n ...\n\n# FIX 4: wrap commit applies in try/except so a mid-commit exception\n# does not leave a half-applied global state; consider abort semantics.\n# FIX 5: add node.version; in tx commit, abort if the global node for\n# a key is not the one seen at tx.get() time.",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "gemma4-31b-gguf",
|
|
"timestamp": "2026-07-28T13:20:00Z",
|
|
"model_name": "Gemma 4 31B",
|
|
"quant": "GGUF",
|
|
"format": "gguf",
|
|
"tok_sec": 10.09,
|
|
"total_tokens": 4536,
|
|
"ttft_sec": 4.39,
|
|
"speed_caveat": "All Gemma 4 models ran abnormally slow (GPU offload appeared inactive despite being set), so tok/sec and TTFT are NOT representative of the model itself \u2014 likely an LM Studio/GGUF config issue. Treat speed numbers for the Gemma 4 batch as suspect.",
|
|
"filename": "outputs/gemma4-31b-gguf.py",
|
|
"tests_pass": true,
|
|
"total_score": 78,
|
|
"breakdown": {
|
|
"complexity": 17,
|
|
"concurrency": 16,
|
|
"isolation": 14,
|
|
"memory_edge_cases": 15,
|
|
"test_integrity": 16
|
|
},
|
|
"verdict": "Minor Logic Flaws",
|
|
"best_for": "Clean, correct, runnable code with solid O(1) structure and good concurrency granularity. A reliable pick for everyday caching/async work after a monotonic-clock + __slots__ pass.",
|
|
"critical_bugs": [
|
|
"Isolation leak: Transaction.get falls back to the PUBLIC cache.get, which calls _update_frequency \u2014 so reading a key inside a transaction mutates GLOBAL frequency state before commit, leaking uncommitted access patterns into global eviction order. Spec requires tx reads not to alter global freq.",
|
|
"Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments corrupt TTL eviction.",
|
|
"No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction \u2014 rubric required it for memory efficiency.",
|
|
"_delete_internal deliberately leaves empty frequency buckets in freq_map (documented but a minor memory leak: empty DoublyLinkedList objects accumulate).",
|
|
"Background evictor does list(self.cache.keys()) = O(N) snapshot every interval \u2014 a linear scan, forbidden by strict O(1).",
|
|
"No MVCC/version check on commit (lost-update possible); commit is not exception-safe across the deletes-vs-puts loops.",
|
|
"Tests pass but don't probe mid-commit read isolation or eviction-under-real-contention (capacity sized so all keys fit), so the isolation leak above goes undetected."
|
|
],
|
|
"patch_code": "# FIX 1 (the isolation leak): tx reads must NOT mutate global freq.\n# Add a read-only global lookup (no _update_frequency) and use it in tx.get:\nasync def _read_raw(self, key): # no freq bump\n node = self.cache.get(key)\n if node is None: return None\n if time.monotonic() > node.expiry:\n await self._delete_internal(key)\n return None\n return node.value\n# then in Transaction.get fallback:\n return await self._cache._read_raw(key) # NOT cache.get\n\n# FIX 2: time.time() -> time.monotonic() everywhere (get/put/_put_internal/bg loop).\n# FIX 3: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 4: prune empty freq buckets on delete, or have _evict_lfu drop them.\n# FIX 5: iterate a dedicated _ttl_keys set (batched) in the bg evictor\n# instead of list(self.cache.keys()) to stay O(batch), not O(N).",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "gemma-4-31b-qat-gguf",
|
|
"timestamp": "2026-07-28T13:25:00Z",
|
|
"model_name": "Gemma 4 31B QAT",
|
|
"quant": "QAT GGUF",
|
|
"format": "gguf",
|
|
"tok_sec": 15.0,
|
|
"total_tokens": 4552,
|
|
"ttft_sec": 4.01,
|
|
"speed_caveat": "Same as the Gemma 4 batch: GPU offload appeared inactive so tok/sec/TTFT are NOT representative of the model. Suspected LM Studio/GGUF config issue.",
|
|
"filename": "outputs/gemma-4-31b-qat-gguf.py",
|
|
"tests_pass": true,
|
|
"total_score": 70,
|
|
"breakdown": {
|
|
"complexity": 16,
|
|
"concurrency": 16,
|
|
"isolation": 11,
|
|
"memory_edge_cases": 13,
|
|
"test_integrity": 14
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Runnable and structurally sound, but the LFU eviction has a stale-min_freq capacity-breach path and the transaction API doesn't match the spec (no commit/rollback). Usable for prototypes if you fix eviction and re-skin transactions.",
|
|
"critical_bugs": [
|
|
"Capacity breach: eviction does self.freq_map[self.min_freq].pop_tail() with NO guard that the bucket exists or is non-empty, and never prunes emptied freq buckets. After manual deletes empty the min-tier, pop_tail returns None silently -> eviction fails -> cache grows PAST capacity. This is the exact stale-min_freq capacity-breach the rubric flags.",
|
|
"Non-conformant transaction API: Transaction has NO commit() or rollback() method (both required by spec). Commit happens via a separate cache.apply_transaction_changes(tx._state) \u2014 wrong surface; the test only passes because it uses this internal path.",
|
|
"Isolation leak: Transaction.get falls back to the public cache.get, which bumps global frequency before commit \u2014 uncommitted tx reads alter global eviction order.",
|
|
"Duplicated eviction logic in apply_transaction_changes re-introduces the stale-min_freq bug at line 211.",
|
|
"Uses time.time() (system clock) via _get_now() instead of time.monotonic() \u2014 NTP jumps corrupt TTL (though _get_now is a clean single fix point).",
|
|
"No __slots__ on Node/DoublyLinkedList/LFUCache/TransactionState/Transaction.",
|
|
"Background sweep does list(self.cache.keys()) = O(N) per interval.",
|
|
"Tests pass but use the non-spec commit path and don't probe capacity breach or isolation leak."
|
|
],
|
|
"patch_code": "# FIX 1 (capacity breach): guard + prune empty buckets on eviction:\nwhile self.min_freq in self.freq_map and self.freq_map[self.min_freq].size == 0:\n del self.freq_map[self.min_freq]\n self.min_freq += 1\n if not self.freq_map:\n break\nif self.min_freq not in self.freq_map:\n return # nothing to evict\nevicted = self.freq_map[self.min_freq].pop_tail()\nif evicted and self.freq_map[self.min_freq].size == 0:\n del self.freq_map[self.min_freq]\n\n# FIX 2 (conformant API): add commit/rollback to Transaction:\nasync def commit(self):\n await self._cache.apply_transaction_changes(self._state)\n self._committed = True\ndef rollback(self):\n self._state.writes.clear()\n self._committed = True\n\n# FIX 3 (isolation leak): tx.get should use a read-only global lookup\n# (no _update_freq), not the public cache.get.\n# FIX 4: _get_now returns time.monotonic().\n# FIX 5: add __slots__ to all node/list classes.",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "gemma-4-12b-coder-heretic-mxfp8-mlx",
|
|
"timestamp": "2026-07-28T13:30:00Z",
|
|
"model_name": "Gemma 4 12B Coder (fable5-composer2.5-v1-uncensored-heretic merge)",
|
|
"quant": "mxfp8 MLX",
|
|
"format": "mlx",
|
|
"tok_sec": 25.28,
|
|
"total_tokens": 2625,
|
|
"ttft_sec": 3.21,
|
|
"filename": "outputs/gemma-4-12b-coder-fable5-composer2.5-v1-uncensored-heretic-mxfp8-mlx.py",
|
|
"tests_pass": false,
|
|
"total_score": 43,
|
|
"breakdown": {
|
|
"complexity": 13,
|
|
"concurrency": 7,
|
|
"isolation": 10,
|
|
"memory_edge_cases": 9,
|
|
"test_integrity": 4
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Not usable as-is \u2014 the cache cannot store its first key and the background evictor would crash the event loop. The smallest model in the set (12B) and lowest-quality output. Avoid for systems work.",
|
|
"critical_bugs": [
|
|
"FATAL: put() line 112 does 'bucket = self.freq_buckets[self.min_freq]' after setting min_freq=1 but NEVER creates freq_buckets[1] -> KeyError: 1 on the very first put. Cache is unusable.",
|
|
"Background evictor is fundamentally broken: start_evictor defines a SYNC 'def evict_loop' and passes it to create_task; inside it calls blocking time.sleep(interval) (freezes the event loop) AND asyncio.run(...) from within a running loop -> RuntimeError. Would crash hard if ever reached.",
|
|
"Eviction-by-re-put: expired keys are 'evicted' by re-inserting them with TTL 0 (line 122) instead of deleting them \u2014 wrong semantics and triggers immediate re-eviction.",
|
|
"Transaction duplicates the entire LFU machinery (local_cache + local_freq + local_min_freq) for snapshot isolation, but _update_local_freq has the same missing-bucket KeyError (line 140).",
|
|
"Class name typo 'DoublyLinkedListList' (doubled word).",
|
|
"No __slots__; time.time() (not monotonic) throughout.",
|
|
"Tests cannot run \u2014 crash at first put."
|
|
],
|
|
"patch_code": "# FIX 1 (the fatal KeyError): create the bucket before use.\n# In put(), new-key branch:\n- bucket = self.freq_buckets[self.min_freq]\n+ bucket = self.freq_buckets.setdefault(self.min_freq, DoublyLinkedListList())\n# Same fix in _update_freq and Transaction._update_local_freq (use setdefault).\n\n# FIX 2 (the broken evictor): make it a real async task that deletes:\nasync def _evict_loop(self, interval):\n while True:\n await asyncio.sleep(interval) # async, non-blocking\n now = time.monotonic()\n async with self.global_lock:\n expired = [k for k, n in list(self.cache.items()) if now > n.ttl_expiry]\n for k in expired:\n node = self.cache.pop(k, None)\n if node:\n self.freq_buckets[node.freq].remove(node) # DELETE, not re-put\n\nasync def start_evictor(self, interval=1.0):\n self.evictor_task = asyncio.create_task(self._evict_loop(interval))\n\n# FIX 3: delete expired keys; do NOT re-insert with TTL 0.\n# FIX 4: time.time() -> time.monotonic().\n# FIX 5: add __slots__ to Node / DoublyLinkedListList / ConcurrentLFUCache / Transaction.",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "deepseekv4flash-cloud",
|
|
"timestamp": "2026-07-28T13:35:00Z",
|
|
"model_name": "DeepSeek V4 Flash (CLOUD baseline)",
|
|
"quant": "n/a (cloud)",
|
|
"format": "cloud",
|
|
"tok_sec": null,
|
|
"total_tokens": null,
|
|
"ttft_sec": null,
|
|
"speed_caveat": "Cloud model (run via opencode, not LM Studio) \u2014 tok_sec/tokens/TTFT are N/A (not measured for cloud). Included as a quality baseline against the local models. NOTE: it took 3 attempts to produce any output and ~12 minutes of thinking before succeeding \u2014 so it is a QUALITY benchmark, not a speed/usability one.",
|
|
"filename": "deepseekv4flash.py",
|
|
"tests_pass": true,
|
|
"total_score": 91,
|
|
"breakdown": {
|
|
"complexity": 18,
|
|
"concurrency": 18,
|
|
"isolation": 19,
|
|
"memory_edge_cases": 18,
|
|
"test_integrity": 18
|
|
},
|
|
"verdict": "Production Ready",
|
|
"best_for": "Reference-quality baseline (91/100) \u2014 the bar the local models are measured against. Only submission with __slots__ + time.monotonic() + delta-based transactional frequency accounting. Sets the ceiling for correctness, though its unreliability (3 attempts, 12-min think time) makes it a poor *local* daily-driver.",
|
|
"critical_bugs": [
|
|
"Two min(self._freq_to_list) linear scans in _evict_one's defensive recovery path (lines 415, 429) \u2014 only triggered when min_freq desyncs, not per-operation, but still a non-O(1) path. Could be replaced with incremental tracking.",
|
|
"Single coarse lock held across the whole commit-apply loop \u2014 not the fine-grained locking the prompt asked for.",
|
|
"__slots__ present on _Node and _DLL but not extended to Transaction / LFUCache.",
|
|
"No explicit lost-update/conflict abort on commit (delta-based freq is applied unconditionally).",
|
|
"Tests pass 20/20 but don't include a mid-commit read-isolation probe or adversarial eviction-under-contention stress."
|
|
],
|
|
"patch_code": "# These are minor refinements on an already production-ready file.\n\n# FIX 1: eliminate the recovery min() scans by keeping min_freq\n# strictly in sync on every insert/bump/remove (it already does on\n# the hot path), so the _evict_one recovery branch is unreachable and\n# can assert instead of scanning:\nassert self._min_freq in self._freq_to_list or not self._freq_to_list\n\n# FIX 2: extend __slots__ to Transaction and LFUCache.\nclass LFUCache(Generic[KT, VT]):\n __slots__ = ('_capacity','_key_to_node','_freq_to_list','_min_freq',\n '_lock','_ttl_index','_evictor_task','_closed')\n\n# FIX 3 (optional): on commit, if a key's global node changed since the\n# tx snapshot, raise LFUCacheError('lost update') instead of overwriting.",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "gemma4-26b-a4b-8bit-mlx",
|
|
"timestamp": "2026-07-28T16:25:00Z",
|
|
"model_name": "Gemma 4 26B-A4B",
|
|
"quant": "8-bit",
|
|
"param_size": "26B-A4B (MoE)",
|
|
"format": "mlx",
|
|
"tok_sec": 58.3,
|
|
"total_tokens": 7390,
|
|
"ttft_sec": 0.93,
|
|
"filename": "outputs/gemma4-26b-a4b-8bit-mlx.py",
|
|
"tests_pass": true,
|
|
"total_score": 82,
|
|
"breakdown": {
|
|
"complexity": 17,
|
|
"concurrency": 16,
|
|
"isolation": 18,
|
|
"memory_edge_cases": 15,
|
|
"test_integrity": 16
|
|
},
|
|
"verdict": "Minor Logic Flaws",
|
|
"best_for": "Tied top local scorer (82). The only local model to use delta-based transactional frequency accounting (freq bumps deferred to commit), matching the cloud baseline's isolation approach. Reliable for async/ACID-pattern work after a __slots__ + min_freq-edge pass.",
|
|
"critical_bugs": [
|
|
"Stale min_freq on manual delete: _remove_node_from_structures empties the min-freq bucket but does `pass` instead of recomputing min_freq (lines 199-202, documented as 'a simplification'). Correct only because eviction has a min_freq-in-freq_map guard + arbitrary-key fallback (lines 237-244) \u2014 latent fragility under concurrent deletes.",
|
|
"No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction despite Generic dataclasses (rubric required it for memory efficiency).",
|
|
"Background evictor does list(self.cache_data.keys()) = O(N) snapshot every interval \u2014 a linear scan, forbidden by strict O(1).",
|
|
"No MVCC/version check on commit (lost-update possible if the global key changes between tx.get and commit); commit applies deletes->reads->puts without try/except, so a mid-commit exception leaves partial state."
|
|
],
|
|
"patch_code": "# FIX 1 (stale min_freq): recompute or invalidate when the min bucket empties.\n# In _remove_node_from_structures, replace the `pass`:\nif dll.size == 0:\n del self.freq_map[node.freq]\n if self.min_freq == node.freq:\n # bump to next existing tier (frequencies are contiguous under normal use)\n self.min_freq = self.min_freq + 1 if (self.min_freq + 1) in self.freq_map else min(self.freq_map, default=1)\n\n# FIX 2: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 3: background evictor \u2014 maintain a _ttl_keys set and iterate IT in\n# batches instead of list(self.cache_data.keys()) to stay O(batch).\n# FIX 4: wrap commit's three loops in try/except with rollback semantics on failure.",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "qwen3.6-27b-8bit-mlx",
|
|
"timestamp": "2026-07-28T16:30:00Z",
|
|
"model_name": "Qwen 3.6 27B",
|
|
"quant": "8-bit",
|
|
"param_size": "27B dense",
|
|
"format": "mlx",
|
|
"tok_sec": 12.29,
|
|
"total_tokens": 12790,
|
|
"ttft_sec": 2.9,
|
|
"speed_caveat": "This model 'thought' for 12m48s before producing output and ran at 12.29 tok/sec \u2014 anomalously slow for an 8-bit MLX on M3 Max. Likely an inference/quant issue worth investigating; the slow generation is NOT representative of normal 27B-8bit throughput.",
|
|
"filename": "outputs/qwen3.6-27b-8bit-mlx.py",
|
|
"tests_pass": true,
|
|
"total_score": 78,
|
|
"breakdown": {
|
|
"complexity": 17,
|
|
"concurrency": 16,
|
|
"isolation": 14,
|
|
"memory_edge_cases": 15,
|
|
"test_integrity": 16
|
|
},
|
|
"verdict": "Minor Logic Flaws",
|
|
"best_for": "Clean, correct, runnable \u2014 same tier as Gemma 4 31B (78). Good O(1) structure and concurrency granularity. Reliable for everyday async/caching work after a monotonic-clock + __slots__ + isolation pass. Caveat: was anomalously slow to generate.",
|
|
"critical_bugs": [
|
|
"Isolation leak: Transaction.get falls back to the PUBLIC cache.get (line 78), which calls _update_freq \u2014 so reading a key inside a transaction mutates GLOBAL frequency state before commit, leaking uncommitted access patterns into global eviction order.",
|
|
"Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments corrupt TTL eviction.",
|
|
"No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction \u2014 rubric required it for memory efficiency.",
|
|
"Background _evict_loop materializes list(self.nodes.keys()) (O(N)) before checking only batch_size keys \u2014 the break caps work but not the snapshot cost, a hidden O(N) per sweep.",
|
|
"Commit re-implements put+evict inline (lines 102-118) duplicating the public path = duplicated bug surface; not wrapped in try/except so a mid-commit exception leaves partial state. No MVCC version check (lost-update possible)."
|
|
],
|
|
"patch_code": "# FIX 1 (isolation leak): add a read-only global lookup (no freq bump)\n# and use it in tx.get instead of the public cache.get:\nasync def _read_raw(self, key):\n async with self.lock:\n node = self.nodes.get(key)\n if node is None: return None\n if time.monotonic() > node.expires_at:\n self._remove_node(key); return None\n return node.value\n# then: return await self._cache._read_raw(key)\n# FIX 2: time.time() -> time.monotonic() everywhere.\n# FIX 3: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 4: maintain a _ttl_keys set; iterate IT (batched) in the bg loop\n# instead of list(self.nodes.keys()).\n# FIX 5: factor commit's put/evict to reuse the internal helpers; wrap\n# the commit loop in try/except with rollback-on-failure.",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "qwen3-coder-30b-6bit-mlx",
|
|
"timestamp": "2026-07-28T16:35:00Z",
|
|
"model_name": "Qwen3 Coder 30B",
|
|
"quant": "6-bit",
|
|
"param_size": "30B",
|
|
"format": "mlx",
|
|
"tok_sec": 72.7,
|
|
"total_tokens": 2779,
|
|
"ttft_sec": 0.9,
|
|
"filename": "outputs/qwen3-coder-30b-6bit-mlx.py",
|
|
"tests_pass": false,
|
|
"total_score": 50,
|
|
"breakdown": {
|
|
"complexity": 15,
|
|
"concurrency": 9,
|
|
"isolation": 11,
|
|
"memory_edge_cases": 11,
|
|
"test_integrity": 4
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Not usable as-is \u2014 transactions crash immediately due to an async/sync lock mismatch. The coder-specialist produced terse, fast output (2779 tok, 72.7 t/s) with competent freq-bucket structure, but fumbled the async primitive. Fix the one lock bug and it would likely score 70+.",
|
|
"critical_bugs": [
|
|
"FATAL: _transaction_lock is an asyncio.Lock() (line 101) but begin_transaction() is a SYNC def that uses synchronous `with self._transaction_lock:` (line 199). asyncio.Lock does not support the sync context-manager protocol -> TypeError on the first transaction, crashing the entire test suite.",
|
|
"Test suite cannot run: crashes at cache.begin_transaction() in main(); the transaction and concurrency assertions never execute.",
|
|
"Unused `import threading` and `import weakref` \u2014 vestigial confusion between threading and asyncio primitives.",
|
|
"Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP jumps corrupt TTL eviction.",
|
|
"No __slots__ on CacheNode/FrequencyBucket/InMemoryLFUCache/Transaction \u2014 rubric required it.",
|
|
"FrequencyBucket stores nodes in BOTH a DLL and a parallel `nodes` dict \u2014 redundant memory per bucket."
|
|
],
|
|
"patch_code": "# FIX 1 (the fatal crash): make begin_transaction async and use async with.\nasync def begin_transaction(self) -> 'Transaction':\n async with self._transaction_lock:\n self._transaction_counter += 1\n tx = Transaction(self)\n self._transactions[self._transaction_counter] = tx\n return tx\n# (and update callers: `tx = await cache.begin_transaction()`)\n# Alternative if sync creation is required: use threading.Lock for the\n# counter, but that is wrong in an asyncio codebase \u2014 go async.\n\n# FIX 2: remove unused `import threading` and `import weakref`.\n# FIX 3: time.time() -> time.monotonic() everywhere.\n# FIX 4: add __slots__ to all classes.\n# FIX 5: drop the redundant FrequencyBucket.nodes dict; the DLL already\n# tracks membership, so the dict is duplicate storage.",
|
|
"prompt_id": "lfu"
|
|
},
|
|
{
|
|
"id": "qwen3.6-35b-a3b-6bit-mlx-tts",
|
|
"prompt_id": "tts",
|
|
"timestamp": "2026-07-29T00:25:00Z",
|
|
"model_name": "Qwen 3.6 35B-A3B",
|
|
"quant": "6-bit MLX",
|
|
"param_size": "35B-A3B (MoE)",
|
|
"format": "mlx",
|
|
"tok_sec": 69.87,
|
|
"total_tokens": 11996,
|
|
"ttft_sec": 0.84,
|
|
"filename": "outputs/qwen3.6-35b-a3b-6bit-mlx-tts.py",
|
|
"tests_pass": false,
|
|
"total_score": 49,
|
|
"breakdown": {
|
|
"complexity": 10,
|
|
"concurrency": 8,
|
|
"error_handling": 13,
|
|
"resource_safety": 12,
|
|
"test_integrity": 6
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "NOT a safe offload for async-pipeline / queue work despite scoring 82 on the LFU exam. Same model+quant, different prompt: 82 -> 49. Use it for data-structure/ACID tasks; route queue/backpressure/retry work elsewhere (or to the cloud model).",
|
|
"critical_bugs": [
|
|
"FATAL: file does not parse. 'async with self._callbacks_lock' (L103) and 'await coro' (L98) are inside on_event, a SYNC def. SyntaxError before any test runs.",
|
|
"Bounded-concurrency design is broken: only ONE worker task is created (start() does a single create_task), and the Semaphore(4) is acquired inside that single loop. Effective concurrency is 1, not 4 \u2014 the semaphore is decorative.",
|
|
"Test A asserts max_concurrency <= 4, but since real concurrency is ~1 it passes trivially and proves NOTHING about the cap actually holding under saturation.",
|
|
"Cooperative cancel only sets a flag; it cannot interrupt an in-flight mock_synthesize mid-sleep (acceptable, but not 'cancel at next safe checkpoint' for a long synth).",
|
|
"submit() after stop() silently enqueues to a dead worker \u2014 no rejection, jobs leak (never processed, never failed).",
|
|
"_final_states dict grows unbounded (no eviction) \u2014 memory leak for a long-running service.",
|
|
"Callback errors are silently swallowed (broad try/except Exception) \u2014 a silent-failure pattern; debug visibility lost."
|
|
],
|
|
"patch_code": "# FIX 1 (the parse error): make on_event async (or don't await inside it).\n# Simplest correct version:\nasync def on_event(self, callback):\n async with self._callbacks_lock:\n self._callbacks.append(callback)\n\n# FIX 2 (real bounded concurrency): spawn N workers OR create_task per job\n# gated by the semaphore. Option B (concurrency from the semaphore itself):\nasync def _worker_loop(self):\n while self._running:\n job = await self._queue.get()\n # do NOT hold the semaphore in the single worker; instead launch\n # each job as its own task, gated so at most max_concurrency run:\n async def _run(j):\n async with self._semaphore:\n await self._process_job(j)\n self._queue.task_done()\n asyncio.create_task(_run(job))\n# (and add a test that submits >max_concurrency long jobs and asserts\n# exactly max_concurrency run at once.)\n\n# FIX 3: reject submit() after stop() (guard on self._running).\n# FIX 4: bound _final_states (e.g. keep last N, or evict terminal >TTL).\n# FIX 5: log callback errors instead of swallowing them silently."
|
|
},
|
|
{
|
|
"id": "qwen3.6-35b-a3b-6bit-mlx-rust",
|
|
"prompt_id": "rust",
|
|
"timestamp": "2026-07-29T00:50:00Z",
|
|
"model_name": "Qwen 3.6 35B-A3B",
|
|
"quant": "6-bit MLX",
|
|
"param_size": "35B-A3B (MoE)",
|
|
"format": "mlx",
|
|
"lang": "rust",
|
|
"tok_sec": 68.99,
|
|
"total_tokens": 13310,
|
|
"ttft_sec": 1.07,
|
|
"filename": "outputs/qwen3.6-35b-a3b-6bit-mlx-rust.rs",
|
|
"tests_pass": false,
|
|
"total_score": 50,
|
|
"breakdown": {
|
|
"ownership": 8,
|
|
"concurrency": 13,
|
|
"error_handling": 12,
|
|
"cancellation": 11,
|
|
"test_integrity": 6
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "NOT usable for Rust as-is \u2014 7 compile errors. Same model that scored 82 on LFU now: TTS 49, Rust 50. The profile is now clear: Qwen 6-bit is strong on single-file Python data-structure work but repeatedly ships non-compiling/non-parsing code on multi-task async + typed-language prompts. Keep it on Python ACID/data-structure tasks; do NOT offload Rust or async-pipeline work to it.",
|
|
"critical_bugs": [
|
|
"FATAL: does not compile \u2014 7 errors (verified with cargo 1.94). Key ones: mpsc::bounded(32) (tokio has no bounded(); should be mpsc::channel(32) \u2014 an async-std/flume API hallucination); no `main` fn (lib-style file, won't 'run directly' as the prompt required); u32/u64 type mismatch in Duration::from_millis(20 + id % 30).",
|
|
"Ownership errors (would not compile): shutdown() tries to move e.join_handle out of &WatcherEntry, and drop(self.item_tx)/drop(self.health_tx) out of &self. Needs Option<Sender> + &mut self or mem::take.",
|
|
"remove_watcher is broken: removes the HashMap entry but NEVER cancels that watcher's task. The orphaned task keeps polling until global shutdown() \u2014 contradicting the 'clean per-watcher removal' requirement.",
|
|
"Test (b) failure-injection is DEAD CODE: a global TEST_ALWAYS_FAIL flag is toggled in the test, but watcher_loop calls mock_fetch directly, not the test_fetch override that reads the flag. So the unhealthy-marking test relies on the natural ~15% failure rate over 400ms \u2014 flaky, may never reach 5 consecutive failures.",
|
|
"Test (a) asserts nothing real \u2014 the comment admits 'we trust the join'; prompt required asserting >0 items received and no hang.",
|
|
"mock_fetch randomness uses SystemTime nanos + id; across watchers polled in the same tick the high bits are shared, so failure/item counts cluster (poor randomness, not truly independent).",
|
|
"Several `let _ = tx.send(...)` silently swallow channel-closed errors."
|
|
],
|
|
"patch_code": "// FIX 1 (the API hallucination): tokio mpsc has no bounded().\n// let (item_tx, item_rx) = mpsc::bounded(32);\nlet (item_tx, item_rx) = mpsc::channel(32);\n\n// FIX 2 (type mismatch):\ntokio::time::sleep(Duration::from_millis(20 + (id as u64 % 30))).await;\n\n// FIX 3 (ownership in shutdown): store JoinHandles in Option + take them,\n// and make shutdown take &mut self (or hold senders in Option):\nstruct WatcherEntry { status: WatcherStatus, consecutive_failures: u32, join_handle: Option<JoinHandle<()>> }\n// in shutdown: let handles: Vec<_> = inner.watchers.values_mut().map(|e| e.join_handle.take()).flatten().collect();\n// drop(self.item_tx.take()) etc. with Option<Sender> fields.\n\n// FIX 4 (remove_watcher must actually stop the task): either send on a per-watcher\n// oneshot/CancellationToken, or broadcast shutdown to that watcher's sub-channel.\n// Simplest: give each watcher a CancellationToken; remove_watcher cancels it, then awaits the handle.\n\n// FIX 5 (test isolation): inject the fetch fn into watcher_loop as a parameter so tests\n// can pass a failing mock; drop the dead global flag.\n// FIX 6: add `fn main() { ... }` or make it `cargo test`-only and document that."
|
|
}
|
|
]
|
|
} |