- prompts/: LFU cache exam + 5-pillar grading rubric - outputs/: 8 model .py outputs (local + cloud baseline) - data/benchmark_history.json: graded results (scores, metrics, bugs, patches) - generate_dashboard.py: builds dashboard.html + pages/*.html from JSON - Dockerfile + DEPLOY.md: Gitea→Coolify deploy (build-step, nginx static) - .gitignore: generated HTML excluded (built on deploy) Co-Authored-By: Claude <noreply@anthropic.com>
270 lines
25 KiB
JSON
270 lines
25 KiB
JSON
{
|
|
"meta": {
|
|
"project": "Local LLM Benchmark Suite — LFU Cache & ACID Audit",
|
|
"machine": "Apple M3 Max, 48GB unified memory, LM Studio",
|
|
"exam_prompt": "prompts/lfu_cache_prompt.txt",
|
|
"grading_rubric": "prompts/grading.txt",
|
|
"pillars": ["complexity", "concurrency", "isolation", "memory_edge_cases", "test_integrity"],
|
|
"max_per_pillar": 20,
|
|
"schema_version": 1
|
|
},
|
|
"models": [
|
|
{
|
|
"id": "qwen3.6-35b-a3b-6bit-mlx",
|
|
"timestamp": "2026-07-28T13:00:00Z",
|
|
"model_name": "Qwen 3.6 35B-A3B",
|
|
"quant": "6-bit MLX",
|
|
"format": "mlx",
|
|
"tok_sec": 68.86,
|
|
"total_tokens": 14883,
|
|
"ttft_sec": 0.94,
|
|
"filename": "outputs/qwen3.6-35b-a3b-6bit-mlx.py",
|
|
"tests_pass": true,
|
|
"total_score": 82,
|
|
"breakdown": {
|
|
"complexity": 16,
|
|
"concurrency": 16,
|
|
"isolation": 17,
|
|
"memory_edge_cases": 16,
|
|
"test_integrity": 17
|
|
},
|
|
"verdict": "Minor Logic Flaws",
|
|
"best_for": "Solid daily-driver scaffolding for ACID/async patterns — produces runnable, well-structured code, but needs a human pass for __slots__, monotonic clocks, and lock granularity before production.",
|
|
"critical_bugs": [
|
|
"No __slots__ declared on Node/Transaction/LFUCache — rubric explicitly required it for memory efficiency.",
|
|
"Uses time.time() (system clock) throughout instead of time.monotonic() — NTP adjustments can cause premature/incorrect TTL eviction.",
|
|
"_cleanup_freq_lists() performs a hidden O(F) scan (iterates all freq tiers + min()), called after every eviction, background batch, AND inside commit() — violates the strict O(1) requirement.",
|
|
"Transaction commit holds the single cache lock across all write/delete/bump loops + cleanup — coarse-grained, blocks all readers for the whole commit window; no fine-grained locking.",
|
|
"Lost-update risk: commit applies tx-local writes without any MVCC/version check, so a key modified by the background evictor or another committer between tx.get() and commit() is overwritten blindly.",
|
|
"Tests dodge hard cases: 50-task stress uses unique keys with capacity 100, so no eviction-under-contention ever happens; no test for rollback-after-partial-application or mid-commit read isolation."
|
|
],
|
|
"patch_code": "# FIX 1: Add __slots__ for memory efficiency\n@dataclass\nclass Node:\n __slots__ = ('key', 'value', 'freq', 'expires_at', 'prev', 'next')\n key: Any\n value: Any\n freq: int\n expires_at: Optional[float]\n prev: Optional['Node'] = None\n next: Optional['Node'] = None\n\n# FIX 2: Use monotonic clock everywhere (get/put/_add_node/commit)\n# time.time() -> time.monotonic()\n# e.g.\nexpires_at = time.monotonic() + ttl_seconds if ttl_seconds else None\n\n# FIX 3: Make _cleanup_freq_lists O(1) — bump min_freq incrementally\n# instead of recomputing min() across all tiers:\n# In _update_freq, when emptying the min_freq bucket, only bump min_freq\n# if you're evicting from it; otherwise leave it. Delete the global\n# min(self.freq_map.keys()) scan. For background sweeps, prune empty\n# buckets lazily on next _evict() rather than scanning proactively.\n\n# FIX 4: Shrink commit critical section — apply writes into a staging\n# structure under the lock, then release; or use per-bucket locks so\n# readers on unrelated keys aren't blocked.\n\n# FIX 5: Add MVCC version to Node; in commit, raise/abort if the\n# stored version != the version seen at tx.get() time (lost-update detect)."
|
|
},
|
|
{
|
|
"id": "qwen3.6-35b-a3b-4bit-mlx",
|
|
"timestamp": "2026-07-28T13:05:00Z",
|
|
"model_name": "Qwen 3.6 35B-A3B",
|
|
"quant": "4-bit MLX",
|
|
"format": "mlx",
|
|
"tok_sec": 83.31,
|
|
"total_tokens": 13278,
|
|
"ttft_sec": 0.73,
|
|
"filename": "outputs/qwen3.6-35b-a3b-4bit-mlx.py",
|
|
"tests_pass": false,
|
|
"total_score": 57,
|
|
"breakdown": {
|
|
"complexity": 15,
|
|
"concurrency": 15,
|
|
"isolation": 13,
|
|
"memory_edge_cases": 10,
|
|
"test_integrity": 4
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Not recommended for systems code as-is. The 4-bit quant degrades logic sharply vs the 6-bit sibling (82->57) — usable only for boilerplate/scaffolding drafts that a human will heavily rewrite.",
|
|
"critical_bugs": [
|
|
"FATAL: _evict() double-removes nodes — pop() already calls remove() (nulling node.prev/next), then _remove_node() calls remove() AGAIN -> AttributeError: 'NoneType' on first eviction. The cache cannot survive reaching capacity.",
|
|
"Test suite never executes: test_lfu_eviction crashes at the first eviction, so the 'All tests passed' message is unreachable and assertions are effectively unverified.",
|
|
"Background _eviction_loop materializes list(self.key_to_node.keys())[:50] every sweep — an O(N) linear scan, forbidden by the strict O(1) requirement.",
|
|
"_FreqList.pop() has no empty-guard — calling pop() on an empty list dereferences self.head.next (the dummy tail) and corrupts the DLL.",
|
|
"Transaction _apply_put applies the buffered value into the EXACT original_node captured at tx.put() time; if the global cache evicted/relocated that node between put and commit, you mutate a stale/dangling node (no MVCC/version check).",
|
|
"Commit is not atomic across exceptions: a crash mid-_apply loop leaves half-applied global state with no rollback.",
|
|
"Uses time.time() (system clock) throughout instead of time.monotonic() — NTP jumps corrupt TTL eviction."
|
|
],
|
|
"patch_code": "# FIX 1 (the crash): _evict double-removes. pop() already unlinks,\n# so do NOT call _remove_node on a popped node. Either:\n# (a) pop and then only delete the key_map entry + min_freq bookkeeping:\ndef _evict(self):\n if not self.freq_to_list:\n return\n evict_list = self.freq_to_list[self.min_freq]\n if evict_list.size == 0: # guard against empty\n del self.freq_to_list[self.min_freq]\n return\n node = evict_list.pop() # pop() unlinks + nulls prev/next\n del self.key_to_node[node.key] # DON'T call _remove_node again\n if self.freq_to_list[self.min_freq].size == 0:\n del self.freq_to_list[self.min_freq]\n self.min_freq += 1\n\n# FIX 2: _FreqList.pop empty-guard\ndef pop(self) -> _Node:\n if self.size == 0:\n raise IndexError('pop from empty _FreqList')\n node = self.head.next\n self.remove(node)\n return node\n\n# FIX 3: kill the O(N) scan in background sweep — maintain a separate\n# set of keys that have a TTL, and iterate that set in batches:\nasync with self.lock:\n batch = list(self._ttl_keys)[:50]\n for k in batch:\n node = self.key_to_node.get(k)\n if node and 0 < node.expires_at <= time.monotonic():\n self._remove_node(node)\n\n# FIX 4: time.time() -> time.monotonic() everywhere.\n# FIX 5: add node.version; in tx._apply_put, abort/refresh if\n# cache.key_to_node[key] is a different node than original_node."
|
|
},
|
|
{
|
|
"id": "qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf",
|
|
"timestamp": "2026-07-28T13:10:00Z",
|
|
"model_name": "Qwen 3.6 35B-A3B (uncensored hauhaucs aggressive)",
|
|
"quant": "GGUF",
|
|
"format": "gguf",
|
|
"tok_sec": 62.54,
|
|
"total_tokens": 13897,
|
|
"ttft_sec": 1.09,
|
|
"filename": "outputs/qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf.py",
|
|
"tests_pass": false,
|
|
"total_score": 49,
|
|
"breakdown": {
|
|
"complexity": 12,
|
|
"concurrency": 10,
|
|
"isolation": 12,
|
|
"memory_edge_cases": 11,
|
|
"test_integrity": 4
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Not recommended for production code. Reasonable API shape and correctly used time.monotonic(), but the module does not parse (syntax error), contains an infinite while:pass loop, and has data races. Avoid for systems/concurrency work.",
|
|
"critical_bugs": [
|
|
"SyntaxError: line 305 'assert val := await cache.get(...)' is invalid Python — walrus operator cannot appear in an assert statement. The ENTIRE module fails to compile, so nothing runs and no test can execute.",
|
|
"Infinite busy-loop: _evict_lfu lines 159-160 — 'while self.min_freq in self.freq_map and self.min_freq < max(...): pass' has an empty body that never updates min_freq, recomputes max() (O(F)) each iteration, and can never terminate.",
|
|
"Hidden O(F) scan: min(self.freq_map.keys()) / max(self.freq_map.keys()) appears at 6 call sites (every eviction and removal) — violates the strict O(1) requirement.",
|
|
"Race condition: get() and put() perform lazy-TTL _remove_key() BEFORE acquiring the lock (lines 71-73, 107-108), mutating shared state unlocked while other coroutines read/write.",
|
|
"Deadlock risk: background_loop holds self._lock, then calls await self._remove_key() which is itself a lock-acquiring method — asyncio.Lock is NOT reentrant -> deadlock when the evictor runs.",
|
|
"No __slots__ on _Node despite using a dataclass (rubric required it for memory efficiency).",
|
|
"_remove_key will KeyError on self.freq_map[freq] if a concurrent operation already deleted that bucket."
|
|
],
|
|
"patch_code": "# FIX 1 (the parse error): assign first, then assert.\nval_d = await cache.get(\"D\")\nassert val_d, \"D should exist\"\nval_a = await cache.get(\"A\")\nassert val_a, \"A should exist (highest freq)\"\n\n# FIX 2: delete the broken while:pass loop. Bump min_freq incrementally:\n# only when the min_freq bucket empties, and only ever UP by 1 (a key\n# whose freq increased must land at min_freq+1). Never call min()/max().\nif not old_bucket:\n del self.freq_map[old_freq]\n if self.min_freq == old_freq:\n self.min_freq += 1 # next tier up; never scan\n\n# FIX 3: do ALL lazy eviction INSIDE the lock, not before it:\nasync def get(self, key):\n async with self._lock:\n if self.ttl_map.get(key, inf) <= time.monotonic():\n await self._remove_key(key) # now locked\n return None\n ...\n\n# FIX 4: make _remove_key a non-locking private helper, called from\n# inside already-locked public methods, so background_loop doesn't try\n# to re-acquire the non-reentrant asyncio.Lock.\n\n# FIX 5: add __slots__ = ('key','value','ttl_expiry') to _Node."
|
|
},
|
|
{
|
|
"id": "kat-coder-v2.5-dev-xl-mlx",
|
|
"timestamp": "2026-07-28T13:15:00Z",
|
|
"model_name": "KAT-Coder v2.5 Dev XL",
|
|
"quant": "MLX",
|
|
"format": "mlx",
|
|
"tok_sec": 65.26,
|
|
"total_tokens": 6172,
|
|
"ttft_sec": 7.42,
|
|
"filename": "outputs/kat-coder-v2.5-dev-xl-mlx.py",
|
|
"tests_pass": false,
|
|
"total_score": 65,
|
|
"breakdown": {
|
|
"complexity": 15,
|
|
"concurrency": 16,
|
|
"isolation": 16,
|
|
"memory_edge_cases": 13,
|
|
"test_integrity": 5
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Promising code-design instincts (cleanest abstractions and best transaction isolation design in the set) but undone by a single fatal one-line bug that stops it running. With the bug fixed it would likely score 80+; as-is, only useful as a structural reference.",
|
|
"critical_bugs": [
|
|
"FATAL: _put_internal line 305 inserts a NEW key with 'self._freq_map[1].push_front(...)' but never ensures the freq-1 bucket exists — KeyError: 1 on the very first put. The cache cannot store a single key. The _ensure_freq_list(1) helper it should use exists and is used correctly everywhere else (lines 294, 354).",
|
|
"Transaction.commit() calls _put_internal for buffered writes, so it hits the same KeyError: 1 — committed transactions crash too.",
|
|
"Test suite cannot execute: crashes at the first cache.put() in main(); the well-built test harness (pass/fail counter, 4 real scenarios) validates nothing.",
|
|
"No __slots__ on _DLLNode/_CacheNode/_DoublyLinkedList despite the rubric requiring it for memory efficiency.",
|
|
"_evict_node uses min(self._freq_map) (line 325) when the min-tier empties — a hidden O(F) scan, violating strict O(1).",
|
|
"No MVCC/version check on transaction commit (lost-update possible if the global key is modified between tx.get and commit); commit is not exception-safe across the writes-vs-deletes loops."
|
|
],
|
|
"patch_code": "# FIX 1 (the fatal one-liner): use the helper that already exists.\n# line 305, in _put_internal, new-key branch:\n- self._freq_map[1].push_front(dll_node)\n+ self._ensure_freq_list(1).push_front(dll_node)\n# (This single change makes the cache and transactions functional.)\n\n# FIX 2: replace the O(F) min() scan with an incremental bump:\n# in _evict_node, when the min-tier bucket empties, min_freq is the\n# lowest remaining tier. Since freq only ever increments by 1, the\n# next min is almost always min_freq+1; track it incrementally rather\n# than scanning. Or, since this only happens on full eviction, accept\n# O(F) but only on the empty-cache edge — document it.\n\n# FIX 3: add __slots__ to all internal classes:\nclass _CacheNode:\n __slots__ = ('key','value','ttl_seconds','expiry_time','freq','dll_node')\n ...\n\n# FIX 4: wrap commit applies in try/except so a mid-commit exception\n# does not leave a half-applied global state; consider abort semantics.\n# FIX 5: add node.version; in tx commit, abort if the global node for\n# a key is not the one seen at tx.get() time."
|
|
},
|
|
{
|
|
"id": "gemma4-31b-gguf",
|
|
"timestamp": "2026-07-28T13:20:00Z",
|
|
"model_name": "Gemma 4 31B",
|
|
"quant": "GGUF",
|
|
"format": "gguf",
|
|
"tok_sec": 10.09,
|
|
"total_tokens": 4536,
|
|
"ttft_sec": 4.39,
|
|
"speed_caveat": "All Gemma 4 models ran abnormally slow (GPU offload appeared inactive despite being set), so tok/sec and TTFT are NOT representative of the model itself — likely an LM Studio/GGUF config issue. Treat speed numbers for the Gemma 4 batch as suspect.",
|
|
"filename": "outputs/gemma4-31b-gguf.py",
|
|
"tests_pass": true,
|
|
"total_score": 78,
|
|
"breakdown": {
|
|
"complexity": 17,
|
|
"concurrency": 16,
|
|
"isolation": 14,
|
|
"memory_edge_cases": 15,
|
|
"test_integrity": 16
|
|
},
|
|
"verdict": "Minor Logic Flaws",
|
|
"best_for": "Clean, correct, runnable code with solid O(1) structure and good concurrency granularity. A reliable pick for everyday caching/async work after a monotonic-clock + __slots__ pass.",
|
|
"critical_bugs": [
|
|
"Isolation leak: Transaction.get falls back to the PUBLIC cache.get, which calls _update_frequency — so reading a key inside a transaction mutates GLOBAL frequency state before commit, leaking uncommitted access patterns into global eviction order. Spec requires tx reads not to alter global freq.",
|
|
"Uses time.time() (system clock) throughout instead of time.monotonic() — NTP adjustments corrupt TTL eviction.",
|
|
"No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction — rubric required it for memory efficiency.",
|
|
"_delete_internal deliberately leaves empty frequency buckets in freq_map (documented but a minor memory leak: empty DoublyLinkedList objects accumulate).",
|
|
"Background evictor does list(self.cache.keys()) = O(N) snapshot every interval — a linear scan, forbidden by strict O(1).",
|
|
"No MVCC/version check on commit (lost-update possible); commit is not exception-safe across the deletes-vs-puts loops.",
|
|
"Tests pass but don't probe mid-commit read isolation or eviction-under-real-contention (capacity sized so all keys fit), so the isolation leak above goes undetected."
|
|
],
|
|
"patch_code": "# FIX 1 (the isolation leak): tx reads must NOT mutate global freq.\n# Add a read-only global lookup (no _update_frequency) and use it in tx.get:\nasync def _read_raw(self, key): # no freq bump\n node = self.cache.get(key)\n if node is None: return None\n if time.monotonic() > node.expiry:\n await self._delete_internal(key)\n return None\n return node.value\n# then in Transaction.get fallback:\n return await self._cache._read_raw(key) # NOT cache.get\n\n# FIX 2: time.time() -> time.monotonic() everywhere (get/put/_put_internal/bg loop).\n# FIX 3: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 4: prune empty freq buckets on delete, or have _evict_lfu drop them.\n# FIX 5: iterate a dedicated _ttl_keys set (batched) in the bg evictor\n# instead of list(self.cache.keys()) to stay O(batch), not O(N)."
|
|
},
|
|
{
|
|
"id": "gemma-4-31b-qat-gguf",
|
|
"timestamp": "2026-07-28T13:25:00Z",
|
|
"model_name": "Gemma 4 31B QAT",
|
|
"quant": "QAT GGUF",
|
|
"format": "gguf",
|
|
"tok_sec": 15.0,
|
|
"total_tokens": 4552,
|
|
"ttft_sec": 4.01,
|
|
"speed_caveat": "Same as the Gemma 4 batch: GPU offload appeared inactive so tok/sec/TTFT are NOT representative of the model. Suspected LM Studio/GGUF config issue.",
|
|
"filename": "outputs/gemma-4-31b-qat-gguf.py",
|
|
"tests_pass": true,
|
|
"total_score": 70,
|
|
"breakdown": {
|
|
"complexity": 16,
|
|
"concurrency": 16,
|
|
"isolation": 11,
|
|
"memory_edge_cases": 13,
|
|
"test_integrity": 14
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Runnable and structurally sound, but the LFU eviction has a stale-min_freq capacity-breach path and the transaction API doesn't match the spec (no commit/rollback). Usable for prototypes if you fix eviction and re-skin transactions.",
|
|
"critical_bugs": [
|
|
"Capacity breach: eviction does self.freq_map[self.min_freq].pop_tail() with NO guard that the bucket exists or is non-empty, and never prunes emptied freq buckets. After manual deletes empty the min-tier, pop_tail returns None silently -> eviction fails -> cache grows PAST capacity. This is the exact stale-min_freq capacity-breach the rubric flags.",
|
|
"Non-conformant transaction API: Transaction has NO commit() or rollback() method (both required by spec). Commit happens via a separate cache.apply_transaction_changes(tx._state) — wrong surface; the test only passes because it uses this internal path.",
|
|
"Isolation leak: Transaction.get falls back to the public cache.get, which bumps global frequency before commit — uncommitted tx reads alter global eviction order.",
|
|
"Duplicated eviction logic in apply_transaction_changes re-introduces the stale-min_freq bug at line 211.",
|
|
"Uses time.time() (system clock) via _get_now() instead of time.monotonic() — NTP jumps corrupt TTL (though _get_now is a clean single fix point).",
|
|
"No __slots__ on Node/DoublyLinkedList/LFUCache/TransactionState/Transaction.",
|
|
"Background sweep does list(self.cache.keys()) = O(N) per interval.",
|
|
"Tests pass but use the non-spec commit path and don't probe capacity breach or isolation leak."
|
|
],
|
|
"patch_code": "# FIX 1 (capacity breach): guard + prune empty buckets on eviction:\nwhile self.min_freq in self.freq_map and self.freq_map[self.min_freq].size == 0:\n del self.freq_map[self.min_freq]\n self.min_freq += 1\n if not self.freq_map:\n break\nif self.min_freq not in self.freq_map:\n return # nothing to evict\nevicted = self.freq_map[self.min_freq].pop_tail()\nif evicted and self.freq_map[self.min_freq].size == 0:\n del self.freq_map[self.min_freq]\n\n# FIX 2 (conformant API): add commit/rollback to Transaction:\nasync def commit(self):\n await self._cache.apply_transaction_changes(self._state)\n self._committed = True\ndef rollback(self):\n self._state.writes.clear()\n self._committed = True\n\n# FIX 3 (isolation leak): tx.get should use a read-only global lookup\n# (no _update_freq), not the public cache.get.\n# FIX 4: _get_now returns time.monotonic().\n# FIX 5: add __slots__ to all node/list classes."
|
|
},
|
|
{
|
|
"id": "gemma-4-12b-coder-heretic-mxfp8-mlx",
|
|
"timestamp": "2026-07-28T13:30:00Z",
|
|
"model_name": "Gemma 4 12B Coder (fable5-composer2.5-v1-uncensored-heretic merge)",
|
|
"quant": "mxfp8 MLX",
|
|
"format": "mlx",
|
|
"tok_sec": 25.28,
|
|
"total_tokens": 2625,
|
|
"ttft_sec": 3.21,
|
|
"filename": "outputs/gemma-4-12b-coder-fable5-composer2.5-v1-uncensored-heretic-mxfp8-mlx.py",
|
|
"tests_pass": false,
|
|
"total_score": 43,
|
|
"breakdown": {
|
|
"complexity": 13,
|
|
"concurrency": 7,
|
|
"isolation": 10,
|
|
"memory_edge_cases": 9,
|
|
"test_integrity": 4
|
|
},
|
|
"verdict": "Critical Bugs",
|
|
"best_for": "Not usable as-is — the cache cannot store its first key and the background evictor would crash the event loop. The smallest model in the set (12B) and lowest-quality output. Avoid for systems work.",
|
|
"critical_bugs": [
|
|
"FATAL: put() line 112 does 'bucket = self.freq_buckets[self.min_freq]' after setting min_freq=1 but NEVER creates freq_buckets[1] -> KeyError: 1 on the very first put. Cache is unusable.",
|
|
"Background evictor is fundamentally broken: start_evictor defines a SYNC 'def evict_loop' and passes it to create_task; inside it calls blocking time.sleep(interval) (freezes the event loop) AND asyncio.run(...) from within a running loop -> RuntimeError. Would crash hard if ever reached.",
|
|
"Eviction-by-re-put: expired keys are 'evicted' by re-inserting them with TTL 0 (line 122) instead of deleting them — wrong semantics and triggers immediate re-eviction.",
|
|
"Transaction duplicates the entire LFU machinery (local_cache + local_freq + local_min_freq) for snapshot isolation, but _update_local_freq has the same missing-bucket KeyError (line 140).",
|
|
"Class name typo 'DoublyLinkedListList' (doubled word).",
|
|
"No __slots__; time.time() (not monotonic) throughout.",
|
|
"Tests cannot run — crash at first put."
|
|
],
|
|
"patch_code": "# FIX 1 (the fatal KeyError): create the bucket before use.\n# In put(), new-key branch:\n- bucket = self.freq_buckets[self.min_freq]\n+ bucket = self.freq_buckets.setdefault(self.min_freq, DoublyLinkedListList())\n# Same fix in _update_freq and Transaction._update_local_freq (use setdefault).\n\n# FIX 2 (the broken evictor): make it a real async task that deletes:\nasync def _evict_loop(self, interval):\n while True:\n await asyncio.sleep(interval) # async, non-blocking\n now = time.monotonic()\n async with self.global_lock:\n expired = [k for k, n in list(self.cache.items()) if now > n.ttl_expiry]\n for k in expired:\n node = self.cache.pop(k, None)\n if node:\n self.freq_buckets[node.freq].remove(node) # DELETE, not re-put\n\nasync def start_evictor(self, interval=1.0):\n self.evictor_task = asyncio.create_task(self._evict_loop(interval))\n\n# FIX 3: delete expired keys; do NOT re-insert with TTL 0.\n# FIX 4: time.time() -> time.monotonic().\n# FIX 5: add __slots__ to Node / DoublyLinkedListList / ConcurrentLFUCache / Transaction."
|
|
},
|
|
{
|
|
"id": "deepseekv4flash-cloud",
|
|
"timestamp": "2026-07-28T13:35:00Z",
|
|
"model_name": "DeepSeek V4 Flash (CLOUD baseline)",
|
|
"quant": "n/a (cloud)",
|
|
"format": "cloud",
|
|
"tok_sec": null,
|
|
"total_tokens": null,
|
|
"ttft_sec": null,
|
|
"speed_caveat": "Cloud model (run via opencode, not LM Studio) — tok_sec/tokens/TTFT are N/A (not measured for cloud). Included as a quality baseline against the local models. NOTE: it took 3 attempts to produce any output and ~12 minutes of thinking before succeeding — so it is a QUALITY benchmark, not a speed/usability one.",
|
|
"filename": "deepseekv4flash.py",
|
|
"tests_pass": true,
|
|
"total_score": 91,
|
|
"breakdown": {
|
|
"complexity": 18,
|
|
"concurrency": 18,
|
|
"isolation": 19,
|
|
"memory_edge_cases": 18,
|
|
"test_integrity": 18
|
|
},
|
|
"verdict": "Production Ready",
|
|
"best_for": "Reference-quality baseline (91/100) — the bar the local models are measured against. Only submission with __slots__ + time.monotonic() + delta-based transactional frequency accounting. Sets the ceiling for correctness, though its unreliability (3 attempts, 12-min think time) makes it a poor *local* daily-driver.",
|
|
"critical_bugs": [
|
|
"Two min(self._freq_to_list) linear scans in _evict_one's defensive recovery path (lines 415, 429) — only triggered when min_freq desyncs, not per-operation, but still a non-O(1) path. Could be replaced with incremental tracking.",
|
|
"Single coarse lock held across the whole commit-apply loop — not the fine-grained locking the prompt asked for.",
|
|
"__slots__ present on _Node and _DLL but not extended to Transaction / LFUCache.",
|
|
"No explicit lost-update/conflict abort on commit (delta-based freq is applied unconditionally).",
|
|
"Tests pass 20/20 but don't include a mid-commit read-isolation probe or adversarial eviction-under-contention stress."
|
|
],
|
|
"patch_code": "# These are minor refinements on an already production-ready file.\n\n# FIX 1: eliminate the recovery min() scans by keeping min_freq\n# strictly in sync on every insert/bump/remove (it already does on\n# the hot path), so the _evict_one recovery branch is unreachable and\n# can assert instead of scanning:\nassert self._min_freq in self._freq_to_list or not self._freq_to_list\n\n# FIX 2: extend __slots__ to Transaction and LFUCache.\nclass LFUCache(Generic[KT, VT]):\n __slots__ = ('_capacity','_key_to_node','_freq_to_list','_min_freq',\n '_lock','_ttl_index','_evictor_task','_closed')\n\n# FIX 3 (optional): on commit, if a key's global node changed since the\n# tx snapshot, raise LFUCacheError('lost update') instead of overwriting."
|
|
}
|
|
]
|
|
}
|