admin/modelTesting
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Grade 3 more models + dashboard v2 layout (quant/format as first-class)

Browse Source 
New graded (11 total now):
  gemma4-26b-a4b-8bit-mlx    82  Minor Flaws  (tied top local; delta-based tx freq)
  qwen3.6-27b-8bit-mlx       78  Minor Flaws  (clean; anom. slow generation flagged)
  qwen3-coder-30b-6bit-mlx   50  Critical     (asyncio.Lock used with sync with -> crash)

Dashboard redesign:
  - Bar chart is now the full-width hero row (was cramped half-width)
  - 4 stat tiles squished 2x2 beside the radar up top
  - Quant + Format are dedicated columns in the leaderboard (MLX/GGUF/CLOUD chips)
  - New 'Format & Quant Showdown' panel: groups same-family variants so
    GGUF-vs-MLX and quant-depth comparisons are side by side
  - Bar-chart axis labels now include the quant so duplicate model names
    are distinguishable, with rotation for readability

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
admin
2026-07-28 16:51:50 -07:00
co-authored by Claude
parent 4056470a46
commit 8d2d5cd562
5 changed files with 1372 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/benchmark_history.json
+138 -38
View File
@@ -1,10 +1,16 @@
{
"meta": {
"project": "Local LLM Benchmark Suite — LFU Cache & ACID Audit",
"project": "Local LLM Benchmark Suite \u2014 LFU Cache & ACID Audit",
"machine": "Apple M3 Max, 48GB unified memory, LM Studio",
"exam_prompt": "prompts/lfu_cache_prompt.txt",
"grading_rubric": "prompts/grading.txt",
"pillars": ["complexity", "concurrency", "isolation", "memory_edge_cases", "test_integrity"],
"pillars": [
"complexity",
"concurrency",
"isolation",
"memory_edge_cases",
"test_integrity"
],
"max_per_pillar": 20,
"schema_version": 1
},
@@ -29,16 +35,16 @@
"test_integrity": 17
},
"verdict": "Minor Logic Flaws",
"best_for": "Solid daily-driver scaffolding for ACID/async patterns — produces runnable, well-structured code, but needs a human pass for __slots__, monotonic clocks, and lock granularity before production.",
"best_for": "Solid daily-driver scaffolding for ACID/async patterns \u2014 produces runnable, well-structured code, but needs a human pass for __slots__, monotonic clocks, and lock granularity before production.",
"critical_bugs": [
"No __slots__ declared on Node/Transaction/LFUCache — rubric explicitly required it for memory efficiency.",
"Uses time.time() (system clock) throughout instead of time.monotonic() — NTP adjustments can cause premature/incorrect TTL eviction.",
"_cleanup_freq_lists() performs a hidden O(F) scan (iterates all freq tiers + min()), called after every eviction, background batch, AND inside commit() — violates the strict O(1) requirement.",
"Transaction commit holds the single cache lock across all write/delete/bump loops + cleanup — coarse-grained, blocks all readers for the whole commit window; no fine-grained locking.",
"No __slots__ declared on Node/Transaction/LFUCache \u2014 rubric explicitly required it for memory efficiency.",
"Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments can cause premature/incorrect TTL eviction.",
"_cleanup_freq_lists() performs a hidden O(F) scan (iterates all freq tiers + min()), called after every eviction, background batch, AND inside commit() \u2014 violates the strict O(1) requirement.",
"Transaction commit holds the single cache lock across all write/delete/bump loops + cleanup \u2014 coarse-grained, blocks all readers for the whole commit window; no fine-grained locking.",
"Lost-update risk: commit applies tx-local writes without any MVCC/version check, so a key modified by the background evictor or another committer between tx.get() and commit() is overwritten blindly.",
"Tests dodge hard cases: 50-task stress uses unique keys with capacity 100, so no eviction-under-contention ever happens; no test for rollback-after-partial-application or mid-commit read isolation."
],
"patch_code": "# FIX 1: Add __slots__ for memory efficiency\n@dataclass\nclass Node:\n __slots__ = ('key', 'value', 'freq', 'expires_at', 'prev', 'next')\n key: Any\n value: Any\n freq: int\n expires_at: Optional[float]\n prev: Optional['Node'] = None\n next: Optional['Node'] = None\n\n# FIX 2: Use monotonic clock everywhere (get/put/_add_node/commit)\n# time.time() -> time.monotonic()\n# e.g.\nexpires_at = time.monotonic() + ttl_seconds if ttl_seconds else None\n\n# FIX 3: Make _cleanup_freq_lists O(1) — bump min_freq incrementally\n# instead of recomputing min() across all tiers:\n# In _update_freq, when emptying the min_freq bucket, only bump min_freq\n# if you're evicting from it; otherwise leave it. Delete the global\n# min(self.freq_map.keys()) scan. For background sweeps, prune empty\n# buckets lazily on next _evict() rather than scanning proactively.\n\n# FIX 4: Shrink commit critical section — apply writes into a staging\n# structure under the lock, then release; or use per-bucket locks so\n# readers on unrelated keys aren't blocked.\n\n# FIX 5: Add MVCC version to Node; in commit, raise/abort if the\n# stored version != the version seen at tx.get() time (lost-update detect)."
"patch_code": "# FIX 1: Add __slots__ for memory efficiency\n@dataclass\nclass Node:\n __slots__ = ('key', 'value', 'freq', 'expires_at', 'prev', 'next')\n key: Any\n value: Any\n freq: int\n expires_at: Optional[float]\n prev: Optional['Node'] = None\n next: Optional['Node'] = None\n\n# FIX 2: Use monotonic clock everywhere (get/put/_add_node/commit)\n# time.time() -> time.monotonic()\n# e.g.\nexpires_at = time.monotonic() + ttl_seconds if ttl_seconds else None\n\n# FIX 3: Make _cleanup_freq_lists O(1) \u2014 bump min_freq incrementally\n# instead of recomputing min() across all tiers:\n# In _update_freq, when emptying the min_freq bucket, only bump min_freq\n# if you're evicting from it; otherwise leave it. Delete the global\n# min(self.freq_map.keys()) scan. For background sweeps, prune empty\n# buckets lazily on next _evict() rather than scanning proactively.\n\n# FIX 4: Shrink commit critical section \u2014 apply writes into a staging\n# structure under the lock, then release; or use per-bucket locks so\n# readers on unrelated keys aren't blocked.\n\n# FIX 5: Add MVCC version to Node; in commit, raise/abort if the\n# stored version != the version seen at tx.get() time (lost-update detect)."
},
{
"id": "qwen3.6-35b-a3b-4bit-mlx",
@@ -60,17 +66,17 @@
"test_integrity": 4
},
"verdict": "Critical Bugs",
"best_for": "Not recommended for systems code as-is. The 4-bit quant degrades logic sharply vs the 6-bit sibling (82->57) — usable only for boilerplate/scaffolding drafts that a human will heavily rewrite.",
"best_for": "Not recommended for systems code as-is. The 4-bit quant degrades logic sharply vs the 6-bit sibling (82->57) \u2014 usable only for boilerplate/scaffolding drafts that a human will heavily rewrite.",
"critical_bugs": [
"FATAL: _evict() double-removes nodes — pop() already calls remove() (nulling node.prev/next), then _remove_node() calls remove() AGAIN -> AttributeError: 'NoneType' on first eviction. The cache cannot survive reaching capacity.",
"FATAL: _evict() double-removes nodes \u2014 pop() already calls remove() (nulling node.prev/next), then _remove_node() calls remove() AGAIN -> AttributeError: 'NoneType' on first eviction. The cache cannot survive reaching capacity.",
"Test suite never executes: test_lfu_eviction crashes at the first eviction, so the 'All tests passed' message is unreachable and assertions are effectively unverified.",
"Background _eviction_loop materializes list(self.key_to_node.keys())[:50] every sweep — an O(N) linear scan, forbidden by the strict O(1) requirement.",
"_FreqList.pop() has no empty-guard — calling pop() on an empty list dereferences self.head.next (the dummy tail) and corrupts the DLL.",
"Background _eviction_loop materializes list(self.key_to_node.keys())[:50] every sweep \u2014 an O(N) linear scan, forbidden by the strict O(1) requirement.",
"_FreqList.pop() has no empty-guard \u2014 calling pop() on an empty list dereferences self.head.next (the dummy tail) and corrupts the DLL.",
"Transaction _apply_put applies the buffered value into the EXACT original_node captured at tx.put() time; if the global cache evicted/relocated that node between put and commit, you mutate a stale/dangling node (no MVCC/version check).",
"Commit is not atomic across exceptions: a crash mid-_apply loop leaves half-applied global state with no rollback.",
"Uses time.time() (system clock) throughout instead of time.monotonic() — NTP jumps corrupt TTL eviction."
"Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP jumps corrupt TTL eviction."
],
"patch_code": "# FIX 1 (the crash): _evict double-removes. pop() already unlinks,\n# so do NOT call _remove_node on a popped node. Either:\n# (a) pop and then only delete the key_map entry + min_freq bookkeeping:\ndef _evict(self):\n if not self.freq_to_list:\n return\n evict_list = self.freq_to_list[self.min_freq]\n if evict_list.size == 0: # guard against empty\n del self.freq_to_list[self.min_freq]\n return\n node = evict_list.pop() # pop() unlinks + nulls prev/next\n del self.key_to_node[node.key] # DON'T call _remove_node again\n if self.freq_to_list[self.min_freq].size == 0:\n del self.freq_to_list[self.min_freq]\n self.min_freq += 1\n\n# FIX 2: _FreqList.pop empty-guard\ndef pop(self) -> _Node:\n if self.size == 0:\n raise IndexError('pop from empty _FreqList')\n node = self.head.next\n self.remove(node)\n return node\n\n# FIX 3: kill the O(N) scan in background sweep — maintain a separate\n# set of keys that have a TTL, and iterate that set in batches:\nasync with self.lock:\n batch = list(self._ttl_keys)[:50]\n for k in batch:\n node = self.key_to_node.get(k)\n if node and 0 < node.expires_at <= time.monotonic():\n self._remove_node(node)\n\n# FIX 4: time.time() -> time.monotonic() everywhere.\n# FIX 5: add node.version; in tx._apply_put, abort/refresh if\n# cache.key_to_node[key] is a different node than original_node."
"patch_code": "# FIX 1 (the crash): _evict double-removes. pop() already unlinks,\n# so do NOT call _remove_node on a popped node. Either:\n# (a) pop and then only delete the key_map entry + min_freq bookkeeping:\ndef _evict(self):\n if not self.freq_to_list:\n return\n evict_list = self.freq_to_list[self.min_freq]\n if evict_list.size == 0: # guard against empty\n del self.freq_to_list[self.min_freq]\n return\n node = evict_list.pop() # pop() unlinks + nulls prev/next\n del self.key_to_node[node.key] # DON'T call _remove_node again\n if self.freq_to_list[self.min_freq].size == 0:\n del self.freq_to_list[self.min_freq]\n self.min_freq += 1\n\n# FIX 2: _FreqList.pop empty-guard\ndef pop(self) -> _Node:\n if self.size == 0:\n raise IndexError('pop from empty _FreqList')\n node = self.head.next\n self.remove(node)\n return node\n\n# FIX 3: kill the O(N) scan in background sweep \u2014 maintain a separate\n# set of keys that have a TTL, and iterate that set in batches:\nasync with self.lock:\n batch = list(self._ttl_keys)[:50]\n for k in batch:\n node = self.key_to_node.get(k)\n if node and 0 < node.expires_at <= time.monotonic():\n self._remove_node(node)\n\n# FIX 4: time.time() -> time.monotonic() everywhere.\n# FIX 5: add node.version; in tx._apply_put, abort/refresh if\n# cache.key_to_node[key] is a different node than original_node."
},
{
"id": "qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive-gguf",
@@ -94,11 +100,11 @@
"verdict": "Critical Bugs",
"best_for": "Not recommended for production code. Reasonable API shape and correctly used time.monotonic(), but the module does not parse (syntax error), contains an infinite while:pass loop, and has data races. Avoid for systems/concurrency work.",
"critical_bugs": [
"SyntaxError: line 305 'assert val := await cache.get(...)' is invalid Python — walrus operator cannot appear in an assert statement. The ENTIRE module fails to compile, so nothing runs and no test can execute.",
"Infinite busy-loop: _evict_lfu lines 159-160 — 'while self.min_freq in self.freq_map and self.min_freq < max(...): pass' has an empty body that never updates min_freq, recomputes max() (O(F)) each iteration, and can never terminate.",
"Hidden O(F) scan: min(self.freq_map.keys()) / max(self.freq_map.keys()) appears at 6 call sites (every eviction and removal) — violates the strict O(1) requirement.",
"SyntaxError: line 305 'assert val := await cache.get(...)' is invalid Python \u2014 walrus operator cannot appear in an assert statement. The ENTIRE module fails to compile, so nothing runs and no test can execute.",
"Infinite busy-loop: _evict_lfu lines 159-160 \u2014 'while self.min_freq in self.freq_map and self.min_freq < max(...): pass' has an empty body that never updates min_freq, recomputes max() (O(F)) each iteration, and can never terminate.",
"Hidden O(F) scan: min(self.freq_map.keys()) / max(self.freq_map.keys()) appears at 6 call sites (every eviction and removal) \u2014 violates the strict O(1) requirement.",
"Race condition: get() and put() perform lazy-TTL _remove_key() BEFORE acquiring the lock (lines 71-73, 107-108), mutating shared state unlocked while other coroutines read/write.",
"Deadlock risk: background_loop holds self._lock, then calls await self._remove_key() which is itself a lock-acquiring method — asyncio.Lock is NOT reentrant -> deadlock when the evictor runs.",
"Deadlock risk: background_loop holds self._lock, then calls await self._remove_key() which is itself a lock-acquiring method \u2014 asyncio.Lock is NOT reentrant -> deadlock when the evictor runs.",
"No __slots__ on _Node despite using a dataclass (rubric required it for memory efficiency).",
"_remove_key will KeyError on self.freq_map[freq] if a concurrent operation already deleted that bucket."
],
@@ -126,14 +132,14 @@
"verdict": "Critical Bugs",
"best_for": "Promising code-design instincts (cleanest abstractions and best transaction isolation design in the set) but undone by a single fatal one-line bug that stops it running. With the bug fixed it would likely score 80+; as-is, only useful as a structural reference.",
"critical_bugs": [
"FATAL: _put_internal line 305 inserts a NEW key with 'self._freq_map[1].push_front(...)' but never ensures the freq-1 bucket exists — KeyError: 1 on the very first put. The cache cannot store a single key. The _ensure_freq_list(1) helper it should use exists and is used correctly everywhere else (lines 294, 354).",
"Transaction.commit() calls _put_internal for buffered writes, so it hits the same KeyError: 1 — committed transactions crash too.",
"FATAL: _put_internal line 305 inserts a NEW key with 'self._freq_map[1].push_front(...)' but never ensures the freq-1 bucket exists \u2014 KeyError: 1 on the very first put. The cache cannot store a single key. The _ensure_freq_list(1) helper it should use exists and is used correctly everywhere else (lines 294, 354).",
"Transaction.commit() calls _put_internal for buffered writes, so it hits the same KeyError: 1 \u2014 committed transactions crash too.",
"Test suite cannot execute: crashes at the first cache.put() in main(); the well-built test harness (pass/fail counter, 4 real scenarios) validates nothing.",
"No __slots__ on _DLLNode/_CacheNode/_DoublyLinkedList despite the rubric requiring it for memory efficiency.",
"_evict_node uses min(self._freq_map) (line 325) when the min-tier empties — a hidden O(F) scan, violating strict O(1).",
"_evict_node uses min(self._freq_map) (line 325) when the min-tier empties \u2014 a hidden O(F) scan, violating strict O(1).",
"No MVCC/version check on transaction commit (lost-update possible if the global key is modified between tx.get and commit); commit is not exception-safe across the writes-vs-deletes loops."
],
"patch_code": "# FIX 1 (the fatal one-liner): use the helper that already exists.\n# line 305, in _put_internal, new-key branch:\n- self._freq_map[1].push_front(dll_node)\n+ self._ensure_freq_list(1).push_front(dll_node)\n# (This single change makes the cache and transactions functional.)\n\n# FIX 2: replace the O(F) min() scan with an incremental bump:\n# in _evict_node, when the min-tier bucket empties, min_freq is the\n# lowest remaining tier. Since freq only ever increments by 1, the\n# next min is almost always min_freq+1; track it incrementally rather\n# than scanning. Or, since this only happens on full eviction, accept\n# O(F) but only on the empty-cache edge — document it.\n\n# FIX 3: add __slots__ to all internal classes:\nclass _CacheNode:\n __slots__ = ('key','value','ttl_seconds','expiry_time','freq','dll_node')\n ...\n\n# FIX 4: wrap commit applies in try/except so a mid-commit exception\n# does not leave a half-applied global state; consider abort semantics.\n# FIX 5: add node.version; in tx commit, abort if the global node for\n# a key is not the one seen at tx.get() time."
"patch_code": "# FIX 1 (the fatal one-liner): use the helper that already exists.\n# line 305, in _put_internal, new-key branch:\n- self._freq_map[1].push_front(dll_node)\n+ self._ensure_freq_list(1).push_front(dll_node)\n# (This single change makes the cache and transactions functional.)\n\n# FIX 2: replace the O(F) min() scan with an incremental bump:\n# in _evict_node, when the min-tier bucket empties, min_freq is the\n# lowest remaining tier. Since freq only ever increments by 1, the\n# next min is almost always min_freq+1; track it incrementally rather\n# than scanning. Or, since this only happens on full eviction, accept\n# O(F) but only on the empty-cache edge \u2014 document it.\n\n# FIX 3: add __slots__ to all internal classes:\nclass _CacheNode:\n __slots__ = ('key','value','ttl_seconds','expiry_time','freq','dll_node')\n ...\n\n# FIX 4: wrap commit applies in try/except so a mid-commit exception\n# does not leave a half-applied global state; consider abort semantics.\n# FIX 5: add node.version; in tx commit, abort if the global node for\n# a key is not the one seen at tx.get() time."
},
{
"id": "gemma4-31b-gguf",
@@ -144,7 +150,7 @@
"tok_sec": 10.09,
"total_tokens": 4536,
"ttft_sec": 4.39,
"speed_caveat": "All Gemma 4 models ran abnormally slow (GPU offload appeared inactive despite being set), so tok/sec and TTFT are NOT representative of the model itself — likely an LM Studio/GGUF config issue. Treat speed numbers for the Gemma 4 batch as suspect.",
"speed_caveat": "All Gemma 4 models ran abnormally slow (GPU offload appeared inactive despite being set), so tok/sec and TTFT are NOT representative of the model itself \u2014 likely an LM Studio/GGUF config issue. Treat speed numbers for the Gemma 4 batch as suspect.",
"filename": "outputs/gemma4-31b-gguf.py",
"tests_pass": true,
"total_score": 78,
@@ -158,11 +164,11 @@
"verdict": "Minor Logic Flaws",
"best_for": "Clean, correct, runnable code with solid O(1) structure and good concurrency granularity. A reliable pick for everyday caching/async work after a monotonic-clock + __slots__ pass.",
"critical_bugs": [
"Isolation leak: Transaction.get falls back to the PUBLIC cache.get, which calls _update_frequency — so reading a key inside a transaction mutates GLOBAL frequency state before commit, leaking uncommitted access patterns into global eviction order. Spec requires tx reads not to alter global freq.",
"Uses time.time() (system clock) throughout instead of time.monotonic() — NTP adjustments corrupt TTL eviction.",
"No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction — rubric required it for memory efficiency.",
"Isolation leak: Transaction.get falls back to the PUBLIC cache.get, which calls _update_frequency \u2014 so reading a key inside a transaction mutates GLOBAL frequency state before commit, leaking uncommitted access patterns into global eviction order. Spec requires tx reads not to alter global freq.",
"Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments corrupt TTL eviction.",
"No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction \u2014 rubric required it for memory efficiency.",
"_delete_internal deliberately leaves empty frequency buckets in freq_map (documented but a minor memory leak: empty DoublyLinkedList objects accumulate).",
"Background evictor does list(self.cache.keys()) = O(N) snapshot every interval — a linear scan, forbidden by strict O(1).",
"Background evictor does list(self.cache.keys()) = O(N) snapshot every interval \u2014 a linear scan, forbidden by strict O(1).",
"No MVCC/version check on commit (lost-update possible); commit is not exception-safe across the deletes-vs-puts loops.",
"Tests pass but don't probe mid-commit read isolation or eviction-under-real-contention (capacity sized so all keys fit), so the isolation leak above goes undetected."
],
@@ -192,10 +198,10 @@
"best_for": "Runnable and structurally sound, but the LFU eviction has a stale-min_freq capacity-breach path and the transaction API doesn't match the spec (no commit/rollback). Usable for prototypes if you fix eviction and re-skin transactions.",
"critical_bugs": [
"Capacity breach: eviction does self.freq_map[self.min_freq].pop_tail() with NO guard that the bucket exists or is non-empty, and never prunes emptied freq buckets. After manual deletes empty the min-tier, pop_tail returns None silently -> eviction fails -> cache grows PAST capacity. This is the exact stale-min_freq capacity-breach the rubric flags.",
"Non-conformant transaction API: Transaction has NO commit() or rollback() method (both required by spec). Commit happens via a separate cache.apply_transaction_changes(tx._state) — wrong surface; the test only passes because it uses this internal path.",
"Isolation leak: Transaction.get falls back to the public cache.get, which bumps global frequency before commit — uncommitted tx reads alter global eviction order.",
"Non-conformant transaction API: Transaction has NO commit() or rollback() method (both required by spec). Commit happens via a separate cache.apply_transaction_changes(tx._state) \u2014 wrong surface; the test only passes because it uses this internal path.",
"Isolation leak: Transaction.get falls back to the public cache.get, which bumps global frequency before commit \u2014 uncommitted tx reads alter global eviction order.",
"Duplicated eviction logic in apply_transaction_changes re-introduces the stale-min_freq bug at line 211.",
"Uses time.time() (system clock) via _get_now() instead of time.monotonic() — NTP jumps corrupt TTL (though _get_now is a clean single fix point).",
"Uses time.time() (system clock) via _get_now() instead of time.monotonic() \u2014 NTP jumps corrupt TTL (though _get_now is a clean single fix point).",
"No __slots__ on Node/DoublyLinkedList/LFUCache/TransactionState/Transaction.",
"Background sweep does list(self.cache.keys()) = O(N) per interval.",
"Tests pass but use the non-spec commit path and don't probe capacity breach or isolation leak."
@@ -222,15 +228,15 @@
"test_integrity": 4
},
"verdict": "Critical Bugs",
"best_for": "Not usable as-is — the cache cannot store its first key and the background evictor would crash the event loop. The smallest model in the set (12B) and lowest-quality output. Avoid for systems work.",
"best_for": "Not usable as-is \u2014 the cache cannot store its first key and the background evictor would crash the event loop. The smallest model in the set (12B) and lowest-quality output. Avoid for systems work.",
"critical_bugs": [
"FATAL: put() line 112 does 'bucket = self.freq_buckets[self.min_freq]' after setting min_freq=1 but NEVER creates freq_buckets[1] -> KeyError: 1 on the very first put. Cache is unusable.",
"Background evictor is fundamentally broken: start_evictor defines a SYNC 'def evict_loop' and passes it to create_task; inside it calls blocking time.sleep(interval) (freezes the event loop) AND asyncio.run(...) from within a running loop -> RuntimeError. Would crash hard if ever reached.",
"Eviction-by-re-put: expired keys are 'evicted' by re-inserting them with TTL 0 (line 122) instead of deleting them — wrong semantics and triggers immediate re-eviction.",
"Eviction-by-re-put: expired keys are 'evicted' by re-inserting them with TTL 0 (line 122) instead of deleting them \u2014 wrong semantics and triggers immediate re-eviction.",
"Transaction duplicates the entire LFU machinery (local_cache + local_freq + local_min_freq) for snapshot isolation, but _update_local_freq has the same missing-bucket KeyError (line 140).",
"Class name typo 'DoublyLinkedListList' (doubled word).",
"No __slots__; time.time() (not monotonic) throughout.",
"Tests cannot run — crash at first put."
"Tests cannot run \u2014 crash at first put."
],
"patch_code": "# FIX 1 (the fatal KeyError): create the bucket before use.\n# In put(), new-key branch:\n- bucket = self.freq_buckets[self.min_freq]\n+ bucket = self.freq_buckets.setdefault(self.min_freq, DoublyLinkedListList())\n# Same fix in _update_freq and Transaction._update_local_freq (use setdefault).\n\n# FIX 2 (the broken evictor): make it a real async task that deletes:\nasync def _evict_loop(self, interval):\n while True:\n await asyncio.sleep(interval) # async, non-blocking\n now = time.monotonic()\n async with self.global_lock:\n expired = [k for k, n in list(self.cache.items()) if now > n.ttl_expiry]\n for k in expired:\n node = self.cache.pop(k, None)\n if node:\n self.freq_buckets[node.freq].remove(node) # DELETE, not re-put\n\nasync def start_evictor(self, interval=1.0):\n self.evictor_task = asyncio.create_task(self._evict_loop(interval))\n\n# FIX 3: delete expired keys; do NOT re-insert with TTL 0.\n# FIX 4: time.time() -> time.monotonic().\n# FIX 5: add __slots__ to Node / DoublyLinkedListList / ConcurrentLFUCache / Transaction."
},
@@ -243,7 +249,7 @@
"tok_sec": null,
"total_tokens": null,
"ttft_sec": null,
"speed_caveat": "Cloud model (run via opencode, not LM Studio) — tok_sec/tokens/TTFT are N/A (not measured for cloud). Included as a quality baseline against the local models. NOTE: it took 3 attempts to produce any output and ~12 minutes of thinking before succeeding — so it is a QUALITY benchmark, not a speed/usability one.",
"speed_caveat": "Cloud model (run via opencode, not LM Studio) \u2014 tok_sec/tokens/TTFT are N/A (not measured for cloud). Included as a quality baseline against the local models. NOTE: it took 3 attempts to produce any output and ~12 minutes of thinking before succeeding \u2014 so it is a QUALITY benchmark, not a speed/usability one.",
"filename": "deepseekv4flash.py",
"tests_pass": true,
"total_score": 91,
@@ -255,15 +261,109 @@
"test_integrity": 18
},
"verdict": "Production Ready",
"best_for": "Reference-quality baseline (91/100) — the bar the local models are measured against. Only submission with __slots__ + time.monotonic() + delta-based transactional frequency accounting. Sets the ceiling for correctness, though its unreliability (3 attempts, 12-min think time) makes it a poor *local* daily-driver.",
"best_for": "Reference-quality baseline (91/100) \u2014 the bar the local models are measured against. Only submission with __slots__ + time.monotonic() + delta-based transactional frequency accounting. Sets the ceiling for correctness, though its unreliability (3 attempts, 12-min think time) makes it a poor *local* daily-driver.",
"critical_bugs": [
"Two min(self._freq_to_list) linear scans in _evict_one's defensive recovery path (lines 415, 429) — only triggered when min_freq desyncs, not per-operation, but still a non-O(1) path. Could be replaced with incremental tracking.",
"Single coarse lock held across the whole commit-apply loop — not the fine-grained locking the prompt asked for.",
"Two min(self._freq_to_list) linear scans in _evict_one's defensive recovery path (lines 415, 429) \u2014 only triggered when min_freq desyncs, not per-operation, but still a non-O(1) path. Could be replaced with incremental tracking.",
"Single coarse lock held across the whole commit-apply loop \u2014 not the fine-grained locking the prompt asked for.",
"__slots__ present on _Node and _DLL but not extended to Transaction / LFUCache.",
"No explicit lost-update/conflict abort on commit (delta-based freq is applied unconditionally).",
"Tests pass 20/20 but don't include a mid-commit read-isolation probe or adversarial eviction-under-contention stress."
],
"patch_code": "# These are minor refinements on an already production-ready file.\n\n# FIX 1: eliminate the recovery min() scans by keeping min_freq\n# strictly in sync on every insert/bump/remove (it already does on\n# the hot path), so the _evict_one recovery branch is unreachable and\n# can assert instead of scanning:\nassert self._min_freq in self._freq_to_list or not self._freq_to_list\n\n# FIX 2: extend __slots__ to Transaction and LFUCache.\nclass LFUCache(Generic[KT, VT]):\n __slots__ = ('_capacity','_key_to_node','_freq_to_list','_min_freq',\n '_lock','_ttl_index','_evictor_task','_closed')\n\n# FIX 3 (optional): on commit, if a key's global node changed since the\n# tx snapshot, raise LFUCacheError('lost update') instead of overwriting."
},
{
"id": "gemma4-26b-a4b-8bit-mlx",
"timestamp": "2026-07-28T16:25:00Z",
"model_name": "Gemma 4 26B-A4B",
"quant": "8-bit",
"param_size": "26B-A4B (MoE)",
"format": "mlx",
"tok_sec": 58.3,
"total_tokens": 7390,
"ttft_sec": 0.93,
"filename": "outputs/gemma4-26b-a4b-8bit-mlx.py",
"tests_pass": true,
"total_score": 82,
"breakdown": {
"complexity": 17,
"concurrency": 16,
"isolation": 18,
"memory_edge_cases": 15,
"test_integrity": 16
},
"verdict": "Minor Logic Flaws",
"best_for": "Tied top local scorer (82). The only local model to use delta-based transactional frequency accounting (freq bumps deferred to commit), matching the cloud baseline's isolation approach. Reliable for async/ACID-pattern work after a __slots__ + min_freq-edge pass.",
"critical_bugs": [
"Stale min_freq on manual delete: _remove_node_from_structures empties the min-freq bucket but does `pass` instead of recomputing min_freq (lines 199-202, documented as 'a simplification'). Correct only because eviction has a min_freq-in-freq_map guard + arbitrary-key fallback (lines 237-244) \u2014 latent fragility under concurrent deletes.",
"No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction despite Generic dataclasses (rubric required it for memory efficiency).",
"Background evictor does list(self.cache_data.keys()) = O(N) snapshot every interval \u2014 a linear scan, forbidden by strict O(1).",
"No MVCC/version check on commit (lost-update possible if the global key changes between tx.get and commit); commit applies deletes->reads->puts without try/except, so a mid-commit exception leaves partial state."
],
"patch_code": "# FIX 1 (stale min_freq): recompute or invalidate when the min bucket empties.\n# In _remove_node_from_structures, replace the `pass`:\nif dll.size == 0:\n del self.freq_map[node.freq]\n if self.min_freq == node.freq:\n # bump to next existing tier (frequencies are contiguous under normal use)\n self.min_freq = self.min_freq + 1 if (self.min_freq + 1) in self.freq_map else min(self.freq_map, default=1)\n\n# FIX 2: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 3: background evictor \u2014 maintain a _ttl_keys set and iterate IT in\n# batches instead of list(self.cache_data.keys()) to stay O(batch).\n# FIX 4: wrap commit's three loops in try/except with rollback semantics on failure."
},
{
"id": "qwen3.6-27b-8bit-mlx",
"timestamp": "2026-07-28T16:30:00Z",
"model_name": "Qwen 3.6 27B",
"quant": "8-bit",
"param_size": "27B dense",
"format": "mlx",
"tok_sec": 12.29,
"total_tokens": 12790,
"ttft_sec": 2.9,
"speed_caveat": "This model 'thought' for 12m48s before producing output and ran at 12.29 tok/sec \u2014 anomalously slow for an 8-bit MLX on M3 Max. Likely an inference/quant issue worth investigating; the slow generation is NOT representative of normal 27B-8bit throughput.",
"filename": "outputs/qwen3.6-27b-8bit-mlx.py",
"tests_pass": true,
"total_score": 78,
"breakdown": {
"complexity": 17,
"concurrency": 16,
"isolation": 14,
"memory_edge_cases": 15,
"test_integrity": 16
},
"verdict": "Minor Logic Flaws",
"best_for": "Clean, correct, runnable \u2014 same tier as Gemma 4 31B (78). Good O(1) structure and concurrency granularity. Reliable for everyday async/caching work after a monotonic-clock + __slots__ + isolation pass. Caveat: was anomalously slow to generate.",
"critical_bugs": [
"Isolation leak: Transaction.get falls back to the PUBLIC cache.get (line 78), which calls _update_freq \u2014 so reading a key inside a transaction mutates GLOBAL frequency state before commit, leaking uncommitted access patterns into global eviction order.",
"Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP adjustments corrupt TTL eviction.",
"No __slots__ on Node/DoublyLinkedList/LFUCache/Transaction \u2014 rubric required it for memory efficiency.",
"Background _evict_loop materializes list(self.nodes.keys()) (O(N)) before checking only batch_size keys \u2014 the break caps work but not the snapshot cost, a hidden O(N) per sweep.",
"Commit re-implements put+evict inline (lines 102-118) duplicating the public path = duplicated bug surface; not wrapped in try/except so a mid-commit exception leaves partial state. No MVCC version check (lost-update possible)."
],
"patch_code": "# FIX 1 (isolation leak): add a read-only global lookup (no freq bump)\n# and use it in tx.get instead of the public cache.get:\nasync def _read_raw(self, key):\n async with self.lock:\n node = self.nodes.get(key)\n if node is None: return None\n if time.monotonic() > node.expires_at:\n self._remove_node(key); return None\n return node.value\n# then: return await self._cache._read_raw(key)\n# FIX 2: time.time() -> time.monotonic() everywhere.\n# FIX 3: add __slots__ to Node, DoublyLinkedList, LFUCache, Transaction.\n# FIX 4: maintain a _ttl_keys set; iterate IT (batched) in the bg loop\n# instead of list(self.nodes.keys()).\n# FIX 5: factor commit's put/evict to reuse the internal helpers; wrap\n# the commit loop in try/except with rollback-on-failure."
},
{
"id": "qwen3-coder-30b-6bit-mlx",
"timestamp": "2026-07-28T16:35:00Z",
"model_name": "Qwen3 Coder 30B",
"quant": "6-bit",
"param_size": "30B",
"format": "mlx",
"tok_sec": 72.7,
"total_tokens": 2779,
"ttft_sec": 0.9,
"filename": "outputs/qwen3-coder-30b-6bit-mlx.py",
"tests_pass": false,
"total_score": 50,
"breakdown": {
"complexity": 15,
"concurrency": 9,
"isolation": 11,
"memory_edge_cases": 11,
"test_integrity": 4
},
"verdict": "Critical Bugs",
"best_for": "Not usable as-is \u2014 transactions crash immediately due to an async/sync lock mismatch. The coder-specialist produced terse, fast output (2779 tok, 72.7 t/s) with competent freq-bucket structure, but fumbled the async primitive. Fix the one lock bug and it would likely score 70+.",
"critical_bugs": [
"FATAL: _transaction_lock is an asyncio.Lock() (line 101) but begin_transaction() is a SYNC def that uses synchronous `with self._transaction_lock:` (line 199). asyncio.Lock does not support the sync context-manager protocol -> TypeError on the first transaction, crashing the entire test suite.",
"Test suite cannot run: crashes at cache.begin_transaction() in main(); the transaction and concurrency assertions never execute.",
"Unused `import threading` and `import weakref` \u2014 vestigial confusion between threading and asyncio primitives.",
"Uses time.time() (system clock) throughout instead of time.monotonic() \u2014 NTP jumps corrupt TTL eviction.",
"No __slots__ on CacheNode/FrequencyBucket/InMemoryLFUCache/Transaction \u2014 rubric required it.",
"FrequencyBucket stores nodes in BOTH a DLL and a parallel `nodes` dict \u2014 redundant memory per bucket."
],
"patch_code": "# FIX 1 (the fatal crash): make begin_transaction async and use async with.\nasync def begin_transaction(self) -> 'Transaction':\n async with self._transaction_lock:\n self._transaction_counter += 1\n tx = Transaction(self)\n self._transactions[self._transaction_counter] = tx\n return tx\n# (and update callers: `tx = await cache.begin_transaction()`)\n# Alternative if sync creation is required: use threading.Lock for the\n# counter, but that is wrong in an asyncio codebase \u2014 go async.\n\n# FIX 2: remove unused `import threading` and `import weakref`.\n# FIX 3: time.time() -> time.monotonic() everywhere.\n# FIX 4: add __slots__ to all classes.\n# FIX 5: drop the redundant FrequencyBucket.nodes dict; the DLL already\n# tracks membership, so the dict is duplicate storage."
}
]
}
}
generate_dashboard.py
+144 -20
View File
@@ -4,7 +4,7 @@ Generator: builds dashboard.html + pages/<slug>..html from data/benchmark_histor
Re-run after each grading batch to regenerate everything.
Cyberpunk-terminal aesthetic. Pure stdlib + Chart.js via CDN.
"""
import json, html, os, sys
import json, html, os, sys, re as _re
HERE = os.path.dirname(os.path.abspath(__file__))
DATA = os.path.join(HERE, "data", "benchmark_history.json")
@@ -98,11 +98,28 @@ h1{font-size:1.5rem; margin:.3em 0 0; letter-spacing:.02em; text-shadow:0 0 16px
.stat::after{content:"";position:absolute;left:0;top:0;bottom:0;width:3px;background:var(--cyan);box-shadow:0 0 12px var(--cyan)}
.stat .lbl{color:var(--dim);font-size:.68rem;letter-spacing:.18em;text-transform:uppercase}
.stat .num{font-size:1.6rem;margin-top:4px;color:var(--ink)}
/* hero top row: squished stat tiles beside the radar */
.toprow{display:grid;grid-template-columns:1.35fr 1fr;gap:18px;align-items:stretch;margin:18px 0 22px}
.stats-squish{display:grid;grid-template-columns:repeat(2,1fr);gap:12px;align-content:start}
.stats-squish .stat{padding:16px 18px}
.stats-squish .stat .num{font-size:1.9rem}
.radar-panel{display:flex;flex-direction:column}
.radar-panel .chart-box{flex:1;min-height:220px}
/* compact top-performers list */
.toplist{display:flex;flex-direction:column;gap:6px}
a.tr{display:grid;grid-template-columns:28px 1fr auto auto auto;align-items:center;gap:10px;padding:8px 12px;background:rgba(0,0,0,0.2);border:1px solid rgba(255,255,255,0.05);border-radius:5px;transition:all .16s;cursor:pointer;text-decoration:none;font-size:.84rem}
a.tr:hover{border-color:rgba(0,255,200,0.4);background:rgba(0,255,200,0.05)}
.tr-r{color:var(--dim)}
.tr-n{color:var(--ink)}
.tr-q{color:var(--dim);font-size:.74rem}
.tr-s{color:var(--mag);font-size:.78rem}
.tr-sc{font-size:1rem;font-weight:600;text-align:right;min-width:30px}
@media(max-width:900px){.toprow{grid-template-columns:1fr}.grid2{grid-template-columns:1fr}.stats,.stats-squish{grid-template-columns:repeat(2,1fr)}}
.grid2{display:grid;grid-template-columns:1.1fr .9fr;gap:18px;margin-bottom:26px}
.panel{background:var(--panel);border:1px solid rgba(255,255,255,0.07);border-radius:8px;padding:18px}
.panel h2{font-size:.95rem;letter-spacing:.12em;text-transform:uppercase;color:var(--cyan);margin:0 0 14px;text-shadow:0 0 10px rgba(0,255,200,0.35)}
.chart-box{position:relative;height:340px}
@media(max-width:900px){.grid2{grid-template-columns:1fr}.stats{grid-template-columns:repeat(2,1fr)}}
@media(max-width:900px){.grid2{grid-template-columns:1fr}}
/* leaderboard */
table{width:100%;border-collapse:collapse;font-size:.86rem}
thead th{text-align:left;color:var(--dim);font-size:.66rem;letter-spacing:.16em;text-transform:uppercase;border-bottom:1px solid rgba(0,255,200,0.2);padding:8px 10px}
@@ -122,6 +139,23 @@ tbody tr:hover{background:rgba(0,255,200,0.05);box-shadow:inset 0 0 0 1px rgba(0
tr:hover .score-bar > i{box-shadow:0 0 12px currentColor}
.caveat{color:var(--amber);font-size:.72rem}
.cloud-tag{color:var(--blue);font-size:.7rem;border:1px solid rgba(91,139,255,.4);padding:1px 6px;border-radius:3px;margin-left:6px}
.quant-cell{color:var(--ink);font-size:.82rem;white-space:nowrap}
.fmt-chip{display:inline-block;font-size:.66rem;letter-spacing:.08em;padding:2px 7px;border-radius:3px;border:1px solid currentColor;font-family:'Fira Code',monospace}
.fmt-mlx{color:var(--cyan);background:rgba(0,255,200,0.08)}
.fmt-gguf{color:var(--mag);background:rgba(255,43,214,0.08)}
.fmt-cloud{color:var(--blue);background:rgba(91,139,255,0.08)}
/* format/quant showdown cards */
.fcard{background:var(--panel2);border:1px solid rgba(255,255,255,0.06);border-radius:6px;padding:12px 14px;margin-bottom:10px}
.fcard-h{display:flex;justify-content:space-between;align-items:baseline;gap:10px;flex-wrap:wrap;margin-bottom:9px}
.fcard-n{color:var(--ink);font-weight:600;font-family:'Fira Code',monospace;font-size:.92rem}
.fcard-meta{color:var(--dim);font-size:.72rem}
.fcard-v{display:flex;flex-wrap:wrap;gap:8px}
a.fv{display:grid;grid-template-columns:auto auto auto auto;align-items:center;gap:10px;padding:7px 11px;background:rgba(0,0,0,0.25);border:1px solid rgba(0,255,200,0.15);border-radius:5px;transition:all .18s;cursor:pointer;text-decoration:none}
a.fv:hover{border-color:var(--cyan);box-shadow:0 0 12px rgba(0,255,200,0.3);background:rgba(0,255,200,0.06)}
.fv-q{color:var(--ink);font-size:.82rem;min-width:64px}
.fv-f{font-size:.7rem;letter-spacing:.08em;font-family:'Fira Code',monospace}
.fv-s{color:var(--dim);font-size:.78rem}
.fv-sc{font-size:1rem;font-weight:600;min-width:28px;text-align:right}
footer{color:var(--dim);font-size:.74rem;margin-top:40px;border-top:1px solid rgba(255,255,255,0.06);padding-top:14px;text-align:center}
@media (prefers-reduced-motion: reduce){*{animation:none!important;transition:none!important}}
"""
@@ -165,21 +199,38 @@ def render_dashboard(data):
caveat = ""
if m.get("speed_caveat"):
caveat = '<div class="caveat">⚠ speed suspect</div>'
cloud = '<span class="cloud-tag">CLOUD</span>' if m.get("format") == "cloud" else ""
fmt = (m.get("format") or "").lower()
if fmt == "cloud":
fmt_chip = '<span class="fmt-chip fmt-cloud">CLOUD</span>'
elif fmt == "gguf":
fmt_chip = '<span class="fmt-chip fmt-gguf">GGUF</span>'
elif fmt == "mlx":
fmt_chip = '<span class="fmt-chip fmt-mlx">MLX</span>'
else:
fmt_chip = f'<span class="fmt-chip">{esc(m.get("format") or "—")}</span>'
bar_color = col
rows.append(f"""<tr>
<td class="rank {'top' if i<=3 else ''}">#{i}</td>
<td><div class="model-name">{esc(m['model_name'])}{cloud}</div><div class="quant">{esc(m['quant'])}</div>{caveat}</td>
<td><div class="model-name">{esc(m['model_name'])}</div>{caveat}</td>
<td><span class="quant-cell mono">{esc(m['quant'])}</span></td>
<td>{fmt_chip}</td>
<td class="mono">{speed_str(m)} <span style="color:var(--dim);font-size:.72rem">t/s</span></td>
<td><div class="bar-cell"><span class="mono" style="width:34px;color:{col}">{m['total_score']}</span>
<span class="score-bar"><i style="width:{m['total_score']}%;background:{bar_color};color:{bar_color}"></i></span></div></td>
<td><span class="badge verdict-chip" style="color:{col}">{chip}</span></td>
<td style="color:var(--dim);font-size:.8rem">{esc(m['best_for'])[:70]}…</td>
<td style="color:var(--dim);font-size:.8rem">{esc(m['best_for'])[:60]}…</td>
<td><a class="btn" href="pages/{esc(m['id'])}.html">DECODE ▸</a></td>
</tr>""")
# JSON for charts
bar_labels = json.dumps([m["model_name"].split("(")[0].strip()[:18] for m in local_sorted])
def short_label(m):
# family + quant so duplicates (same model, different quant) are distinguishable
fam = m["model_name"].split("(")[0].strip()
fam = _re.sub(r"(?i)\s+(uncensored|heretic|aggressive|hauhaucs).*$", "", fam)
q = (m.get("quant") or "").strip()
name = f"{fam} · {q}" if q else fam
return name[:28]
bar_labels = json.dumps([short_label(m) for m in local_sorted])
bar_speed = json.dumps([m["tok_sec"] for m in local_sorted])
bar_score = json.dumps([m["total_score"] for m in local_sorted])
@@ -195,14 +246,87 @@ def render_dashboard(data):
})
radar_json = json.dumps(radar_sets)
stats = f"""
stats_tiles = f"""
<div class="stat"><div class="lbl">Models Tested</div><div class="num mono">{n}</div></div>
<div class="stat"><div class="lbl">Top Score</div><div class="num mono" style="color:var(--lime)">{top['total_score'] if top else '—'}</div></div>
<div class="stat"><div class="lbl">Average</div><div class="num mono" style="color:var(--amber)">{avg:.1f}</div></div>
<div class="stat"><div class="lbl">Prod-Ready</div><div class="num mono" style="color:var(--cyan)">{prod}/{n}</div></div>
"""
stats = f'<div class="stats">{stats_tiles}</div>'
top_name = esc(top["model_name"]) if top else "—"
# compact top-5 list for the right-of-radar panel
trows = []
for i, m in enumerate(models[:5], 1):
col, _ = verdict_meta(m["verdict"])
speed = f"{m['tok_sec']:.1f}" if m.get("tok_sec") is not None else "—"
trows.append(
f'<a class="tr" href="pages/{esc(m["id"])}.html">'
f'<span class="tr-r mono">#{i}</span>'
f'<span class="tr-n">{esc(m["model_name"][:20])}</span>'
f'<span class="tr-q mono">{esc(m["quant"])}</span>'
f'<span class="tr-s mono">{speed}</span>'
f'<span class="tr-sc mono" style="color:{col}">{m["total_score"]}</span></a>'
)
toplist = ''.join(trows)
# ---- Format / Quant showdown: group local models by base family ----
def family_of(m):
# strip parentheticals, uncensored/merge tags, quant words, format, params
name = m["model_name"].split("(")[0].strip()
name = _re.sub(r"(?i)\b(uncensored|heretic|aggressive|hauhaucs|coder|composer|fable5|v\d+\.\d+|merge)\b", "", name)
name = _re.sub(r"\b\d+(\.\d+)?[bB](-[aA]\d+[bB])?\b", "", name) # sizes: 35B, 26B-A4B
name = _re.sub(r"\s+", " ", name).strip(" -")
# collapse known families
for fam in ["Qwen 3.6", "Qwen3", "Gemma 4", "KAT-Coder", "DeepSeek"]:
if _re.sub(r"\s+", "", name).lower().startswith(_re.sub(r"\s+", "", fam).lower()):
return fam
return name or m["model_name"]
families = {}
for m in models:
if m.get("format") == "cloud":
continue
f = family_of(m)
families.setdefault(f, []).append(m)
# only show families with >=2 variants (the interesting comparisons)
multi = {f: ms for f, ms in families.items() if len(ms) >= 2}
if multi:
fcards = []
for fam, ms in sorted(multi.items(), key=lambda kv: -max(x["total_score"] for x in kv[1])):
ms_sorted = sorted(ms, key=lambda x: -x["total_score"])
best = ms_sorted[0]
spread = max(x["total_score"] for x in ms) - min(x["total_score"] for x in ms)
variants = []
for x in ms_sorted:
fmt = (x.get("format") or "").upper()
fcol = "var(--cyan)" if x.get("format") == "mlx" else ("var(--mag)" if x.get("format") == "gguf" else "var(--dim)")
col, _ = verdict_meta(x["verdict"])
speed = f"{x['tok_sec']:.0f} t/s" if x.get("tok_sec") is not None else "—"
variants.append(
f'<a class="fv" href="pages/{esc(x["id"])}.html">'
f'<span class="fv-q mono">{esc(x["quant"])}</span>'
f'<span class="fv-f" style="color:{fcol}">{fmt}</span>'
f'<span class="fv-s mono">{speed}</span>'
f'<span class="fv-sc mono" style="color:{col}">{x["total_score"]}</span>'
f'</a>'
)
fcards.append(f"""
<div class="fcard">
<div class="fcard-h"><span class="fcard-n">{esc(fam)}</span>
<span class="fcard-meta">{len(ms)} variants · score spread <b class="mono">{spread}</b> · best <b class="mono" style="color:var(--lime)">{best["total_score"]}</b> ({esc(best["quant"])})</span></div>
<div class="fcard-v">{''.join(variants)}</div>
</div>""")
family_panel = f"""
<div class="panel" style="margin-bottom:26px">
<h2>▮ FORMAT &amp; QUANT SHOWDOWN — same family, different quants/formats</h2>
<div style="color:var(--dim);font-size:.76rem;margin-bottom:12px">Families with 2+ variants. Click a row for the full audit. Compare how quant depth and MLX-vs-GGUF change the score.</div>
{''.join(fcards)}
</div>"""
else:
family_panel = ""
body = f"""
{head_html("LLM Benchmark Suite")}
<header class="hud-bar">
@@ -211,27 +335,27 @@ def render_dashboard(data):
<div class="subtitle">{n} models graded on a strict 5-pillar / 100-pt rubric · O(1) LFU + ACID transactions · M3 Max · LM Studio
&nbsp;·&nbsp; <span style="color:var(--cyan)">TOP: {top_name}</span></div>
</header>
{stats}
<div class="grid2">
<div class="panel">
<h2>▮ Score vs Throughput (tok/sec)</h2>
<div class="chart-box"><canvas id="bar"></canvas></div>
<div style="color:var(--dim);font-size:.72rem;margin-top:8px">Local models only — cloud baseline (DeepSeek) excluded from speed axis. Gemma 4 bars flagged ⚠ (GPU-offload suspect).</div>
</div>
<div class="panel">
<h2>▮ 5-Pillar Radar — Top 3</h2>
<div class="chart-box"><canvas id="radar"></canvas></div>
<div style="color:var(--dim);font-size:.72rem;margin-top:8px">Each pillar scored 0–20. Outer = stronger.</div>
<div class="toprow">
<div class="stats stats-squish">{stats_tiles}</div>
<div class="panel radar-panel">
<h2>▮ 5-PILLAR RADAR — TOP 3</h2>
<div class="chart-box" style="height:230px"><canvas id="radar"></canvas></div>
</div>
</div>
<div class="panel" style="margin-bottom:26px">
<h2>▮ SCORE vs THROUGHPUT (tok/sec)</h2>
<div class="chart-box" style="height:440px"><canvas id="bar"></canvas></div>
<div style="color:var(--dim);font-size:.72rem;margin-top:8px">Local models only — cloud baseline (DeepSeek) excluded from the speed axis. Bars flagged ⚠ have suspected GPU-offload / inference issues (not representative of the model).</div>
</div>
<div class="panel" style="margin-bottom:26px">
<h2>▮ LEADERBOARD</h2>
<div style="overflow-x:auto">
<table>
<thead><tr><th>#</th><th>Model</th><th>Speed</th><th>Score</th><th>Verdict</th><th>Best For</th><th></th></tr></thead>
<thead><tr><th>#</th><th>Model</th><th>Quant</th><th>Format</th><th>Speed</th><th>Score</th><th>Verdict</th><th>Best For</th><th></th></tr></thead>
<tbody>{''.join(rows)}</tbody>
</table></div>
</div>
{family_panel}
<footer>Generated from <span class="mono">data/benchmark_history.json</span> · re-run <span class="mono">generate_dashboard.py</span> to refresh · cyberpunk-terminal UI</footer>
<script>
const NEON={{cyan:'#00ffc8',mag:'#ff2bd6',lime:'#b6ff3a',amber:'#ffb000',red:'#ff3b5c',blue:'#5b8cff'}};
@@ -245,7 +369,7 @@ new Chart(document.getElementById('bar'),{{
options:{{maintainAspectRatio:false,responsive:true,
plugins:{{legend:{{labels:{{color:'#d7e0e6',font:{{family:'Fira Code'}}}}}}}},
scales:{{
x:{{ticks:{{color:'#7a8590',font:{{family:'Fira Code',size:10}}}},grid:{{color:'rgba(255,255,255,0.05)'}}}},
x:{{ticks:{{color:'#7a8590',font:{{family:'Fira Code',size:10}},maxRotation:50,minRotation:35,autoSkip:false}},grid:{{color:'rgba(255,255,255,0.05)'}}}},
y:{{position:'left',max:100,title:{{display:true,text:'Score',color:'#00ffc8'}},ticks:{{color:'#7a8590'}},grid:{{color:'rgba(255,255,255,0.05)'}}}},
y1:{{position:'right',title:{{display:true,text:'tok/sec',color:'#ff2bd6'}},grid:{{drawOnChartArea:false}},ticks:{{color:'#7a8590'}}}}
}}}}
outputs/gemma4-26b-a4b-8bit-mlx.py
+406
View File
@@ -0,0 +1,406 @@
import asyncio
import time
from dataclasses import dataclass, field
from typing import Any, Dict, Generic, Optional, Set, TypeVar, Union
# Type variables for Generics
K = TypeVar("K")
V = TypeVar("V")
@dataclass
class Node(Generic[K, V]):
"""A node in the doubly linked list representing a cache entry."""
key: K
value: V
freq: int = 1
expiry: float = float('inf')
prev: Optional['Node[K, V]'] = None
next: Optional['Node[K, V]'] = None
class DoublyLinkedList(Generic[K, V]):
"""A standard Doubly Linked List to allow O(1) removal and insertion."""
def __init__(self):
self.head: Optional[Node[K, V]] = None
self.tail: Optional[Node[K, V]] = None
self.size: int = 0
def append(self, node: Node[K, V]):
"""Adds a node to the end of the list."""
if not self.head:
self.head = node
self.tail = node
node.prev = None
node.next = None
else:
node.prev = self.tail
node.next = None
if self.tail:
self.tail.next = node
self.tail = node
self.size += 1
def remove(self, node: Node[K, V]):
"""Removes a specific node from the list in O(1)."""
if node.prev:
node.prev.next = node.next
else:
self.head = node.next
if node.next:
node.next.prev = node.prev
else:
self.tail = node.prev
node.next = None
node.prev = None
self.size -= 1
def pop_tail(self) -> Optional[Node[K, V]]:
"""Removes and returns the last node in O(1)."""
if not self.tail:
return None
node = self.tail
self.remove(node)
return node
class Transaction(Generic[K, V]):
"""
Implements ACID-like sub-sessions.
Provides 'Read Your Own Writes' and isolation from the global cache.
"""
def __init__(self, cache: 'LFUCache[K, V]'):
self._cache = cache
self._pending_puts: Dict[K, tuple[V, float]] = {}
self._pending_deletes: Set[K] = set()
# Track keys read from the main cache to update frequency on commit
self._read_cache_keys: Set[K] = set()
self._is_active = True
async def get(self, key: K) -> Optional[V]:
if not self._is_active: raise RuntimeError("Transaction closed")
# 1. Check local deletes (Tombstone)
if key in self._pending_deletes:
return None
# 2. Check local writes (Read Your Own Writes)
if key in self._pending_puts:
val, _ = self._pending_puts[key]
return val
# 3. Check global cache (without affecting LFU frequency until commit)
async with self._cache._lock:
node = self._cache.cache_data.get(key)
if node:
# Lazy TTL check even during transaction get
if time.monotonic() >= node.expiry:
self._cache._remove_node_from_structures(node)
return None
self._read_cache_keys.add(key)
return node.value
return None
async def put(self, key: K, value: V, ttl_seconds: float = float('inf')):
if not self._is_active: raise RuntimeError("Transaction closed")
expiry = time.monotonic() + ttl_seconds
self._pending_puts[key] = (value, expiry)
if key in self._pending_deletes:
self._pending_deletes.remove(key)
async def delete(self, key: K):
if not self._is_active: raise RuntimeError("Transaction closed")
if key in self._pending_puts:
del self._pending_puts[key]
self._pending_deletes.add(key)
async def commit(self):
if not self._is_active: raise RuntimeError("Transaction closed")
async with self._cache._lock:
# 1. Process Deletes
for key in self._pending_deletes:
node = self._cache.cache_data.get(key)
if node:
self._cache._remove_node_from_structures(node)
# 2. Process Reads (Update frequencies for keys read from global cache)
for key in self._read_cache_keys:
node = self._cache.cache_data.get(key)
if node: # Ensure it wasn't deleted by a pending delete in this TX
self._cache._increment_frequency(node)
# 3. Process Puts
for key, (val, expiry) in self._pending_puts.items():
# Internal un-locked put for use within the lock context of commit
self._cache._internal_put(key, val, expiry)
self._is_active = False
self._pending_puts.clear()
async def rollback(self):
if not self._is_active: raise RuntimeError("Transaction closed")
self._pending_puts.clear()
self._pending_deletes.clear()
self._read_cache_keys.clear()
self._is_active = False
class LFUCache(Generic[K, V]):
"""
In-Memory Concurrent LFU Cache.
Time Complexity: O(1) for get and put.
"""
def __init__(self, capacity: int):
if capacity <= 0: raise ValueError("Capacity must be > 0")
self.capacity = capacity
self.cache_data: Dict[K, Node[K, V]] = {}
self.freq_map: Dict[int, DoublyLinkedList[K, V]] = {}
self.min_freq: int = 0
self._lock = asyncio.Lock()
self._evictor_task: Optional[asyncio.Task] = None
async def start_evictor(self, interval: float = 1.0):
"""Starts the background async eviction loop."""
if self._evictor_task is None:
self._evictor_task = asyncio.create_task(self._background_eviction_loop(interval))
async def stop_evictor(self):
"""Stops the background async eviction loop."""
if self._evictor_task:
self._evictor_task.cancel()
try:
await self._evictor_task
except asyncio.CancelledError:
pass
self._evictor_task = None
async def _background_eviction_loop(self, interval: float):
while True:
await asyncio.sleep(interval)
# We do not hold the lock for the entire loop to prevent blocking.
# Instead, we take a snapshot of keys and process in batches.
async with self._lock:
keys_snapshot = list(self.cache_data.keys())
# Process in small batches to yield control
batch_size = 50
for i in range(0, len(keys_snapshot), batch_size):
batch = keys_snapshot[i : i + batch_size]
now = time.monotonic()
async with self._lock:
for key in batch:
node = self.cache_data.get(key)
if node and now >= node.expiry:
self._remove_node_from_structures(node)
def _remove_node_from_structures(self, node: Node[K, V]):
"""Internal: Removes node from freq_map and cache_data. O(1)."""
# Remove from DLL
dll = self.freq_map[node.freq]
dll.remove(node)
if dll.size == 0:
del self.freq_map[node.freq]
if self.min_freq == node.freq:
# This is a simplification; real min_freq update happens in _increment_frequency.
# If current min freq list is empty, we'll find the new min in _increment_frequency or put.
pass
# Remove from dict
if node.key in self.cache_data:
del self.cache_data[node.key]
def _increment_frequency(self, node: Node[K, V]):
"""Internal: Increases frequency of a node. O(1)."""
old_freq = node.freq
new_freq = old_freq + 1
node.freq = new_freq
# Remove from old DLL
old_dll = self.freq_map[old_freq]
old_dll.remove(node)
if old_dll.size == 0:
del self.freq_map[old_freq]
if self.min_freq == old_freq:
self.min_freq = new_freq
# Add to new DLL
if new_freq not in self.freq_map:
self.freq_map[new_freq] = DoublyLinkedList()
self.freq_map[new_freq].append(node)
def _internal_put(self, key: K, value: V, expiry: float):
"""Internal: The core LFU logic. Must be called within a lock."""
if key in self.cache_data:
node = self.cache_data[key]
node.value = value
node.expiry = expiry
self._increment_frequency(node)
else:
if len(self.cache_data) >= self.capacity:
# Evict LFU (min_freq list tail)
if self.min_freq in self.freq_map and self.freq_map[self.min_freq].size > 0:
victim = self.freq_map[self.min_freq].pop_tail()
if victim:
del self.cache_data[victim.key]
else:
# Fallback (should not happen with correct logic)
k_evict = next(iter(self.cache_data))
del self.cache_data[k_evict]
new_node = Node(key, value, freq=1, expiry=expiry)
self.cache_data[key] = new_node
if 1 not in self.freq_map:
self.freq_map[1] = DoublyLinkedList()
self.freq_map[1].append(new_node)
self.min_freq = 1
async def get(self, key: K) -> Optional[V]:
"""O(1) retrieval with lazy TTL eviction."""
async with self._lock:
node = self.cache_data.get(key)
if not node:
return None
# Lazy TTL Eviction
if time.monotonic() >= node.expiry:
self._remove_node_from_structures(node)
return None
self._increment_frequency(node)
return node.value
async def put(self, key: K, value: V, ttl_seconds: float = float('inf')):
"""O(1) insertion with lazy TTL eviction."""
expiry = time.monotonic() + ttl_seconds
async with self._lock:
self._internal_put(key, value, expiry)
async def delete(self, key: K):
"""O(1) deletion."""
async with self._lock:
node = self.cache_data.get(key)
if node:
self._remove_node_from_structures(node)
def begin_transaction(self) -> Transaction[K, V]:
return Transaction(self)
# ==========================================
# UNIT TESTS
# ==========================================
async def test_lfu_eviction():
print("Testing O(1) LFU Eviction Logic...")
cache = LFUCache[str, int](capacity=3)
await cache.put("a", 1) # freq 1
await cache.put("b", 2) # freq 1
await cache.put("c", 3) # freq 1
# Increase frequency of a and b
await cache.get("a") # freq 2
await cache.get("b") # freq 2
# c is still freq 1
await cache.put("d", 4) # Should evict 'c'
assert await cache.get("a") == 1
assert await cache.get("b") == 2
assert await cache.get("c") is None
assert await cache.get("d") == 4
print("✅ LFU Eviction Passed.")
async def test_ttl_eviction():
print("Testing Dual-Layer TTL Eviction...")
cache = LFUCache[str, int](capacity=10)
await cache.start_evictor(interval=0.1)
# Test Lazy Eviction
await cache.put("lazy", 100, ttl_seconds=0.2)
await asyncio.sleep(0.3)
assert await cache.get("lazy") is None, "Lazy eviction failed"
# Test Background Eviction
await cache.put("bg", 200, ttl_seconds=0.1)
assert await cache.get("bg") is not None, "Value should still be there for a millisecond"
await asyncio.sleep(0.4)
# Note: Background loop might not have run yet, but we check if it's gone
# Since background is a separate task, it should have cleared 'bg' by now.
assert await cache.get("bg") is None, "Background eviction failed"
await cache.stop_evictor()
print("✅ TTL Eviction Passed.")
async def test_transactions():
print("Testing Atomic Transactions...")
cache = LFUCache[str, int](capacity=5)
# Test Rollback
tx = cache.begin_transaction()
await tx.put("tx1", 10)
assert await cache.get("tx1") is None, "Uncommitted write visible!"
assert await tx.get("tx1") == 10, "Read Your Own Writes failed"
await tx.rollback()
assert await cache.get("tx1") is None
# Test Commit Visibility
tx = cache.begin_transaction()
await tx.put("tx2", 20)
await tx.commit()
assert await cache.get("tx2") == 20, "Commit visibility failed"
# Test Isolation/Tombstones
await cache.put("exists", 50)
tx = cache.begin_transaction()
await tx.delete("exists")
assert await tx.get("exists") is None, "Transaction delete failed"
assert await cache.get("exists") == 50, "Uncommitted delete visible!"
await tx.commit()
assert await cache.get("exists") is None, "Commit delete failed"
print("✅ Transactions Passed.")
async def test_stress():
print("Running Concurrency Stress Test (50 tasks)...")
cache = LFUCache[int, int](capacity=20)
await cache.start_evictor()
async def worker(worker_id: int):
for i in range(100):
key = (worker_id * 100) + (i % 30) # Overlapping keys to induce contention
op = i % 3
if op == 0:
await cache.put(key, i)
elif op == 1:
await cache.get(key)
else:
# Transactional stress
tx = cache.begin_transaction()
await tx.put(key, i)
if i % 5 == 0:
await tx.rollback()
else:
await tx.commit()
if i % 10 == 0:
await asyncio.sleep(0.01)
tasks = [worker(i) for i in range(50)]
await asyncio.gather(*tasks)
await cache.stop_evictor()
print("✅ Stress Test Completed (No crashes).")
async def main():
start_time = time.perf_counter()
try:
await test_lfu_eviction()
print("-" * 30)
await test_ttl_eviction()
print("-" * 30)
await test_transactions()
print("-" * 30)
await test_stress()
print("-" * 30)
print(f"All tests finished successfully in {time.perf_counter() - start_time:.2f}s")
except Exception as e:
print(f"❌ Tests failed with error: {e}")
import traceback
traceback.print_exc()
if __name__ == "__main__":
asyncio.run(main())
outputs/qwen3-coder-30b-6bit-mlx.py
+335
View File
@@ -0,0 +1,335 @@
import asyncio
import time
import weakref
from dataclasses import dataclass, field
from typing import Optional, Dict, Any, List, Set, Tuple
from collections import OrderedDict
import threading
# --- Data Structures ---
@dataclass
class CacheNode:
"""Doubly-linked node for LFU frequency buckets."""
key: str
value: Any
freq: int = 0
ttl: Optional[float] = None # Unix timestamp of expiry
prev: Optional['CacheNode'] = None
next: Optional['CacheNode'] = None
@dataclass
class FrequencyBucket:
"""A bucket of nodes with the same frequency."""
freq: int
head: CacheNode = field(default_factory=lambda: CacheNode("", None, 0)) # Dummy head
tail: CacheNode = field(default_factory=lambda: CacheNode("", None, 0)) # Dummy tail
nodes: Dict[str, CacheNode] = field(default_factory=dict)
def __post_init__(self):
self.head.next = self.tail
self.tail.prev = self.head
def add_node(self, node: CacheNode):
"""Add a node to the front of the bucket."""
node.next = self.head.next
node.prev = self.head
self.head.next.prev = node
self.head.next = node
self.nodes[node.key] = node
def remove_node(self, node: CacheNode):
"""Remove a node from the bucket."""
node.prev.next = node.next
node.next.prev = node.prev
del self.nodes[node.key]
def is_empty(self) -> bool:
return len(self.nodes) == 0
# --- Transaction Support ---
class Transaction:
def __init__(self, cache: 'InMemoryLFUCache'):
self.cache = cache
self._writes: Dict[str, Tuple[Any, Optional[float]]] = {} # key -> (value, ttl)
self._deletions: Set[str] = set()
self._committed = False
self._rolled_back = False
def get(self, key: str) -> Any:
if self._rolled_back or self._committed:
raise RuntimeError("Transaction is no longer active")
if key in self._deletions:
return None
if key in self._writes:
return self._writes[key][0]
return self.cache.get(key)
def put(self, key: str, value: Any, ttl_seconds: Optional[float] = None):
if self._rolled_back or self._committed:
raise RuntimeError("Transaction is no longer active")
self._writes[key] = (value, ttl_seconds)
def delete(self, key: str):
if self._rolled_back or self._committed:
raise RuntimeError("Transaction is no longer active")
self._deletions.add(key)
async def commit(self):
if self._rolled_back or self._committed:
raise RuntimeError("Transaction is no longer active")
await self.cache._commit_transaction(self)
self._committed = True
async def rollback(self):
if self._rolled_back or self._committed:
raise RuntimeError("Transaction is no longer active")
self._rolled_back = True
# --- Cache Implementation ---
class InMemoryLFUCache:
def __init__(self, capacity: int = 100):
self.capacity = capacity
self._data: Dict[str, CacheNode] = {} # Global cache data
self._freq_buckets: Dict[int, FrequencyBucket] = {}
self._min_freq = 0
self._lock = asyncio.Lock()
self._evictor_task: Optional[asyncio.Task] = None
self._evictor_running = False
self._transaction_lock = asyncio.Lock()
self._transactions: Dict[int, Transaction] = {}
self._transaction_counter = 0
def _create_bucket(self, freq: int) -> FrequencyBucket:
bucket = FrequencyBucket(freq)
self._freq_buckets[freq] = bucket
return bucket
def _get_bucket(self, freq: int) -> FrequencyBucket:
if freq not in self._freq_buckets:
return self._create_bucket(freq)
return self._freq_buckets[freq]
def _evict_lfu(self):
"""Evict the least frequently used item."""
if self._min_freq not in self._freq_buckets:
return None
bucket = self._freq_buckets[self._min_freq]
if bucket.is_empty():
return None
node_to_evict = bucket.tail.prev # Tail's previous is the last item
self._remove_node(node_to_evict)
return node_to_evict.key
def _remove_node(self, node: CacheNode):
"""Remove a node from its bucket and global map."""
bucket = self._get_bucket(node.freq)
bucket.remove_node(node)
if bucket.is_empty():
del self._freq_buckets[node.freq]
del self._data[node.key]
def _update_node_frequency(self, node: CacheNode):
"""Move node to the next frequency bucket and update min_freq if needed."""
old_freq = node.freq
node.freq += 1
new_freq = node.freq
bucket = self._get_bucket(old_freq)
bucket.remove_node(node)
if bucket.is_empty():
del self._freq_buckets[old_freq]
if old_freq == self._min_freq:
self._min_freq = new_freq
new_bucket = self._get_bucket(new_freq)
new_bucket.add_node(node)
def _is_expired(self, node: CacheNode) -> bool:
if node.ttl is None:
return False
return time.time() > node.ttl
def _check_and_remove_expired(self, node: CacheNode) -> bool:
"""Returns True if node was expired and removed."""
if self._is_expired(node):
self._remove_node(node)
return True
return False
async def get(self, key: str) -> Any:
async with self._lock:
if key not in self._data:
return None
node = self._data[key]
# Lazy TTL check
if self._check_and_remove_expired(node):
return None
self._update_node_frequency(node)
return node.value
async def put(self, key: str, value: Any, ttl_seconds: Optional[float] = None):
async with self._lock:
ttl = time.time() + ttl_seconds if ttl_seconds is not None else None
if key in self._data:
node = self._data[key]
# Lazy TTL check
if self._check_and_remove_expired(node):
node = None # Re-add it below
if node:
node.value = value
node.ttl = ttl
self._update_node_frequency(node)
return
# Add new node or replace existing expired one
node = CacheNode(key, value, freq=1, ttl=ttl)
self._data[key] = node
bucket = self._get_bucket(1)
bucket.add_node(node)
self._min_freq = 1
# Check capacity and evict if needed
if len(self._data) > self.capacity:
evicted_key = self._evict_lfu()
if evicted_key is not None:
# Ensure that the bucket is cleaned up if empty
pass
def begin_transaction(self) -> Transaction:
with self._transaction_lock:
self._transaction_counter += 1
tx = Transaction(self)
self._transactions[self._transaction_counter] = tx
return tx
async def _commit_transaction(self, tx: Transaction):
async with self._lock:
# Apply writes
for key, (value, ttl) in tx._writes.items():
await self.put(key, value, ttl)
# Apply deletions
for key in tx._deletions:
if key in self._data:
node = self._data[key]
self._remove_node(node)
async def start_evictor(self, interval_seconds: float = 5.0):
"""Start the background evictor task."""
async def _evict_loop():
while self._evictor_running:
try:
await asyncio.sleep(interval_seconds)
await self._evict_expired_batch()
except Exception:
pass # Silently ignore errors in background task
self._evictor_running = True
self._evictor_task = asyncio.create_task(_evict_loop())
async def stop_evictor(self):
"""Stop the background evictor task."""
self._evictor_running = False
if self._evictor_task:
self._evictor_task.cancel()
try:
await self._evictor_task
except asyncio.CancelledError:
pass
async def _evict_expired_batch(self, batch_size: int = 10):
"""Evict a batch of expired keys without holding the global lock for too long."""
to_remove = []
with self._lock:
# Collect expired nodes
for node in self._data.values():
if self._is_expired(node):
to_remove.append(node.key)
if len(to_remove) >= batch_size:
break
# Remove outside lock to avoid blocking readers
for key in to_remove:
async with self._lock:
if key in self._data:
node = self._data[key]
if self._is_expired(node):
self._remove_node(node)
# --- Unit Tests ---
async def main():
print("Running InMemoryLFUCache tests...")
# Test 1: O(1) LFU eviction
print("Test 1: LFU eviction")
cache = InMemoryLFUCache(capacity=3)
await cache.put("a", 1)
await cache.put("b", 2)
await cache.put("c", 3)
# Access a twice to make it more frequent
await cache.get("a")
await cache.get("a")
# Add d, should evict the least frequent (b)
await cache.put("d", 4)
assert await cache.get("b") is None
assert await cache.get("a") == 1
assert await cache.get("c") == 3
assert await cache.get("d") == 4
print("✓ LFU eviction works")
# Test 2: TTL eviction
print("Test 2: TTL eviction")
cache = InMemoryLFUCache(capacity=10)
await cache.put("a", 1, ttl_seconds=0.1) # Expire quickly
await asyncio.sleep(0.2)
assert await cache.get("a") is None
print("✓ TTL eviction works")
# Test 3: Transaction commit visibility
print("Test 3: Transaction visibility")
cache = InMemoryLFUCache(capacity=10)
tx = cache.begin_transaction()
tx.put("x", 100)
assert await cache.get("x") is None # Not committed yet
await tx.commit()
assert await cache.get("x") == 100 # Now visible
print("✓ Transaction commit works")
# Test 4: Transaction rollback
print("Test 4: Transaction rollback")
cache = InMemoryLFUCache(capacity=10)
await cache.put("y", 200)
tx = cache.begin_transaction()
tx.put("y", 300)
tx.delete("z") # z doesn't exist yet
await tx.rollback()
assert await cache.get("y") == 200 # Original value restored
print("✓ Transaction rollback works")
# Test 5: Concurrent access
print("Test 5: Concurrent access")
cache = InMemoryLFUCache(capacity=10)
async def worker(worker_id: int):
for i in range(5):
key = f"{worker_id}_{i}"
await cache.put(key, i)
val = await cache.get(key)
assert val == i
tasks = [asyncio.create_task(worker(i)) for i in range(50)]
await asyncio.gather(*tasks)
print("✓ Concurrent access works")
# Test 6: Background eviction
print("Test 6: Background eviction")
cache = InMemoryLFUCache(capacity=10)
await cache.put("expiring", 1, ttl_seconds=0.1)
await cache.start_evictor(interval_seconds=0.05)
await asyncio.sleep(0.2)
assert await cache.get("expiring") is None
await cache.stop_evictor()
print("✓ Background eviction works")
print("\nAll tests passed! ✅")
if __name__ == "__main__":
asyncio.run(main())
outputs/qwen3.6-27b-8bit-mlx.py
+349
View File
@@ -0,0 +1,349 @@
"""
In-Memory Concurrent LFU Cache with Async TTL Eviction and Atomic Transactions.
Pure Python 3.11+ implementation using only built-in modules.
"""
import asyncio
import time
from dataclasses import dataclass
from typing import Any, Optional, Dict, Tuple
@dataclass
class Node:
"""Represents a key-value pair in the cache with frequency and TTL tracking."""
key: Any
value: Any
freq: int = 1
expires_at: float = 0.0
prev: Optional['Node'] = None
next: Optional['Node'] = None
class DoublyLinkedList:
"""O(1) Doubly Linked List for maintaining insertion order within frequency buckets."""
def __init__(self) -> None:
self.head = Node(key=None, value=None)
self.tail = Node(key=None, value=None)
self.head.next = self.tail
self.tail.prev = self.head
self.size = 0
def append(self, node: Node) -> None:
"""Append node to the tail (MRU position). O(1)"""
node.prev = self.tail.prev
node.next = self.tail
self.tail.prev.next = node
self.tail.prev = node
self.size += 1
def remove(self, node: Node) -> None:
"""Remove specific node from the list. O(1)"""
node.prev.next = node.next
node.next.prev = node.prev
node.prev = node.next = None
self.size -= 1
def pop_lru(self) -> Optional[Node]:
"""Remove and return the LRU node (head.next). O(1)"""
if self.size == 0:
return None
node = self.head.next
self.remove(node)
return node
class Transaction:
"""
ACID-like transaction handle supporting Read-Your-Own-Writes isolation.
Uncommitted changes are buffered locally and invisible to global readers.
"""
def __init__(self, cache: 'LFUCache') -> None:
self._cache = cache
# key -> (value, expires_at) | None (marks deletion)
self._writes: Dict[Any, Optional[Tuple[Any, float]]] = {}
self._committed = False
self._rolled_back = False
def _check_active(self) -> None:
if self._committed or self._rolled_back:
raise RuntimeError("Transaction is no longer active (already committed or rolled back).")
async def get(self, key: Any) -> Any:
"""Read-Your-Own-Writes: checks local buffer first, then global cache."""
self._check_active()
if key in self._writes and self._writes[key] is not None:
val, exp = self._writes[key]
return None if time.time() > exp else val
return await self._cache.get(key)
async def put(self, key: Any, value: Any, ttl_seconds: float = 0.0) -> None:
"""Buffer write locally without mutating global state."""
self._check_active()
exp = time.time() + ttl_seconds if ttl_seconds > 0 else float('inf')
self._writes[key] = (value, exp)
async def delete(self, key: Any) -> None:
"""Buffer deletion locally."""
self._check_active()
self._writes[key] = None
async def commit(self) -> None:
"""Apply buffered changes to the global cache atomically."""
self._check_active()
async with self._cache.lock:
for key, data in self._writes.items():
if data is None:
# Apply deletion
if key in self._cache.nodes:
self._cache._remove_node(key)
else:
val, exp = data
if key in self._cache.nodes:
# Update existing
node = self._cache.nodes[key]
node.value = val
node.expires_at = exp
self._cache._update_freq(node)
else:
# Insert new
if self._cache.size >= self._cache.capacity:
self._cache._evict_lfu()
node = Node(key=key, value=val, freq=1, expires_at=exp)
self._cache.nodes[key] = node
if 1 not in self._cache.freq_buckets:
self._cache.freq_buckets[1] = DoublyLinkedList()
self._cache.freq_buckets[1].append(node)
self._cache.min_freq = 1
self._cache.size += 1
self._writes.clear()
self._committed = True
async def rollback(self) -> None:
"""Discard all pending changes without affecting global state."""
self._check_active()
self._writes.clear()
self._rolled_back = True
class LFUCache:
"""
O(1) LFU Cache with Dual-Layer TTL Eviction and Async Concurrency.
Uses frequency buckets + doubly linked lists for strict O(1) get/put.
"""
def __init__(self, capacity: int) -> None:
if capacity <= 0:
raise ValueError("Capacity must be a positive integer.")
self.capacity = capacity
self.nodes: Dict[Any, Node] = {}
self.freq_buckets: Dict[int, DoublyLinkedList] = {}
self.min_freq = 0
self.size = 0
self.lock = asyncio.Lock()
self._evictor_task: Optional[asyncio.Task] = None
self._running = False
def begin_transaction(self) -> Transaction:
"""Start a new isolated transaction session."""
return Transaction(self)
async def get(self, key: Any) -> Any:
"""Retrieve value by key. O(1) average time complexity."""
async with self.lock:
if key not in self.nodes:
return None
node = self.nodes[key]
# Lazy TTL Eviction
if time.time() > node.expires_at:
self._remove_node(key)
return None
self._update_freq(node)
return node.value
async def put(self, key: Any, value: Any, ttl_seconds: float = 0.0) -> None:
"""Insert or update key-value pair. O(1) average time complexity."""
async with self.lock:
if key in self.nodes:
node = self.nodes[key]
node.value = value
node.expires_at = time.time() + ttl_seconds if ttl_seconds > 0 else float('inf')
self._update_freq(node)
return
if self.size >= self.capacity:
self._evict_lfu()
node = Node(
key=key,
value=value,
freq=1,
expires_at=time.time() + ttl_seconds if ttl_seconds > 0 else float('inf')
)
self.nodes[key] = node
if 1 not in self.freq_buckets:
self.freq_buckets[1] = DoublyLinkedList()
self.freq_buckets[1].append(node)
self.min_freq = 1
self.size += 1
def _update_freq(self, node: Node) -> None:
"""Move node to next frequency bucket. O(1)"""
freq = node.freq
if freq in self.freq_buckets:
self.freq_buckets[freq].remove(node)
if self.freq_buckets[freq].size == 0:
del self.freq_buckets[freq]
if self.min_freq == freq:
self.min_freq += 1
node.freq += 1
if node.freq not in self.freq_buckets:
self.freq_buckets[node.freq] = DoublyLinkedList()
self.freq_buckets[node.freq].append(node)
def _evict_lfu(self) -> None:
"""Evict LRU key from the minimum frequency bucket. O(1)"""
if self.min_freq not in self.freq_buckets:
return
lru_node = self.freq_buckets[self.min_freq].pop_lru()
if lru_node:
del self.nodes[lru_node.key]
if self.freq_buckets[self.min_freq].size == 0:
del self.freq_buckets[self.min_freq]
self.size -= 1
def _remove_node(self, key: Any) -> None:
"""Remove node from cache and frequency structure. O(1)"""
node = self.nodes.pop(key)
if node.freq in self.freq_buckets:
self.freq_buckets[node.freq].remove(node)
if self.freq_buckets[node.freq].size == 0:
del self.freq_buckets[node.freq]
if self.min_freq == node.freq:
self.min_freq += 1
self.size -= 1
async def start_evictor(self, interval: float = 0.1, batch_size: int = 50) -> None:
"""Start background async TTL sweep task."""
if self._running:
return
self._running = True
self._evictor_task = asyncio.create_task(self._evict_loop(interval, batch_size))
async def stop_evictor(self) -> None:
"""Gracefully stop the background evictor."""
self._running = False
if self._evictor_task:
self._evictor_task.cancel()
try:
await self._evictor_task
except asyncio.CancelledError:
pass
async def _evict_loop(self, interval: float, batch_size: int) -> None:
"""Non-blocking background eviction that processes in small batches."""
while self._running:
async with self.lock:
now = time.time()
checked = 0
# Snapshot keys to avoid RuntimeError during iteration/mutation
for key in list(self.nodes.keys()):
if checked >= batch_size:
break
node = self.nodes.get(key)
if node and now > node.expires_at:
self._remove_node(key)
checked += 1
await asyncio.sleep(interval)
# =============================================================================
# EXECUTABLE UNIT TESTS
# =============================================================================
async def main() -> None:
print("=== Running LFU Cache Test Suite ===\n")
# a) O(1) LFU eviction order
print("[TEST a] LFU Eviction Order...")
cache = LFUCache(3)
await cache.put('a', 1)
await cache.put('b', 2)
await cache.put('c', 3)
await cache.get('a') # freq: a=2, b=1, c=1
await cache.put('d', 4) # Evicts 'b' (LRU among freq=1)
assert await cache.get('b') is None, "LFU eviction failed: 'b' should be evicted"
assert await cache.get('a') == 1
assert await cache.get('c') == 3
assert await cache.get('d') == 4
print(" ✅ PASSED\n")
# b) Lazy TTL vs Background Async Sweep
print("[TEST b] Dual-Layer TTL Eviction...")
cache2 = LFUCache(10)
await cache2.start_evictor(interval=0.05, batch_size=10)
# Lazy eviction test
await cache2.put('lazy', 'val', ttl_seconds=0.1)
await asyncio.sleep(0.12)
assert await cache2.get('lazy') is None, "Lazy TTL eviction failed"
# Background sweep test
await cache2.put('bg1', 'v', ttl_seconds=0.05)
await cache2.put('bg2', 'v', ttl_seconds=0.05)
await asyncio.sleep(0.12)
assert 'bg1' not in cache2.nodes, "Background sweep failed to remove 'bg1'"
assert 'bg2' not in cache2.nodes, "Background sweep failed to remove 'bg2'"
await cache2.stop_evictor()
print(" ✅ PASSED\n")
# c) Transaction commit visibility vs rollback state restoration
print("[TEST c] Transaction Isolation & Rollback...")
cache3 = LFUCache(10)
# Commit test
tx1 = cache3.begin_transaction()
await tx1.put('x', 100)
assert await cache3.get('x') is None, "Isolation broken: global reader saw uncommitted write"
assert await tx1.get('x') == 100, "Read-Your-Own-Writes failed"
await tx1.commit()
assert await cache3.get('x') == 100, "Commit failed: value not visible globally"
# Rollback test
tx2 = cache3.begin_transaction()
await tx2.put('y', 200)
await tx2.rollback()
assert await cache3.get('y') is None, "Rollback failed: uncommitted value leaked"
print(" ✅ PASSED\n")
# d) Stress test: 50 concurrent async tasks
print("[TEST d] Concurrency Stress Test (50 tasks, 100 ops each)...")
cache4 = LFUCache(50)
async def worker(wid: int) -> None:
for i in range(100):
key = f"k_{wid}_{i}"
op = i % 3
if op == 0:
await cache4.put(key, f"v_{i}", ttl_seconds=0.5 if i % 2 == 0 else 0)
elif op == 1:
await cache4.get(key)
else:
tx = cache4.begin_transaction()
await tx.put(key, f"tx_{i}")
if i % 2 == 0:
await tx.commit()
else:
await tx.rollback()
tasks = [asyncio.create_task(worker(i)) for i in range(50)]
await asyncio.gather(*tasks)
assert cache4.size <= 50, f"Capacity violation during stress test: size={cache4.size}"
print(" ✅ PASSED\n")
print("=== All Tests Passed Successfully ===")
if __name__ == "__main__":
asyncio.run(main())