Initial benchmark suite: 8 graded models + cyberpunk dashboard generator
- prompts/: LFU cache exam + 5-pillar grading rubric - outputs/: 8 model .py outputs (local + cloud baseline) - data/benchmark_history.json: graded results (scores, metrics, bugs, patches) - generate_dashboard.py: builds dashboard.html + pages/*.html from JSON - Dockerfile + DEPLOY.md: Gitea→Coolify deploy (build-step, nginx static) - .gitignore: generated HTML excluded (built on deploy) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Claude
@@ -0,0 +1,135 @@
|
||||
```markdown
|
||||
You are a Principal Systems Engineer and Automated LLM Benchmark Auditor.
|
||||
|
||||
Your objective is to evaluate Python code generated by local LLMs, store the evaluation data in a persistent JSON file (`~/dev/modelTesting/data/benchmark_history.json`), and build/update an interactive HTML reporting dashboard (`~/dev/modelTesting/dashboard.html`).
|
||||
|
||||
---
|
||||
|
||||
### STEP 1: Rigorous Code Audit
|
||||
Analyze the provided Python file against these 5 technical pillars (score each from 0 to 20):
|
||||
|
||||
1. Complexity Violations (O(1) Audit):
|
||||
- Check for hidden O(N) or O(log N) operations (e.g., `min()`, `sorted()`, heaps, linear scans during eviction or frequency node relocation, or key iterations).
|
||||
|
||||
2. Async Race Conditions & Deadlocks:
|
||||
- Check for un-synchronized shared state access between async background tasks (e.g., TTL sweep) and foreground operations (`get`, `put`, `commit`).
|
||||
- Check for potential deadlocks or holding `asyncio.Lock` across `asyncio.sleep` or external IO.
|
||||
|
||||
3. Transactional Isolation Leaks:
|
||||
- Verify Read-Your-Own-Writes and Snapshot isolation. Does uncommitted write/delete state leak into the global cache?
|
||||
- Does `rollback()` cleanly clear local buffers without leaving dangling nodes or modified frequency counters?
|
||||
- Does `commit()` handle mid-execution exceptions gracefully?
|
||||
|
||||
4. Memory Leaks & Edge Cases:
|
||||
- Check doubly-linked list node unlinking. Does `unlink` update list `size` and prune empty frequency buckets from `freq_map`?
|
||||
- Does key deletion or TTL eviction correctly adjust `min_freq`?
|
||||
- Is `time.monotonic()` used instead of system clock `time.time()`? Are `__slots__` declared for memory efficiency?
|
||||
|
||||
5. Test Coverage Integrity:
|
||||
- Does the included test suite validate real edge cases (concurrency, race conditions, rollback isolation), or are assertions trivial/always-pass?
|
||||
|
||||
---
|
||||
|
||||
### STEP 2: Metrics & Categorization
|
||||
From your audit, extract the following structured data:
|
||||
- Overall Score (0-100 sum of the 5 pillar scores)
|
||||
- Quant/Model Details (Extracted from file name or user input, e.g., "Qwen 3.6 35B Q6_K")
|
||||
- Token Speed (tok/sec, if provided)
|
||||
- Verdict: ["Production Ready", "Minor Logic Flaws", "Critical Bugs", "Broken / Unusable"]
|
||||
- Archetype / Best For: 1-sentence recommendation (e.g., "Great for rapid local prototyping, but needs manual checks on pointer deletions.")
|
||||
- Critical Flaws Summary: Short bullet list of identified bugs.
|
||||
- Refactored Code Patches: Code snippets fixing the specific bugs.
|
||||
|
||||
---
|
||||
|
||||
### STEP 3: Maintain Persistent JSON Storage
|
||||
Read `~/dev/modelTesting/data/benchmark_history.json` (create it if it doesn't exist).
|
||||
Append or update the entry for the evaluated model in the following format:
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": "2026-07-28T12:00:00Z",
|
||||
"model_name": "Qwen 3.6 35B",
|
||||
"quant": "Q6_K",
|
||||
"tok_sec": 68.86,
|
||||
"filename": "qwen3.6-35b-q6k-2026-07-28.py",
|
||||
"total_score": 88,
|
||||
"breakdown": {
|
||||
"complexity": 18,
|
||||
"concurrency": 16,
|
||||
"isolation": 18,
|
||||
"memory_edge_cases": 18,
|
||||
"test_integrity": 18
|
||||
},
|
||||
"verdict": "Minor Logic Flaws",
|
||||
"best_for": "Fast offline daily driving & agentic refactoring.",
|
||||
"critical_bugs": [
|
||||
"Stale min_freq on key deletion causes capacity breach."
|
||||
],
|
||||
"patch_code": "..."
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### STEP 4: Regenerate/Update `~/dev/modelTesting/dashboard.html`
|
||||
|
||||
Generate (or overwrite) `~/dev/modelTesting/dashboard.html` with a standalone, dark-themed HTML dashboard.
|
||||
|
||||
The HTML dashboard MUST include:
|
||||
|
||||
1. **Header & Summary Stats:** Total models tested, top performer, average benchmark score.
|
||||
2. **Chart.js CDN Integration:**
|
||||
* **Bar Chart:** Model Scores vs. Generation Speed (tok/sec).
|
||||
* **Radar Chart:** Multi-axis comparison of the top 3 models across the 5 audit pillars.
|
||||
|
||||
|
||||
3. **Leaderboard Table:**
|
||||
* Columns: Model Name | Quant | Speed | Score | Verdict | Archetype | Details Button
|
||||
|
||||
|
||||
4. **Collapsible / Accordion Audit Cards:**
|
||||
* Detailed breakdowns for every tested model showing Pillar Scores, Critical Bugs, and exact Python Patch Snippets.
|
||||
|
||||
|
||||
5. **Styling:** Modern dark-mode UI (Tailwind CSS via CDN or raw clean CSS grid/flexbox) with color-coded badges:
|
||||
* Green for Production Ready (90-100)
|
||||
* Yellow/Blue for Minor Flaws (75-89)
|
||||
* Red for Critical Flaws/Broken (<75)
|
||||
|
||||
|
||||
|
||||
Execute these steps automatically whenever a new model code output is submitted for evaluation.
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 What the Resulting Dashboard Will Look Like
|
||||
|
||||
When your Grading Agent processes outputs, it will write out a clean, standalone single-file `dashboard.html` that you can open in any browser (`open ~/dev/modelTesting/dashboard.html`).
|
||||
|
||||
```text
|
||||
+-----------------------------------------------------------------------------+
|
||||
| 🧪 LOCAL LLM BENCHMARK SUITE - LFU CACHE & ACID AUDIT |
|
||||
| Total Models Tested: 5 | Top Performer: DeepSeek V4 Flash Max | Avg: 78.4 |
|
||||
+-----------------------------------------------------------------------------+
|
||||
| |
|
||||
| [ Chart: Score vs tok/sec ] [ Radar: 5-Pillar Comparison ] |
|
||||
| |
|
||||
+-----------------------------------------------------------------------------+
|
||||
| LEADERBOARD |
|
||||
| Model Quant Speed Score Verdict Best For |
|
||||
| ------------------------------------------------------------------------- |
|
||||
| DeepSeek V4 Flash MoE Cloud 98 [PROD] ACID Logic |
|
||||
| Qwen 3.6 35B Q6_K 68.8 t/s 88 [FLAWS] Daily Driver |
|
||||
| KAT-Coder 2.5 MLX 65.2 t/s 85 [FLAWS] OOP Scaffolding|
|
||||
+-----------------------------------------------------------------------------+
|
||||
| ▼ DETAILED AUDIT: Qwen 3.6 35B (Q6_K) |
|
||||
| - Complexity: 18/20 | Concurrency: 16/20 | Isolation: 18/20 |
|
||||
| - Critical Bugs: Stale min_freq on key deletion causes capacity breach. |
|
||||
| - Code Patch: [ View Refactored Diff ] |
|
||||
+-----------------------------------------------------------------------------+
|
||||
|
||||
```
|
||||
@@ -0,0 +1,31 @@
|
||||
Write a complete, single-file Python 3.11+ module that implements an In-Memory Concurrent LFU (Least Frequently Used) Cache with Async TTL Eviction and Atomic Transactions.
|
||||
|
||||
DO NOT use any external third-party libraries—use only pure Python built-ins (asyncio, dataclasses, typing, weakref, time, etc.).
|
||||
|
||||
### Strict Functional Requirements:
|
||||
|
||||
1. Strict O(1) Time Complexity:
|
||||
- Both `get(key)` and `put(key, value, ttl_seconds)` MUST run in true O(1) average time complexity.
|
||||
- You MUST use a frequency-bucket system with doubly linked lists (or an equivalent O(1) data structure) to maintain access counts and frequency tiers. A heap/priority queue or linear scan is strictly forbidden as it is O(log N) or O(N).
|
||||
|
||||
2. Dual-Layer TTL Eviction:
|
||||
- Lazy Eviction: `get()` and `put()` must check and evict stale keys immediately upon access.
|
||||
- Background Async Eviction Loop: Implement a non-blocking background `asyncio` task (`start_evictor()`, `stop_evictor()`) that periodically purges expired keys in small batches without holding a lock across the entire dataset or blocking reads.
|
||||
|
||||
3. Atomic Transaction Isolation (ACID-like sub-sessions):
|
||||
- Implement `cache.begin_transaction() -> Transaction` which returns a transaction handle.
|
||||
- Inside a transaction, calls to `tx.put()`, `tx.get()`, and `tx.delete()` must support "Read Your Own Writes" (uncommitted local changes are visible to this transaction).
|
||||
- Global cache readers MUST NOT see uncommitted transaction writes until `await tx.commit()` is called.
|
||||
- Calling `tx.rollback()` must completely discard all pending changes without mutating global frequencies or TTL states.
|
||||
|
||||
4. Async Concurrency & Thread-Safety:
|
||||
- Handle race conditions between concurrent readers, writers, background evictors, and transaction commits using fine-grained `asyncio.Lock` primitives.
|
||||
|
||||
5. Included Executable Unit Tests:
|
||||
- At the bottom of the file, include an `async def main()` test suite that verifies:
|
||||
a) O(1) LFU eviction order when capacity is reached.
|
||||
b) Lazy TTL vs. Background Async Sweep eviction.
|
||||
c) Transaction commit visibility vs. Transaction rollback state restoration.
|
||||
d) Stress test with 50 concurrent async tasks reading/writing simultaneously.
|
||||
|
||||
Provide clean, well-commented code that executes directly via `python script.py`.
|
||||
Reference in New Issue
Block a user