Add Key Findings panel + Aygea Test prompt battery
Findings panel: live stats from the data (5/10 run tests, quant dominates quality, concurrency is the killer pillar, 2/11 __slots__, 4/11 monotonic). Aygea Test (prompts/aygea_test_battery.md): 5-prompt battery derived from ~/dev + jirachi project shapes. Notes prompt_id schema for multi-prompt. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Claude
@@ -156,6 +156,19 @@ a.fv:hover{border-color:var(--cyan);box-shadow:0 0 12px rgba(0,255,200,0.3);back
|
||||
.fv-f{font-size:.7rem;letter-spacing:.08em;font-family:'Fira Code',monospace}
|
||||
.fv-s{color:var(--dim);font-size:.78rem}
|
||||
.fv-sc{font-size:1rem;font-weight:600;min-width:28px;text-align:right}
|
||||
/* key findings */
|
||||
.findings{display:grid;grid-template-columns:repeat(4,1fr);gap:12px;margin-bottom:16px}
|
||||
.fnd{background:var(--panel2);border:1px solid rgba(255,255,255,0.06);border-radius:6px;padding:14px 16px;position:relative;overflow:hidden}
|
||||
.fnd::after{content:"";position:absolute;left:0;top:0;bottom:0;width:3px;background:var(--mag);box-shadow:0 0 12px var(--mag)}
|
||||
.fnd-n{font-size:1.7rem;color:var(--cyan);text-shadow:0 0 10px rgba(0,255,200,0.3)}
|
||||
.fnd-l{color:var(--ink);font-size:.72rem;letter-spacing:.12em;text-transform:uppercase;margin-top:4px;font-family:'Fira Code',monospace}
|
||||
.fnd-s{color:var(--dim);font-size:.72rem;margin-top:6px;line-height:1.45}
|
||||
ul.findings-notes{list-style:none;padding:0;margin:0}
|
||||
ul.findings-notes li{padding:8px 0 8px 20px;border-bottom:1px solid rgba(255,255,255,0.05);position:relative;font-size:.84rem;color:var(--ink);line-height:1.5}
|
||||
ul.findings-notes li:last-child{border-bottom:none}
|
||||
ul.findings-notes li::before{content:"▸";position:absolute;left:0;color:var(--mag)}
|
||||
ul.findings-notes code{background:rgba(0,255,200,0.1);color:var(--cyan);padding:1px 5px;border-radius:3px;font-family:'Fira Code',monospace;font-size:.8rem}
|
||||
@media(max-width:900px){.findings{grid-template-columns:repeat(2,1fr)}}
|
||||
footer{color:var(--dim);font-size:.74rem;margin-top:40px;border-top:1px solid rgba(255,255,255,0.06);padding-top:14px;text-align:center}
|
||||
@media (prefers-reduced-motion: reduce){*{animation:none!important;transition:none!important}}
|
||||
"""
|
||||
@@ -327,6 +340,61 @@ def render_dashboard(data):
|
||||
else:
|
||||
family_panel = ""
|
||||
|
||||
# ---- Key findings: real stats computed from the data ----
|
||||
import ast as _ast, os as _os
|
||||
def _scan_file(m):
|
||||
fn = m.get("filename", "")
|
||||
path = _os.path.join(HERE, fn) if not _os.path.isabs(fn) else fn
|
||||
try:
|
||||
txt = open(path).read()
|
||||
parses = True
|
||||
try: _ast.parse(txt)
|
||||
except Exception: parses = False
|
||||
return {
|
||||
"slots": txt.count("__slots__") > 0,
|
||||
"monotonic": txt.count("monotonic") > 0,
|
||||
"linear": any(p in txt for p in ("sorted(", ".sort(", "heapq", "min(", "max(")),
|
||||
"parses": parses,
|
||||
}
|
||||
except Exception:
|
||||
return {"slots": False, "monotonic": False, "linear": False, "parses": None}
|
||||
|
||||
scans = {m["id"]: _scan_file(m) for m in local}
|
||||
n_run = sum(1 for m in local if m.get("tests_pass"))
|
||||
n_crit = sum(1 for m in local if m["verdict"] == "Critical Bugs")
|
||||
n_slots = sum(1 for m in local if scans[m["id"]]["slots"])
|
||||
n_mono = sum(1 for m in local if scans[m["id"]]["monotonic"])
|
||||
pillar_avg = {p: round(sum(m["breakdown"][p] for m in local)/len(local), 1) for p in PILLARS}
|
||||
weakest = min(PILLARS, key=lambda p: pillar_avg[p])
|
||||
|
||||
def _fcard(num, label, sub):
|
||||
return (f'<div class="fnd"><div class="fnd-n mono">{num}</div>'
|
||||
f'<div class="fnd-l">{label}</div>'
|
||||
f'<div class="fnd-s">{sub}</div></div>')
|
||||
|
||||
findings_tiles = "".join([
|
||||
_fcard(f"{n_run}/{len(local)}", "RUN THEIR OWN TESTS",
|
||||
"Half of local models crash before completing — runnability is the real filter."),
|
||||
_fcard(f"{n_crit}/{len(local)}", "CRITICAL BUGS",
|
||||
"Cache corruption, evict-crashes, or fatal KeyErrors — not safe for systems work."),
|
||||
_fcard(f"{n_slots}/{len(local)}", "DECLARE __slots__",
|
||||
"Rubric explicitly required it for memory efficiency; nearly all models miss it."),
|
||||
_fcard(f"{n_mono}/{len(local)}", "USE time.monotonic()",
|
||||
"The rest use the system clock — NTP jumps corrupt TTL eviction."),
|
||||
])
|
||||
findings_panel = f"""
|
||||
<div class="panel" style="margin-bottom:26px">
|
||||
<h2>▮ KEY FINDINGS — patterns across {len(local)} local models</h2>
|
||||
<div class="findings">{findings_tiles}</div>
|
||||
<ul class="findings-notes">
|
||||
<li><b>Quant depth dominates quality.</b> Same model, different quant: Qwen 3.6 35B-A3B scores <span class="mono" style="color:var(--lime)">82</span> at 6-bit but <span class="mono" style="color:var(--red)">57</span> at 4-bit — a ~25-point drop. Aggressive quants cost real logic on systems code.</li>
|
||||
<li><b>Speed ≠ quality.</b> The 4-bit Qwen is the <i>fastest</i> (83 t/s) yet scores 57; the 6-bit is slower (69 t/s) but scores 82. Pick quants for correctness first, throughput second.</li>
|
||||
<li><b>Concurrency is the killer pillar</b> (avg <span class="mono">{pillar_avg['concurrency']}/20</span>). Local models most often break on async correctness — lock type mismatches, races, and non-reentrant-lock deadlocks.</li>
|
||||
<li><b>The weakest pillar overall is {PILLAR_LABELS[weakest]}</b> (avg <span class="mono">{pillar_avg[weakest]}/20</span>). Test suites that ship with crashing code validate nothing.</li>
|
||||
<li><b>Only the cloud baseline (DeepSeek, 91) cleared Production-Ready.</b> Best local scores cap at 82 — strong scaffolding, but every submission needs a human pass on <code>__slots__</code>, monotonic clocks, and lock granularity.</li>
|
||||
</ul>
|
||||
</div>"""
|
||||
|
||||
body = f"""
|
||||
{head_html("LLM Benchmark Suite")}
|
||||
<header class="hud-bar">
|
||||
@@ -347,6 +415,7 @@ def render_dashboard(data):
|
||||
<div class="chart-box" style="height:440px"><canvas id="bar"></canvas></div>
|
||||
<div style="color:var(--dim);font-size:.72rem;margin-top:8px">Local models only — cloud baseline (DeepSeek) excluded from the speed axis. Bars flagged ⚠ have suspected GPU-offload / inference issues (not representative of the model).</div>
|
||||
</div>
|
||||
{findings_panel}
|
||||
<div class="panel" style="margin-bottom:26px">
|
||||
<h2>▮ LEADERBOARD</h2>
|
||||
<div style="overflow-x:auto">
|
||||
|
||||
Reference in New Issue
Block a user