Files

T

adminandClaude dd5724ed77 First multi-prompt result: Qwen 6-bit TTS = 49 (vs 82 LFU) + per-prompt schema

TTS grade for qwen3.6-35b-a3b-6bit-mlx: 49/100 Critical (same model that
scored 82 on LFU). File doesn't parse + bounded-concurrency is fake
(1 worker + inner semaphore = real concurrency 1). Per-task signal:
strong on data-structures, weak on async-pipeline work.

Schema: prompt_id + PILLARS_BY_PROMPT so each entry uses its own 5 pillars.
TODO_submission_tool.md sketches the grade-as-a-tool idea for later.

Co-Authored-By: Claude <noreply@anthropic.com>

2026-07-28 17:43:38 -07:00

2.4 KiB

Raw Blame History

TODO: a submission/grading tool (so grading is one command, not a manual pipeline)

The idea (from the user)

Right now grading a model output is a manual multi-step dance:

paste output → save to outputs/<name>.py
run it, see if tests pass
manually audit 5 pillars
hand-write a JSON entry with tok/sec/tokens/TTFT/score/bugs/patch
append to data/benchmark_history.json
re-run generate_dashboard.py + redeploy

The user wants a tool (likely an MCP server you can call from your editor/agent, or a CLI) where you say:

"grade this .py, model name = X, tok/sec = Y, tokens = Z, TTFT = W" … and it runs the tests, captures pass/fail + crash output, and stages the JSON entry (so I/the agent then do the actual audit — the subjective 5-pillar scoring + bug writeup — on top of the auto-collected facts).

What it should auto-collect (deterministic, no judgment)

Run the file; capture: parse OK? tests pass? stderr/crash output?
Static scan: __slots__ present? time.monotonic() used? any min/max/sorted/heapq?
File metrics: line count, token-ish count
Append a draft entry to the JSON with total_score: null + verdict: "pending" and the auto-fields filled, so the human/agent only fills the subjective parts.

What stays human/agent (the actual audit — can't be automated honestly)

The 5-pillar scores (0–20 each)
The critical-bugs list + the patch code
The verdict + best-for recommendation
The prompt_id (which exam: lfu / tts / mcp / rust / data / automation)

Two build options

CLI (./grade.sh outputs/foo.py --model "Qwen 6-bit" --tok 69 --tokens 4000 --ttft 0.9 --prompt tts): simplest, runs anywhere, no MCP setup. Prints the draft JSON entry + a summary.
MCP server (grade_tool, list_results_tool, regenerate_dashboard_tool): callable from Claude Code / your agent so you can grade from inside a chat. Needs the MCP server running (the same pattern as your joplin/vault MCPs). More powerful but more setup.

Recommended path

Start with the CLI (fast to build, works today, no LM-Studio/LiteLLM dependency). Promote to an MCP server later once you've got LiteLLM set up for the MCP-prompt testing — then the same MCP host can serve both the grading tool AND be the endpoint you test against.

Status

Not started. Build after the multi-prompt schema (prompt_id) is in place, since the tool will need to tag which prompt an output is for.

2.4 KiB Raw Blame History Unescape Escape