adminandClaude
dd5724ed77
TTS grade for qwen3.6-35b-a3b-6bit-mlx: 49/100 Critical (same model that scored 82 on LFU). File doesn't parse + bounded-concurrency is fake (1 worker + inner semaphore = real concurrency 1). Per-task signal: strong on data-structures, weak on async-pipeline work. Schema: prompt_id + PILLARS_BY_PROMPT so each entry uses its own 5 pillars. TODO_submission_tool.md sketches the grade-as-a-tool idea for later. Co-Authored-By: Claude <noreply@anthropic.com>
47 lines
2.4 KiB
Markdown
47 lines
2.4 KiB
Markdown
# TODO: a submission/grading tool (so grading is one command, not a manual pipeline)
|
||
|
||
## The idea (from the user)
|
||
Right now grading a model output is a manual multi-step dance:
|
||
1. paste output → save to `outputs/<name>.py`
|
||
2. run it, see if tests pass
|
||
3. manually audit 5 pillars
|
||
4. hand-write a JSON entry with tok/sec/tokens/TTFT/score/bugs/patch
|
||
5. append to `data/benchmark_history.json`
|
||
6. re-run `generate_dashboard.py` + redeploy
|
||
|
||
The user wants a **tool** (likely an MCP server you can call from your editor/agent,
|
||
or a CLI) where you say:
|
||
> "grade this .py, model name = X, tok/sec = Y, tokens = Z, TTFT = W"
|
||
… and it runs the tests, captures pass/fail + crash output, and stages the JSON entry
|
||
(so I/the agent then do the actual *audit* — the subjective 5-pillar scoring + bug writeup —
|
||
on top of the auto-collected facts).
|
||
|
||
## What it should auto-collect (deterministic, no judgment)
|
||
- [ ] Run the file; capture: parse OK? tests pass? stderr/crash output?
|
||
- [ ] Static scan: `__slots__` present? `time.monotonic()` used? any `min/max/sorted/heapq`?
|
||
- [ ] File metrics: line count, token-ish count
|
||
- [ ] Append a **draft** entry to the JSON with `total_score: null` + `verdict: "pending"`
|
||
and the auto-fields filled, so the human/agent only fills the subjective parts.
|
||
|
||
## What stays human/agent (the actual audit — can't be automated honestly)
|
||
- The 5-pillar scores (0–20 each)
|
||
- The critical-bugs list + the patch code
|
||
- The verdict + best-for recommendation
|
||
- The `prompt_id` (which exam: lfu / tts / mcp / rust / data / automation)
|
||
|
||
## Two build options
|
||
1. **CLI** (`./grade.sh outputs/foo.py --model "Qwen 6-bit" --tok 69 --tokens 4000 --ttft 0.9 --prompt tts`):
|
||
simplest, runs anywhere, no MCP setup. Prints the draft JSON entry + a summary.
|
||
2. **MCP server** (`grade_tool`, `list_results_tool`, `regenerate_dashboard_tool`):
|
||
callable from Claude Code / your agent so you can grade from inside a chat. Needs the MCP
|
||
server running (the same pattern as your joplin/vault MCPs). More powerful but more setup.
|
||
|
||
## Recommended path
|
||
Start with the **CLI** (fast to build, works today, no LM-Studio/LiteLLM dependency).
|
||
Promote to an MCP server later once you've got LiteLLM set up for the MCP-prompt testing —
|
||
then the same MCP host can serve both the grading tool AND be the endpoint you test against.
|
||
|
||
## Status
|
||
Not started. Build after the multi-prompt schema (`prompt_id`) is in place, since the tool
|
||
will need to tag which prompt an output is for.
|