# TODO: a submission/grading tool (so grading is one command, not a manual pipeline)

## The idea (from the user)
Right now grading a model output is a manual multi-step dance:
1. paste output → save to `outputs/<name>.py`
2. run it, see if tests pass
3. manually audit 5 pillars
4. hand-write a JSON entry with tok/sec/tokens/TTFT/score/bugs/patch
5. append to `data/benchmark_history.json`
6. re-run `generate_dashboard.py` + redeploy

The user wants a **tool** (likely an MCP server you can call from your editor/agent,
or a CLI) where you say:
> "grade this .py, model name = X, tok/sec = Y, tokens = Z, TTFT = W"
… and it runs the tests, captures pass/fail + crash output, and stages the JSON entry
(so I/the agent then do the actual *audit* — the subjective 5-pillar scoring + bug writeup —
on top of the auto-collected facts).

## What it should auto-collect (deterministic, no judgment)
- [ ] Run the file; capture: parse OK? tests pass? stderr/crash output?
- [ ] Static scan: `__slots__` present? `time.monotonic()` used? any `min/max/sorted/heapq`?
- [ ] File metrics: line count, token-ish count
- [ ] Append a **draft** entry to the JSON with `total_score: null` + `verdict: "pending"`
      and the auto-fields filled, so the human/agent only fills the subjective parts.

## What stays human/agent (the actual audit — can't be automated honestly)
- The 5-pillar scores (0–20 each)
- The critical-bugs list + the patch code
- The verdict + best-for recommendation
- The `prompt_id` (which exam: lfu / tts / mcp / rust / data / automation)

## Two build options
1. **CLI** (`./grade.sh outputs/foo.py --model "Qwen 6-bit" --tok 69 --tokens 4000 --ttft 0.9 --prompt tts`):
   simplest, runs anywhere, no MCP setup. Prints the draft JSON entry + a summary.
2. **MCP server** (`grade_tool`, `list_results_tool`, `regenerate_dashboard_tool`):
   callable from Claude Code / your agent so you can grade from inside a chat. Needs the MCP
   server running (the same pattern as your joplin/vault MCPs). More powerful but more setup.

## Recommended path
Start with the **CLI** (fast to build, works today, no LM-Studio/LiteLLM dependency).
Promote to an MCP server later once you've got LiteLLM set up for the MCP-prompt testing —
then the same MCP host can serve both the grading tool AND be the endpoint you test against.

## Status
Not started. Build after the multi-prompt schema (`prompt_id`) is in place, since the tool
will need to tag which prompt an output is for.