Grade 12-model backlog + fix native-API TTFT capture
Grading (12 new entries, 18→30 total in benchmark_history.json): - KAT-Coder v2.5 Dev XL: lfu 49 / tts 52 / webhook 72 / rust 36 / automation 60 - Qwen3 Coder 30B: lfu 45 / tts 44 / webhook 62 / rust 54 / automation 58 - Qwen 3.6 35B-A3B uncensored (automation): 46 - Gemma 4 26B-A4B (data): 86 [tests pass] All "coder" models scored Critical Bugs across prompts — plausible-looking async code with fatal bugs (broken LFU eviction, in-flight cancel no-op, submit() raising instead of backpressuring, un-awaited async read-through). grade_run.py: switch from OpenAI-compat /v1/chat/completions (empty stats) to native /api/v1/chat — returns full stats incl. time_to_first_token_seconds. Verified on Gemma-26B (53.5 t/s, ttft 1.13). Two native-API gotchas handled: input (string) not messages; max_output_tokens not max_tokens (that 400s). New capture: outputs/gemma-4-26b-a4b-data.py (native-API run). Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1,12 +1,15 @@
|
||||
{
|
||||
"usage": {
|
||||
"prompt_tokens": 593,
|
||||
"completion_tokens": 10029,
|
||||
"total_tokens": 10622,
|
||||
"completion_tokens_details": {
|
||||
"reasoning_tokens": 7460
|
||||
}
|
||||
"model_instance_id": "google/gemma-4-26b-a4b",
|
||||
"stats": {
|
||||
"input_tokens": 718,
|
||||
"total_output_tokens": 5153,
|
||||
"reasoning_output_tokens": 1919,
|
||||
"tokens_per_second": 53.47535329547263,
|
||||
"time_to_first_token_seconds": 1.127
|
||||
},
|
||||
"stats": {},
|
||||
"system_fingerprint": "qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive"
|
||||
"response_id": "resp_d8fafc117fa4e51b927d8cd531ded49984ae59e49c39d3eb",
|
||||
"_output_types": [
|
||||
"reasoning",
|
||||
"message"
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user