AygeaandClaude
3507d33006
Grading (12 new entries, 18→30 total in benchmark_history.json): - KAT-Coder v2.5 Dev XL: lfu 49 / tts 52 / webhook 72 / rust 36 / automation 60 - Qwen3 Coder 30B: lfu 45 / tts 44 / webhook 62 / rust 54 / automation 58 - Qwen 3.6 35B-A3B uncensored (automation): 46 - Gemma 4 26B-A4B (data): 86 [tests pass] All "coder" models scored Critical Bugs across prompts — plausible-looking async code with fatal bugs (broken LFU eviction, in-flight cancel no-op, submit() raising instead of backpressuring, un-awaited async read-through). grade_run.py: switch from OpenAI-compat /v1/chat/completions (empty stats) to native /api/v1/chat — returns full stats incl. time_to_first_token_seconds. Verified on Gemma-26B (53.5 t/s, ttft 1.13). Two native-API gotchas handled: input (string) not messages; max_output_tokens not max_tokens (that 400s). New capture: outputs/gemma-4-26b-a4b-data.py (native-API run). Co-Authored-By: Claude <noreply@anthropic.com>
15 lines
380 B
JSON
15 lines
380 B
JSON
{
|
|
"model_instance_id": "google/gemma-4-26b-a4b",
|
|
"stats": {
|
|
"input_tokens": 718,
|
|
"total_output_tokens": 5153,
|
|
"reasoning_output_tokens": 1919,
|
|
"tokens_per_second": 53.47535329547263,
|
|
"time_to_first_token_seconds": 1.127
|
|
},
|
|
"response_id": "resp_d8fafc117fa4e51b927d8cd531ded49984ae59e49c39d3eb",
|
|
"_output_types": [
|
|
"reasoning",
|
|
"message"
|
|
]
|
|
} |