admin/modelTesting
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
20 Commits 1 Branch 0 Tags
3507d3300644637e4201620756e0958c777be3e0
Commit Graph
3 Commits
Author SHA1 Message Date
AygeaandClaude 3507d33006 Grade 12-model backlog + fix native-API TTFT capture 
Grading (12 new entries, 18→30 total in benchmark_history.json):
- KAT-Coder v2.5 Dev XL: lfu 49 / tts 52 / webhook 72 / rust 36 / automation 60
- Qwen3 Coder 30B: lfu 45 / tts 44 / webhook 62 / rust 54 / automation 58
- Qwen 3.6 35B-A3B uncensored (automation): 46
- Gemma 4 26B-A4B (data): 86 [tests pass]
All "coder" models scored Critical Bugs across prompts — plausible-looking
async code with fatal bugs (broken LFU eviction, in-flight cancel no-op,
submit() raising instead of backpressuring, un-awaited async read-through).

grade_run.py: switch from OpenAI-compat /v1/chat/completions (empty stats)
to native /api/v1/chat — returns full stats incl. time_to_first_token_seconds.
Verified on Gemma-26B (53.5 t/s, ttft 1.13). Two native-API gotchas handled:
input (string) not messages; max_output_tokens not max_tokens (that 400s).

New capture: outputs/gemma-4-26b-a4b-data.py (native-API run).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-28 21:17:08 -07:00
adminandClaude c98b6bb257 Capture uncensored-Qwen-35B automation (others hung via API) 
The uncensored Qwen 35B hangs/timeouts on 4/5 prompts via the API
(only automation saved). qwen3.6-27b and glm-4.7-flash also time out
via API (27B genuinely slow ~12min; GLM loops without stopping — needs
stop-token handling, noted for next session). Capture batch paused:
3 models fully captured (Gemma26B, kat-coder, qwen3-coder-30b),
grading deferred.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-28 20:21:13 -07:00
adminandClaude 0c79a997f5 Capture kat-coder + qwen3-coder-30b outputs (5 prompts each, API-run, NOT graded yet) 
Via tools/grade_run.py against LM Studio (load -> all prompts -> unload -> verify).
kat-coder:      lfu/tts/webhook/automation/rust  (lfu has a real SyntaxError: 'self._ bump_freq')
qwen3-coder-30b: lfu/tts/webhook/automation/rust  (all 4 py parse clean; terse ~2k tok outputs)
Grading deferred to a later session (capture-only mode).

Script hardening:
  - --resident / --unload-all commands (read loaded_instances, POST unload, verify)
  - MEMORY GUARD: aborts if a different model is resident (never stack 30GB models)
  - TTFT self-discovery dump (LM Studio returns empty stats non-streaming;
    fix next session: switch to /api/v1/chat stream:true -> chat.end result.stats)

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-28 19:22:41 -07:00