adminandClaude
ec6fd7157a
Surveyed ~/dev (mewtwo) + jirachi. Battery mirrors actual workload:
mcp_server -> 9 MCP servers (joplin/obsidian/vault/project-rag/...)
tts_pipeline -> TTS/audio pipelines (Chatterbox, aygea-tts, vr-to-tts)
webhook_bridge -> Twitch/Discord bridges (multistream, notifier, overlay)
data_service -> data/API (Supabase MCP, PostgresHA, dashboard)
automation_glue -> batch/cron glue (fix-tokens, notesCleanup)
rust_service -> big Rust services (NineSentry, aystreamer): tokio
channels + Arc/Mutex + error enums + graceful shutdown
Each prompt is ~2-3KB (fits 128k context with output room), single-file,
runnable, graded on the same 5-pillar rubric. aygea_test_battery.md is the
index + scoring notes + the prompt_id schema the dashboard will need for
multi-prompt support.
Co-Authored-By: Claude <noreply@anthropic.com>
89 lines
5.4 KiB
Markdown
89 lines
5.4 KiB
Markdown
# 🎯 The Aygea Test — a multi-prompt battery drawn from your real projects
|
||
|
||
## Why this exists
|
||
|
||
The LFU-cache exam (`prompts/lfu_cache_prompt.txt`) is an excellent probe for
|
||
**systems + async correctness** — O(1) data structures, locks, ACID, TTL. But it's
|
||
one narrow axis. It tells you nothing about whether a model can do the work you
|
||
*actually* do every day.
|
||
|
||
So I surveyed `~/dev` (mewtwo) + `jirachi` and found your real workload clusters into a
|
||
handful of archetypes. This battery mirrors them. Run each model against all five and
|
||
you get a profile — "great at MCP, weak at real-time" — instead of a single score.
|
||
|
||
## What your projects actually are (the evidence)
|
||
|
||
From scanning `~/dev` + `jirachi`:
|
||
|
||
| Archetype | Examples you have | What the code does |
|
||
|---|---|---|
|
||
| **MCP servers** (9!) | joplin-mcp, obsidian-mcp, mySupabaseMCP, project-rag, yt-video-summarizer-mcp, vault-mcp | tool defs, Zod/Pydantic schema validation, stdio/SSE/StreamableHTTP transport, input parsing |
|
||
| **TTS / audio pipelines** | Chatterbox-TTS-Server, aygea-tts-app, vr-to-tts, ffxiv-tts, echokraut-bridge | external HTTP APIs, streaming responses, queueing, device/audio edge cases |
|
||
| **Streaming / chat bridges** | aygeas-multistream, twitch-vod-to-youtube, twitch-discord-notifier, aygeas-chat-overlay | webhooks, OAuth, rate limits, real-time event handling |
|
||
| **Data / API services** | project-rag, mySupabaseMCP, PostgresHA, aygeas-dashboard | SQL, connection pooling, pagination, REST/JSON |
|
||
| **Automation / glue** | fix-tokens, notesCleanup, twitch-discord-notifier | cron-style tasks, idempotency, retries, partial-failure recovery |
|
||
|
||
**Stack signal:** TypeScript/Node is dominant, Python second, async/await is in ~half of
|
||
all files, try/catch is everywhere, Zod (`z.string`/`z.object`) and Pydantic (`BaseModel`)
|
||
are your validation layer, Docker/compose is standard.
|
||
|
||
The LFU exam tests *none* of that. These five prompts do.
|
||
|
||
---
|
||
|
||
## The 6 prompts
|
||
|
||
Each is a standalone file in `prompts/`, scoped to ~1 file, runnable, gradable on the same
|
||
5-pillar / 100-pt rubric, and **small enough to fit well under a 128k context window**
|
||
(short instruction + clear requirements, no large scaffolding). Feed the `.txt` to the model:
|
||
|
||
| # | File | Probe | Lang | Mirrors your projects |
|
||
|---|---|---|---|---|
|
||
| 0 | `lfu_cache_prompt.txt` | systems + async + O(1) + ACID | Python | (the original exam) |
|
||
| 1 | `mcp_server.txt` | tool/schema correctness, transport, errors | TS/Python | joplin-mcp, obsidian-mcp, project-rag, vault-mcp (9 MCPs) |
|
||
| 2 | `tts_pipeline.txt` | async queues, backpressure, retries, cancel | Python/Node | Chatterbox, aygea-tts, vr-to-tts, ffxiv-tts |
|
||
| 3 | `webhook_bridge.txt` | HMAC verify, idempotency, rate-limit, 429 backoff | Python/Node | twitch-discord-notifier, multistream, chat-overlay |
|
||
| 4 | `data_service.txt` | SQL, pooling, pagination, transactions | Python | mySupabaseMCP, PostgresHA, aygeas-dashboard |
|
||
| 5 | `automation_glue.txt` | idempotency, checkpointing, SIGINT, resumability | Python | fix-tokens, notesCleanup, batch jobs |
|
||
| 6 | `rust_service.txt` | tokio channels, Arc/Mutex shared state, error enums, shutdown | **Rust** | **NineSentry, aystreamer** (your big Rust services) |
|
||
|
||
### 6. `rust_service.txt` — Async tokio watcher manager *(Rust)*
|
||
**Probe:** channels, shared state, error enums, graceful shutdown — your big-Rust shape.
|
||
> A `WatcherManager` owns N async watcher tasks that poll a flaky mock source and forward
|
||
> items through `tokio::sync::mpsc` to a single consumer. Live watcher set shared via
|
||
> `Arc<Mutex<_>>`, add/remove race-free. Define an error enum; a watcher failing >5 times
|
||
> consecutively is marked unhealthy without crashing others. `shutdown()` via a
|
||
> cancellation token joins everything cleanly (no leaked tasks, no hang). Bounded channel
|
||
> with documented backpressure. Idiomatic traits/enums, `Result` everywhere, `serde` on
|
||
> output. Tests: 4-watchers run+shutdown no-hang; unhealthy marking; concurrent add/remove
|
||
> no panic.
|
||
|
||
---
|
||
|
||
(Full text of prompts 1–6 lives in their `.txt` files; summaries above for reference.)
|
||
|
||
## How to score (reuse the existing rubric)
|
||
|
||
Each prompt grades on the same 5 pillars (0–20 each, 100 total):
|
||
1. **Complexity / correctness** — does it actually work, edge cases handled?
|
||
2. **Async / concurrency** — locks, backpressure, cancellation, no races
|
||
3. **Error handling** — no silent failures, retries, timeouts, graceful degradation
|
||
4. **Resource / state safety** — connection leaks, idempotency, checkpoint integrity
|
||
5. **Test integrity** — real assertions vs always-pass; do the tests catch the bugs above?
|
||
|
||
> Note: pillars 3–5 map cleanly to your repeated patterns (try/catch everywhere,
|
||
> retries, validation, "no silent failures" — your own recurring concern).
|
||
|
||
## How the dashboard should evolve for this
|
||
|
||
The current JSON schema assumes one prompt (`exam_prompt`). To support a battery:
|
||
- Add `prompt_id` to each model entry (e.g. `"lfu"`, `"mcp"`, `"tts"`).
|
||
- The leaderboard gets a **prompt filter** (default: show a model's average across all
|
||
prompts it has run).
|
||
- A new **per-model radar across prompts** shows the profile ("strong at MCP, weak at async
|
||
pipelines") — the real value of a battery over a single exam.
|
||
|
||
When you're ready to run these, tell me which prompt + model and I'll wire up grading the
|
||
same way as the LFU set. The generator will need the `prompt_id` field + the filter; I can
|
||
do that in one pass once you have ≥1 result from a second prompt.
|