modelTesting/prompts/aygea_test_battery.md

🎯 The Aygea Test — a multi-prompt battery drawn from your real projects

Why this exists

The LFU-cache exam (prompts/lfu_cache_prompt.txt) is an excellent probe for systems + async correctness — O(1) data structures, locks, ACID, TTL. But it's one narrow axis. It tells you nothing about whether a model can do the work you actually do every day.

So I surveyed ~/dev (mewtwo) + jirachi and found your real workload clusters into a handful of archetypes. This battery mirrors them. Run each model against all five and you get a profile — "great at MCP, weak at real-time" — instead of a single score.

What your projects actually are (the evidence)

From scanning ~/dev + jirachi:

Archetype	Examples you have	What the code does
MCP servers (9!)	joplin-mcp, obsidian-mcp, mySupabaseMCP, project-rag, yt-video-summarizer-mcp, vault-mcp	tool defs, Zod/Pydantic schema validation, stdio/SSE/StreamableHTTP transport, input parsing
TTS / audio pipelines	Chatterbox-TTS-Server, aygea-tts-app, vr-to-tts, ffxiv-tts, echokraut-bridge	external HTTP APIs, streaming responses, queueing, device/audio edge cases
Streaming / chat bridges	aygeas-multistream, twitch-vod-to-youtube, twitch-discord-notifier, aygeas-chat-overlay	webhooks, OAuth, rate limits, real-time event handling
Data / API services	project-rag, mySupabaseMCP, PostgresHA, aygeas-dashboard	SQL, connection pooling, pagination, REST/JSON
Automation / glue	fix-tokens, notesCleanup, twitch-discord-notifier	cron-style tasks, idempotency, retries, partial-failure recovery

Stack signal: TypeScript/Node is dominant, Python second, async/await is in ~half of all files, try/catch is everywhere, Zod (z.string/z.object) and Pydantic (BaseModel) are your validation layer, Docker/compose is standard.

The LFU exam tests none of that. These five prompts do.

---

The 6 prompts

Each is a standalone file in prompts/, scoped to ~1 file, runnable, gradable on the same 5-pillar / 100-pt rubric, and small enough to fit well under a 128k context window (short instruction + clear requirements, no large scaffolding). Feed the .txt to the model:

#	File	Probe	Lang	Mirrors your projects
0	lfu_cache_prompt.txt	systems + async + O(1) + ACID	Python	(the original exam)
1	mcp_server.txt	tool/schema correctness, transport, errors	TS/Python	joplin-mcp, obsidian-mcp, project-rag, vault-mcp (9 MCPs)
2	tts_pipeline.txt	async queues, backpressure, retries, cancel	Python/Node	Chatterbox, aygea-tts, vr-to-tts, ffxiv-tts
3	webhook_bridge.txt	HMAC verify, idempotency, rate-limit, 429 backoff	Python/Node	twitch-discord-notifier, multistream, chat-overlay
4	data_service.txt	SQL, pooling, pagination, transactions	Python	mySupabaseMCP, PostgresHA, aygeas-dashboard
5	automation_glue.txt	idempotency, checkpointing, SIGINT, resumability	Python	fix-tokens, notesCleanup, batch jobs
6	rust_service.txt	tokio channels, Arc/Mutex shared state, error enums, shutdown	Rust	NineSentry, aystreamer (your big Rust services)

6. rust_service.txt — Async tokio watcher manager (Rust)

Probe: channels, shared state, error enums, graceful shutdown — your big-Rust shape.

A WatcherManager owns N async watcher tasks that poll a flaky mock source and forward items through tokio::sync::mpsc to a single consumer. Live watcher set shared via Arc<Mutex<_>>, add/remove race-free. Define an error enum; a watcher failing >5 times consecutively is marked unhealthy without crashing others. shutdown() via a cancellation token joins everything cleanly (no leaked tasks, no hang). Bounded channel with documented backpressure. Idiomatic traits/enums, Result everywhere, serde on output. Tests: 4-watchers run+shutdown no-hang; unhealthy marking; concurrent add/remove no panic.

---

(Full text of prompts 1–6 lives in their .txt files; summaries above for reference.)

How to score (reuse the existing rubric)

Each prompt grades on the same 5 pillars (0–20 each, 100 total):

1. Complexity / correctness — does it actually work, edge cases handled?
2. Async / concurrency — locks, backpressure, cancellation, no races
3. Error handling — no silent failures, retries, timeouts, graceful degradation
4. Resource / state safety — connection leaks, idempotency, checkpoint integrity
5. Test integrity — real assertions vs always-pass; do the tests catch the bugs above?

Note: pillars 3–5 map cleanly to your repeated patterns (try/catch everywhere, retries, validation, "no silent failures" — your own recurring concern).

How the dashboard should evolve for this

The current JSON schema assumes one prompt (exam_prompt). To support a battery:

• Add prompt_id to each model entry (e.g. "lfu", "mcp", "tts").
• The leaderboard gets a prompt filter (default: show a model's average across all prompts it has run).
• A new per-model radar across prompts shows the profile ("strong at MCP, weak at async pipelines") — the real value of a battery over a single exam.

When you're ready to run these, tell me which prompt + model and I'll wire up grading the same way as the LFU set. The generator will need the prompt_id field + the filter; I can do that in one pass once you have ≥1 result from a second prompt.