modelTesting/prompts/aygea_test_battery.md

🎯 The Aygea Test — a multi-prompt battery drawn from your real projects

Why this exists

The LFU-cache exam (prompts/lfu_cache_prompt.txt) is an excellent probe for systems + async correctness — O(1) data structures, locks, ACID, TTL. But it's one narrow axis. It tells you nothing about whether a model can do the work you actually do every day.

So I surveyed ~/dev (mewtwo) + jirachi and found your real workload clusters into a handful of archetypes. This battery mirrors them. Run each model against all five and you get a profile — "great at MCP, weak at real-time" — instead of a single score.

What your projects actually are (the evidence)

From scanning ~/dev + jirachi:

Archetype	Examples you have	What the code does
MCP servers (9!)	joplin-mcp, obsidian-mcp, mySupabaseMCP, project-rag, yt-video-summarizer-mcp, vault-mcp	tool defs, Zod/Pydantic schema validation, stdio/SSE/StreamableHTTP transport, input parsing
TTS / audio pipelines	Chatterbox-TTS-Server, aygea-tts-app, vr-to-tts, ffxiv-tts, echokraut-bridge	external HTTP APIs, streaming responses, queueing, device/audio edge cases
Streaming / chat bridges	aygeas-multistream, twitch-vod-to-youtube, twitch-discord-notifier, aygeas-chat-overlay	webhooks, OAuth, rate limits, real-time event handling
Data / API services	project-rag, mySupabaseMCP, PostgresHA, aygeas-dashboard	SQL, connection pooling, pagination, REST/JSON
Automation / glue	fix-tokens, notesCleanup, twitch-discord-notifier	cron-style tasks, idempotency, retries, partial-failure recovery

Stack signal: TypeScript/Node is dominant, Python second, async/await is in ~half of all files, try/catch is everywhere, Zod (z.string/z.object) and Pydantic (BaseModel) are your validation layer, Docker/compose is standard.

The LFU exam tests none of that. These five prompts do.

---

The 5 prompts

Each is scoped to ~1 file, runnable, and gradable on the same 5-pillar / 100-pt rubric. Save each as prompts/<name>.txt and feed it to the model.

1. mcp-server.txt — Build an MCP tool server

Probe: tool/schema correctness, transport, error handling. Your most common project.

Write a single-file MCP server (TypeScript @modelcontextprotocol/sdk OR Python mcp) that exposes 3 tools against a JSONPlaceholder REST API: get_user(id), list_posts_by_user(user_id, limit), search_posts(query). Each tool must: validate inputs with a schema (Zod or Pydantic), return typed results, handle HTTP errors + timeouts gracefully (no silent failures), and not crash on bad input. Run over stdio transport. Include 3 runnable tests (happy path, bad-id 404, malformed input). No external state — pure stdlib + fetch/httpx + the MCP SDK.

2. tts-pipeline.txt — Audio job queue with backpressure

Probe: async queues, streaming, external-API resilience. Your TTS/audio shape.

Single-file async service (Python asyncio or Node) that accepts TTS "jobs" via an async submit(text, voice) function, queues them, and processes them through a mock synthesizer (await mock_synthesize(text) -> bytes, variable 50-300ms latency). Requirements: bounded concurrency (max 4 in-flight), backpressure (reject when queue > 100), per-job retry-on-failure (max 3, exponential backoff), a drain() that awaits all queued jobs, and clean cancellation. Emit job lifecycle events to a callback. Include a 50-job stress test proving the concurrency cap holds and no jobs are dropped on cancel.

3. webhook-bridge.txt — Twitch/Discord event bridge

Probe: webhook signature verification, rate limiting, idempotency. Your bridge shape.

Single-file HTTP service that receives Twitch EventSub webhooks (POST /webhook) and forwards chat events to Discord via a mock webhook. Requirements: HMAC-SHA256 signature verification of every request (reject 401 on mismatch), an in-memory idempotency store keyed by the event id (skip replays within 5 min), a token-bucket rate limiter capping Discord forwards to 5/sec, and graceful handling of Discord 429 (read Retry-After, back off). No framework deps beyond a stdlib http server. Include tests for: valid vs tampered signature, replayed event skipped, rate-limit trigger.

4. data-service.txt — Paginated query service with a connection pool

Probe: SQL, pooling, pagination, resource cleanup. Your data-service shape.

Single-file service wrapping a (mock) Postgres pool exposing: get_users(page, page_size), get_user_with_posts(user_id), and a bulk relabel_users(id_label_pairs). Requirements: a real pooled-connection pattern (checkout / return / leak-proof), parameterized queries (no string-interpolated SQL), correct offset/limit pagination with a total-count, a transactional bulk update that rolls back on any failure, and connection-checkout timeouts. Mock the DB; include tests proving: no connection leak across 100 calls, pagination math, rollback on partial failure.

5. automation-glue.txt — Idempotent batch job with retries

Probe: idempotency, partial-failure recovery, observability. Your automation shape.

Single-file async batch processor that reads a list of "items", calls a flaky external process(item) (fails ~20% randomly), and must: be idempotent (re-running resumes from a checkpoint file, never reprocessing done items), retry failures with backoff (max 3), write a progress checkpoint after each item, log a structured JSON summary at the end (succeeded/failed/skipped counts + durations), and exit cleanly on SIGINT (flushing checkpoint). Include a test that kills mid-run and proves resume skips completed items.

---

How to score (reuse the existing rubric)

Each prompt grades on the same 5 pillars (0–20 each, 100 total):

1. Complexity / correctness — does it actually work, edge cases handled?
2. Async / concurrency — locks, backpressure, cancellation, no races
3. Error handling — no silent failures, retries, timeouts, graceful degradation
4. Resource / state safety — connection leaks, idempotency, checkpoint integrity
5. Test integrity — real assertions vs always-pass; do the tests catch the bugs above?

Note: pillars 3–5 map cleanly to your repeated patterns (try/catch everywhere, retries, validation, "no silent failures" — your own recurring concern).

How the dashboard should evolve for this

The current JSON schema assumes one prompt (exam_prompt). To support a battery:

• Add prompt_id to each model entry (e.g. "lfu", "mcp", "tts").
• The leaderboard gets a prompt filter (default: show a model's average across all prompts it has run).
• A new per-model radar across prompts shows the profile ("strong at MCP, weak at async pipelines") — the real value of a battery over a single exam.

When you're ready to run these, tell me which prompt + model and I'll wire up grading the same way as the LFU set. The generator will need the prompt_id field + the filter; I can do that in one pass once you have ≥1 result from a second prompt.