admin/modelTesting
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Aygea Test prompt battery: 6 prompts from real project shapes

Browse Source 
Surveyed ~/dev (mewtwo) + jirachi. Battery mirrors actual workload:
  mcp_server      -> 9 MCP servers (joplin/obsidian/vault/project-rag/...)
  tts_pipeline    -> TTS/audio pipelines (Chatterbox, aygea-tts, vr-to-tts)
  webhook_bridge  -> Twitch/Discord bridges (multistream, notifier, overlay)
  data_service    -> data/API (Supabase MCP, PostgresHA, dashboard)
  automation_glue -> batch/cron glue (fix-tokens, notesCleanup)
  rust_service    -> big Rust services (NineSentry, aystreamer): tokio
                     channels + Arc/Mutex + error enums + graceful shutdown

Each prompt is ~2-3KB (fits 128k context with output room), single-file,
runnable, graded on the same 5-pillar rubric. aygea_test_battery.md is the
index + scoring notes + the prompt_id schema the dashboard will need for
multi-prompt support.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
admin
2026-07-28 17:16:03 -07:00
co-authored by Claude
parent de748f4854
commit ec6fd7157a
7 changed files with 165 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
prompts/aygea_test_battery.md
+25 -50
View File
@@ -31,62 +31,37 @@ The LFU exam tests *none* of that. These five prompts do.
---
## The 5 prompts
## The 6 prompts
Each is scoped to ~1 file, runnable, and gradable on the same 5-pillar / 100-pt rubric.
Save each as `prompts/<name>.txt` and feed it to the model.
Each is a standalone file in `prompts/`, scoped to ~1 file, runnable, gradable on the same
5-pillar / 100-pt rubric, and **small enough to fit well under a 128k context window**
(short instruction + clear requirements, no large scaffolding). Feed the `.txt` to the model:
### 1. `mcp-server.txt` — Build an MCP tool server
**Probe:** tool/schema correctness, transport, error handling. Your most common project.
> Write a single-file MCP server (TypeScript `@modelcontextprotocol/sdk` OR Python `mcp`) that
> exposes 3 tools against a JSONPlaceholder REST API:
> `get_user(id)`, `list_posts_by_user(user_id, limit)`, `search_posts(query)`.
> Each tool must: validate inputs with a schema (Zod or Pydantic), return typed results,
> handle HTTP errors + timeouts gracefully (no silent failures), and not crash on bad input.
> Run over stdio transport. Include 3 runnable tests (happy path, bad-id 404, malformed input).
> No external state — pure stdlib + fetch/httpx + the MCP SDK.
| # | File | Probe | Lang | Mirrors your projects |
|---|---|---|---|---|
| 0 | `lfu_cache_prompt.txt` | systems + async + O(1) + ACID | Python | (the original exam) |
| 1 | `mcp_server.txt` | tool/schema correctness, transport, errors | TS/Python | joplin-mcp, obsidian-mcp, project-rag, vault-mcp (9 MCPs) |
| 2 | `tts_pipeline.txt` | async queues, backpressure, retries, cancel | Python/Node | Chatterbox, aygea-tts, vr-to-tts, ffxiv-tts |
| 3 | `webhook_bridge.txt` | HMAC verify, idempotency, rate-limit, 429 backoff | Python/Node | twitch-discord-notifier, multistream, chat-overlay |
| 4 | `data_service.txt` | SQL, pooling, pagination, transactions | Python | mySupabaseMCP, PostgresHA, aygeas-dashboard |
| 5 | `automation_glue.txt` | idempotency, checkpointing, SIGINT, resumability | Python | fix-tokens, notesCleanup, batch jobs |
| 6 | `rust_service.txt` | tokio channels, Arc/Mutex shared state, error enums, shutdown | **Rust** | **NineSentry, aystreamer** (your big Rust services) |
### 2. `tts-pipeline.txt` — Audio job queue with backpressure
**Probe:** async queues, streaming, external-API resilience. Your TTS/audio shape.
> Single-file async service (Python asyncio or Node) that accepts TTS "jobs" via an async
> `submit(text, voice)` function, queues them, and processes them through a mock synthesizer
> (`await mock_synthesize(text) -> bytes`, variable 50-300ms latency). Requirements:
> bounded concurrency (max 4 in-flight), backpressure (reject when queue > 100), per-job
> retry-on-failure (max 3, exponential backoff), a `drain()` that awaits all queued jobs,
> and clean cancellation. Emit job lifecycle events to a callback. Include a 50-job stress
> test proving the concurrency cap holds and no jobs are dropped on cancel.
### 3. `webhook-bridge.txt` — Twitch/Discord event bridge
**Probe:** webhook signature verification, rate limiting, idempotency. Your bridge shape.
> Single-file HTTP service that receives Twitch EventSub webhooks (POST /webhook) and
> forwards chat events to Discord via a mock webhook. Requirements:
> HMAC-SHA256 signature verification of every request (reject 401 on mismatch), an
> in-memory idempotency store keyed by the event id (skip replays within 5 min), a
> token-bucket rate limiter capping Discord forwards to 5/sec, and graceful handling of
> Discord 429 (read Retry-After, back off). No framework deps beyond a stdlib http server.
> Include tests for: valid vs tampered signature, replayed event skipped, rate-limit trigger.
### 4. `data-service.txt` — Paginated query service with a connection pool
**Probe:** SQL, pooling, pagination, resource cleanup. Your data-service shape.
> Single-file service wrapping a (mock) Postgres pool exposing:
> `get_users(page, page_size)`, `get_user_with_posts(user_id)`, and a bulk
> `relabel_users(id_label_pairs)`. Requirements: a real pooled-connection pattern (checkout /
> return / leak-proof), parameterized queries (no string-interpolated SQL), correct
> offset/limit pagination with a total-count, a transactional bulk update that rolls back on
> any failure, and connection-checkout timeouts. Mock the DB; include tests proving: no
> connection leak across 100 calls, pagination math, rollback on partial failure.
### 5. `automation-glue.txt` — Idempotent batch job with retries
**Probe:** idempotency, partial-failure recovery, observability. Your automation shape.
> Single-file async batch processor that reads a list of "items", calls a flaky external
> `process(item)` (fails ~20% randomly), and must: be idempotent (re-running resumes from a
> checkpoint file, never reprocessing done items), retry failures with backoff (max 3),
> write a progress checkpoint after each item, log a structured JSON summary at the end
> (succeeded/failed/skipped counts + durations), and exit cleanly on SIGINT (flushing
> checkpoint). Include a test that kills mid-run and proves resume skips completed items.
### 6. `rust_service.txt` — Async tokio watcher manager *(Rust)*
**Probe:** channels, shared state, error enums, graceful shutdown — your big-Rust shape.
> A `WatcherManager` owns N async watcher tasks that poll a flaky mock source and forward
> items through `tokio::sync::mpsc` to a single consumer. Live watcher set shared via
> `Arc<Mutex<_>>`, add/remove race-free. Define an error enum; a watcher failing >5 times
> consecutively is marked unhealthy without crashing others. `shutdown()` via a
> cancellation token joins everything cleanly (no leaked tasks, no hang). Bounded channel
> with documented backpressure. Idiomatic traits/enums, `Result` everywhere, `serde` on
> output. Tests: 4-watchers run+shutdown no-hang; unhealthy marking; concurrent add/remove
> no panic.
---
(Full text of prompts 1–6 lives in their `.txt` files; summaries above for reference.)
## How to score (reuse the existing rubric)
Each prompt grades on the same 5 pillars (0–20 each, 100 total):