Add Aygea Test prompt battery: 6 prompts from real project shapes

Surveyed ~/dev (mewtwo) + jirachi. Battery mirrors actual workload:
  mcp_server      -> 9 MCP servers (joplin/obsidian/vault/project-rag/...)
  tts_pipeline    -> TTS/audio pipelines (Chatterbox, aygea-tts, vr-to-tts)
  webhook_bridge  -> Twitch/Discord bridges (multistream, notifier, overlay)
  data_service    -> data/API (Supabase MCP, PostgresHA, dashboard)
  automation_glue -> batch/cron glue (fix-tokens, notesCleanup)
  rust_service    -> big Rust services (NineSentry, aystreamer): tokio
                     channels + Arc/Mutex + error enums + graceful shutdown

Each prompt is ~2-3KB (fits 128k context with output room), single-file,
runnable, graded on the same 5-pillar rubric. aygea_test_battery.md is the
index + scoring notes + the prompt_id schema the dashboard will need for
multi-prompt support.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-28 17:16:03 -07:00
co-authored by Claude
parent de748f4854
commit ec6fd7157a
7 changed files with 165 additions and 50 deletions
+22
View File
@@ -0,0 +1,22 @@
Write a complete, single-file async batch processor (Python 3.11+ asyncio, stdlib only) that processes a list of items through a flaky external call with checkpointing, idempotency, and clean interruption — the kind of "automation glue" that runs unattended and must be re-runnable.
### Behavior
- Input: a list of items (e.g. `["job-a", "job-b", ...]`, 50200 of them).
- For each item, call `async def process(item: str) -> str` (provided) that succeeds ~80% of the time and raises `ProcessingError` otherwise, with random latency 1080ms.
- Persist progress to a checkpoint file (`checkpoint.json`) after each item: the set of completed item ids + a running summary.
### Requirements
1. **Idempotency / resumability:** on start, load the checkpoint; skip any item already marked completed. Re-running with the same input + checkpoint must NEVER reprocess a completed item and must converge to all-done.
2. **Retries with backoff:** each item retries up to 3 times on `ProcessingError` with exponential backoff (e.g. 0.1s, 0.2s, 0.4s) before being recorded as `failed`. (After retries, a failed item is terminal — it does not block the rest.)
3. **Bounded concurrency:** process up to 8 items at once.
4. **Checkpoint integrity:** the checkpoint file must never be left half-written / corrupt if the process dies mid-write (write to a temp file then atomically rename). A crash at any point must leave a valid checkpoint.
5. **Graceful SIGINT:** on Ctrl-C / SIGINT, stop accepting new items, let in-flight ones finish (or cancel cleanly), flush the checkpoint, then exit 0. No partial item is ever recorded as completed.
6. **Structured summary:** at the end, print a single JSON line: `{"succeeded": n, "failed": n, "skipped": n, "total": n, "elapsed_ms": ...}`.
### Included test
Include a runnable test that:
- a) Runs a batch, kills mid-run (simulate via a small in-process cancellation), restarts, and asserts: no completed item was reprocessed, and the final state is all items either succeeded-or-failed.
- b) Asserts the checkpoint file is valid JSON at every observed moment (write a watcher that reads it repeatedly during a run and confirms it always parses).
- c) Asserts the concurrency cap of 8 holds.
Provide clean, well-commented code that runs directly via `python file.py`.