diff --git a/prompts/automation_glue.txt b/prompts/automation_glue.txt new file mode 100644 index 0000000..959e18c --- /dev/null +++ b/prompts/automation_glue.txt @@ -0,0 +1,22 @@ +Write a complete, single-file async batch processor (Python 3.11+ asyncio, stdlib only) that processes a list of items through a flaky external call with checkpointing, idempotency, and clean interruption — the kind of "automation glue" that runs unattended and must be re-runnable. + +### Behavior +- Input: a list of items (e.g. `["job-a", "job-b", ...]`, 50–200 of them). +- For each item, call `async def process(item: str) -> str` (provided) that succeeds ~80% of the time and raises `ProcessingError` otherwise, with random latency 10–80ms. +- Persist progress to a checkpoint file (`checkpoint.json`) after each item: the set of completed item ids + a running summary. + +### Requirements +1. **Idempotency / resumability:** on start, load the checkpoint; skip any item already marked completed. Re-running with the same input + checkpoint must NEVER reprocess a completed item and must converge to all-done. +2. **Retries with backoff:** each item retries up to 3 times on `ProcessingError` with exponential backoff (e.g. 0.1s, 0.2s, 0.4s) before being recorded as `failed`. (After retries, a failed item is terminal — it does not block the rest.) +3. **Bounded concurrency:** process up to 8 items at once. +4. **Checkpoint integrity:** the checkpoint file must never be left half-written / corrupt if the process dies mid-write (write to a temp file then atomically rename). A crash at any point must leave a valid checkpoint. +5. **Graceful SIGINT:** on Ctrl-C / SIGINT, stop accepting new items, let in-flight ones finish (or cancel cleanly), flush the checkpoint, then exit 0. No partial item is ever recorded as completed. +6. **Structured summary:** at the end, print a single JSON line: `{"succeeded": n, "failed": n, "skipped": n, "total": n, "elapsed_ms": ...}`. + +### Included test +Include a runnable test that: +- a) Runs a batch, kills mid-run (simulate via a small in-process cancellation), restarts, and asserts: no completed item was reprocessed, and the final state is all items either succeeded-or-failed. +- b) Asserts the checkpoint file is valid JSON at every observed moment (write a watcher that reads it repeatedly during a run and confirms it always parses). +- c) Asserts the concurrency cap of 8 holds. + +Provide clean, well-commented code that runs directly via `python file.py`. diff --git a/prompts/aygea_test_battery.md b/prompts/aygea_test_battery.md index 5127db6..e83b2db 100644 --- a/prompts/aygea_test_battery.md +++ b/prompts/aygea_test_battery.md @@ -31,62 +31,37 @@ The LFU exam tests *none* of that. These five prompts do. --- -## The 5 prompts +## The 6 prompts -Each is scoped to ~1 file, runnable, and gradable on the same 5-pillar / 100-pt rubric. -Save each as `prompts/.txt` and feed it to the model. +Each is a standalone file in `prompts/`, scoped to ~1 file, runnable, gradable on the same +5-pillar / 100-pt rubric, and **small enough to fit well under a 128k context window** +(short instruction + clear requirements, no large scaffolding). Feed the `.txt` to the model: -### 1. `mcp-server.txt` — Build an MCP tool server -**Probe:** tool/schema correctness, transport, error handling. Your most common project. -> Write a single-file MCP server (TypeScript `@modelcontextprotocol/sdk` OR Python `mcp`) that -> exposes 3 tools against a JSONPlaceholder REST API: -> `get_user(id)`, `list_posts_by_user(user_id, limit)`, `search_posts(query)`. -> Each tool must: validate inputs with a schema (Zod or Pydantic), return typed results, -> handle HTTP errors + timeouts gracefully (no silent failures), and not crash on bad input. -> Run over stdio transport. Include 3 runnable tests (happy path, bad-id 404, malformed input). -> No external state — pure stdlib + fetch/httpx + the MCP SDK. +| # | File | Probe | Lang | Mirrors your projects | +|---|---|---|---|---| +| 0 | `lfu_cache_prompt.txt` | systems + async + O(1) + ACID | Python | (the original exam) | +| 1 | `mcp_server.txt` | tool/schema correctness, transport, errors | TS/Python | joplin-mcp, obsidian-mcp, project-rag, vault-mcp (9 MCPs) | +| 2 | `tts_pipeline.txt` | async queues, backpressure, retries, cancel | Python/Node | Chatterbox, aygea-tts, vr-to-tts, ffxiv-tts | +| 3 | `webhook_bridge.txt` | HMAC verify, idempotency, rate-limit, 429 backoff | Python/Node | twitch-discord-notifier, multistream, chat-overlay | +| 4 | `data_service.txt` | SQL, pooling, pagination, transactions | Python | mySupabaseMCP, PostgresHA, aygeas-dashboard | +| 5 | `automation_glue.txt` | idempotency, checkpointing, SIGINT, resumability | Python | fix-tokens, notesCleanup, batch jobs | +| 6 | `rust_service.txt` | tokio channels, Arc/Mutex shared state, error enums, shutdown | **Rust** | **NineSentry, aystreamer** (your big Rust services) | -### 2. `tts-pipeline.txt` — Audio job queue with backpressure -**Probe:** async queues, streaming, external-API resilience. Your TTS/audio shape. -> Single-file async service (Python asyncio or Node) that accepts TTS "jobs" via an async -> `submit(text, voice)` function, queues them, and processes them through a mock synthesizer -> (`await mock_synthesize(text) -> bytes`, variable 50-300ms latency). Requirements: -> bounded concurrency (max 4 in-flight), backpressure (reject when queue > 100), per-job -> retry-on-failure (max 3, exponential backoff), a `drain()` that awaits all queued jobs, -> and clean cancellation. Emit job lifecycle events to a callback. Include a 50-job stress -> test proving the concurrency cap holds and no jobs are dropped on cancel. - -### 3. `webhook-bridge.txt` — Twitch/Discord event bridge -**Probe:** webhook signature verification, rate limiting, idempotency. Your bridge shape. -> Single-file HTTP service that receives Twitch EventSub webhooks (POST /webhook) and -> forwards chat events to Discord via a mock webhook. Requirements: -> HMAC-SHA256 signature verification of every request (reject 401 on mismatch), an -> in-memory idempotency store keyed by the event id (skip replays within 5 min), a -> token-bucket rate limiter capping Discord forwards to 5/sec, and graceful handling of -> Discord 429 (read Retry-After, back off). No framework deps beyond a stdlib http server. -> Include tests for: valid vs tampered signature, replayed event skipped, rate-limit trigger. - -### 4. `data-service.txt` — Paginated query service with a connection pool -**Probe:** SQL, pooling, pagination, resource cleanup. Your data-service shape. -> Single-file service wrapping a (mock) Postgres pool exposing: -> `get_users(page, page_size)`, `get_user_with_posts(user_id)`, and a bulk -> `relabel_users(id_label_pairs)`. Requirements: a real pooled-connection pattern (checkout / -> return / leak-proof), parameterized queries (no string-interpolated SQL), correct -> offset/limit pagination with a total-count, a transactional bulk update that rolls back on -> any failure, and connection-checkout timeouts. Mock the DB; include tests proving: no -> connection leak across 100 calls, pagination math, rollback on partial failure. - -### 5. `automation-glue.txt` — Idempotent batch job with retries -**Probe:** idempotency, partial-failure recovery, observability. Your automation shape. -> Single-file async batch processor that reads a list of "items", calls a flaky external -> `process(item)` (fails ~20% randomly), and must: be idempotent (re-running resumes from a -> checkpoint file, never reprocessing done items), retry failures with backoff (max 3), -> write a progress checkpoint after each item, log a structured JSON summary at the end -> (succeeded/failed/skipped counts + durations), and exit cleanly on SIGINT (flushing -> checkpoint). Include a test that kills mid-run and proves resume skips completed items. +### 6. `rust_service.txt` — Async tokio watcher manager *(Rust)* +**Probe:** channels, shared state, error enums, graceful shutdown — your big-Rust shape. +> A `WatcherManager` owns N async watcher tasks that poll a flaky mock source and forward +> items through `tokio::sync::mpsc` to a single consumer. Live watcher set shared via +> `Arc>`, add/remove race-free. Define an error enum; a watcher failing >5 times +> consecutively is marked unhealthy without crashing others. `shutdown()` via a +> cancellation token joins everything cleanly (no leaked tasks, no hang). Bounded channel +> with documented backpressure. Idiomatic traits/enums, `Result` everywhere, `serde` on +> output. Tests: 4-watchers run+shutdown no-hang; unhealthy marking; concurrent add/remove +> no panic. --- +(Full text of prompts 1–6 lives in their `.txt` files; summaries above for reference.) + ## How to score (reuse the existing rubric) Each prompt grades on the same 5 pillars (0–20 each, 100 total): diff --git a/prompts/data_service.txt b/prompts/data_service.txt new file mode 100644 index 0000000..bec1bbe --- /dev/null +++ b/prompts/data_service.txt @@ -0,0 +1,30 @@ +Write a complete, single-file data-access service (Python 3.11+ asyncio, stdlib only) that wraps a **mock** Postgres connection pool. It must demonstrate correct pooling, parameterized queries, pagination, and transactional integrity. + +### Mock layer +Provide an in-memory mock that stands in for a real pool. Something like: +`async def acquire() -> Connection` / `connection.release()`, where `Connection` has +`async def fetch(query, params)` and `async def execute(query, params)`. Seed it with ~25 +"users" and ~5 "posts" per user. Make `acquire()` sometimes wait briefly and make it +**track how many connections are checked out** so leaks are detectable. + +### Service API +1. `async def get_users(page: int, page_size: int) -> dict` — returns `{"items": [...], "page": p, "page_size": s, "total": N}`. Validate page/page_size (page ≥ 1, 1 ≤ page_size ≤ 100). +2. `async def get_user_with_posts(user_id: int) -> dict` — one user + their posts (efficient: don't N+1). +3. `async def relabel_users(pairs: list[tuple[int, str]]) -> int` — bulk update each user's `label`. Transactional: if ANY update fails (e.g. an id doesn't exist), the whole batch rolls back and returns 0 (or raises) — never partially applied. + +### Requirements +1. **Connection-pool discipline:** every acquire is paired with a release in a `finally`/context manager. The service must prove across 100 calls that connections-in-use returns to 0 (no leak). +2. **Parameterized queries:** all queries pass values as parameters (mock `(query, params)`), NEVER string-interpolated. Include at least one test that would "fail" on injection if interpolation were used. +3. **Pagination correctness:** `total` is the true count regardless of page; last page returns the correct remainder; out-of-range page returns empty (not an error). +4. **Transactional bulk:** `relabel_users` wraps all updates in one transaction; a partial failure rolls back. Prove it. +5. **Acquire timeout:** `acquire()` must respect a max-wait (e.g. 2s) and raise a clear error if the pool is exhausted, rather than hanging. +6. **No silent failures:** missing user → clear `KeyError`-style result; bad params → validation error. + +### Included tests +Include a runnable test section that asserts: +- a) 100 sequential `get_users` calls leave the pool at 0 checked-out (no leak). +- b) Pagination math: total matches seed count; last page has the right remainder. +- c) `relabel_users` with one bad id rolls back — verify NO labels changed afterward. +- d) Concurrent `get_user_with_posts` for 10 users at once completes without pool exhaustion. + +Provide clean, well-commented code that runs directly via `python file.py`. diff --git a/prompts/mcp_server.txt b/prompts/mcp_server.txt new file mode 100644 index 0000000..f948863 --- /dev/null +++ b/prompts/mcp_server.txt @@ -0,0 +1,21 @@ +Write a complete, single-file MCP (Model Context Protocol) tool server in TypeScript using `@modelcontextprotocol/sdk` (OR Python using the `mcp` package) that exposes 3 tools against the public JSONPlaceholder API (https://jsonplaceholder.typicode.com). + +### Tools to implement +1. `get_user(id)` — fetch a single user by numeric id. Validate `id` is a positive integer. +2. `list_posts_by_user(user_id, limit)` — fetch a user's posts. `user_id` is a positive int; `limit` is an optional int between 1 and 100 (default 10). +3. `search_posts(query)` — fetch posts whose `title` contains the query string (case-insensitive). `query` is a non-empty string max 200 chars. + +### Requirements +- **Schema validation:** every tool must validate its inputs with a schema (Zod for TS, Pydantic for Python). Invalid input returns a structured error, never crashes. +- **Typed results:** tools return typed/structured output (not raw strings). +- **Error handling:** handle HTTP errors, timeouts, and non-JSON responses gracefully — NO silent failures. Each failure path returns a clear error result. Use a reasonable per-request timeout (e.g. 8s). +- **Transport:** run over stdio (`StdioServerTransport` / `stdio_server`). +- **No external state:** pure stdlib + fetch/httpx + the MCP SDK. No database, no files. + +### Included tests +At the bottom of the file, include a runnable test section that exercises: +- a) Happy path: `get_user(1)` returns a user with the expected fields. +- b) Bad-id 404 / not-found is handled (does not throw). +- c) Malformed input (e.g. `get_user(-5)`, `list_posts_by_user("x")`, `search_posts("")`) is rejected by validation with a clear error. + +Provide clean, well-commented code. The file must be directly runnable: `npx tsx file.ts` (or `python file.py`). diff --git a/prompts/rust_service.txt b/prompts/rust_service.txt new file mode 100644 index 0000000..3779935 --- /dev/null +++ b/prompts/rust_service.txt @@ -0,0 +1,20 @@ +Write a complete, single-file Rust service using tokio that manages a set of background "watchers" connected by channels, with safe shared state and robust error handling. This mirrors a real async Rust service (à la ninesentry / aystreamer). + +### What it does +A `WatcherManager` owns N async "watcher" tasks. Each watcher periodically polls a mock source (`async fn mock_fetch(id: u32) -> Result, FetchError>` — returns Ok with 0–3 random items, or Err ~15% of the time). Items a watcher produces are forwarded through a channel to a single consumer task that aggregates them. The manager supports adding/removing watchers at runtime and a clean shutdown. + +### Requirements (idiomatic async Rust) +1. **Channels:** use `tokio::sync::mpsc` (or `broadcast`) to send items from watchers to the consumer. Demonstrate correct sender/receiver ownership and dropping (no deadlock on shutdown). +2. **Shared state:** the manager's live watcher set must be shared safely across tasks — use `Arc>` or `Arc>`. Add/remove must be race-free. +3. **Error handling:** define a `thiserror`-style (or manual `enum`) error type. Polling failures must be logged/handled per-watcher and must NOT abort the whole manager; a watcher that fails repeatedly (>5 consecutive) should be marked unhealthy. +4. **Cancellation / shutdown:** on a `shutdown()` signal (use a `tokio::sync::oneshot` or `CancellationToken`), every watcher and the consumer must terminate cleanly within their current poll cycle — no leaked tasks, no panics, no hung select!. +5. **Backpressure:** if the consumer is slow, the channel must not grow unbounded (bounded channel); document the chosen bound and what happens when full. +6. **Idiomatic types:** use traits + enums where natural, `Result` everywhere errors can occur, `#[derive(Debug, Clone)]` on shared data, and `serde::Serialize` on the aggregated output struct. + +### Included test +Include a `#[tokio::test]` (or a `main` that asserts) that: +- a) Spawns a manager with 4 watchers, lets it run ~500ms, shuts down, and asserts all tasks joined (no hang) and the consumer received >0 items. +- b) Asserts that injecting a watcher whose `mock_fetch` always errors gets marked unhealthy after 5 consecutive failures (without crashing others). +- c) Asserts add + remove of a watcher at runtime updates the live count with no race (no panic on concurrent mutation). + +Provide clean, well-commented code. Must compile and run with current stable Rust + tokio (single file, deps via comments or assume `tokio = { features = ["full"] }`, `serde`, `thiserror`). diff --git a/prompts/tts_pipeline.txt b/prompts/tts_pipeline.txt new file mode 100644 index 0000000..446695e --- /dev/null +++ b/prompts/tts_pipeline.txt @@ -0,0 +1,25 @@ +Write a complete, single-file async TTS (text-to-speech) job pipeline in Python 3.11+ (asyncio, stdlib only) OR Node.js (no external deps beyond stdlib). It accepts text jobs and processes them through a mock synthesizer with bounded concurrency. + +### API +- `async def submit(text: str, voice: str) -> str` — enqueue a job, return a job id immediately (non-blocking). +- `async def drain()` — wait until all queued AND in-flight jobs are finished. +- `async def cancel(job_id)` — cancel a queued job; if in-flight, mark for cancellation when safe. +- A way to register a callback that receives job lifecycle events: `queued`, `started`, `completed`, `failed`, `cancelled`. + +### Mock synthesizer +Provide `async def mock_synthesize(text: str) -> bytes` that sleeps a random 50–300ms and returns `b"\x00" * len(text)` (or similar). Make it fail ~10% of the time (raise a synthetic error) so retry logic is exercised. + +### Requirements +1. **Bounded concurrency:** at most 4 jobs synthesize at once. +2. **Backpressure:** when the queue length exceeds 100, `submit` must reject immediately with a clear error (not block, not grow unbounded). +3. **Retry on failure:** a job that fails must retry up to 3 times with exponential backoff (e.g. 0.1s, 0.2s, 0.4s). After exhausting retries it emits `failed`. +4. **Clean cancellation:** cancelling must not leak tasks or leave the worker pool in a bad state; `drain()` must always return. +5. **Thread/async safety:** no shared mutable state races between submit, workers, and callbacks. + +### Included test +Include an `async def main()` test that: +- a) Submits 50 jobs, drains, and asserts exactly 4-or-fewer ran concurrently at any time (record max concurrency). +- b) Proves the 100-job backpressure cap rejects when exceeded. +- c) Submits a batch, cancels one mid-flight, drains, and asserts no jobs were dropped/duplicated and the pool is still usable afterward. + +Provide clean, well-commented code that runs directly via `python file.py`. diff --git a/prompts/webhook_bridge.txt b/prompts/webhook_bridge.txt new file mode 100644 index 0000000..382526d --- /dev/null +++ b/prompts/webhook_bridge.txt @@ -0,0 +1,22 @@ +Write a complete, single-file HTTP webhook bridge in Python 3.11+ (stdlib `http.server` / `asyncio`) OR Node.js (stdlib `http`). It receives Twitch-style EventSub webhook events on `POST /webhook` and forwards them to a mock Discord webhook. + +### Behavior +- Receive JSON webhook payloads of the form `{"event_id": "...", "type": "chat", "data": {...}}`. +- For each valid event, POST a small summary to the Discord webhook (mock it as `async def discord_send(payload)` that succeeds ~95% of the time and occasionally returns HTTP 429 with a `Retry-After` header). +- Reject anything that isn't a valid signed event. + +### Requirements +1. **HMAC signature verification:** every request must carry a header `X-Signature: ` computed with a shared secret. Mismatch → `401 Unauthorized`. Use `hmac.compare_digest` for constant-time comparison. +2. **Idempotency:** maintain an in-memory store keyed by `event_id`. If the same id is seen again within 5 minutes, skip forwarding and return `200` (replayed). Evict stale entries. +3. **Rate limiting:** a token-bucket limiter capping Discord forwards to **5 per second**. When exceeded, queue/delay rather than spamming. +4. **429 backoff:** when Discord returns 429, read `Retry-After` and back off that many seconds before retrying (once). +5. **No silent failures:** malformed JSON, missing headers, and downstream errors must produce clear responses/logs, never a bare 500 or a swallowed exception. + +### Included tests +Include a runnable test section that: +- a) Sends a correctly-signed request → forwarded once, returns 200. +- b) Sends a tampered signature → 401, nothing forwarded. +- c) Replays the same event_id within 5 min → skipped (forward count does not increase). +- d) Bursts >5 events in one second → asserts the rate limiter delayed the excess (no more than 5 forwards/sec). + +Provide clean, well-commented code that runs directly via `python file.py`.