diff --git a/dev/v0.3.1/PRD.md b/dev/v0.3.1/PRD.md new file mode 100644 index 0000000..fc9c294 --- /dev/null +++ b/dev/v0.3.1/PRD.md @@ -0,0 +1,192 @@ +# pepa v0.3.1 — PRD: LLM prompt cost optimization + +**Status**: Design draft. No code in this version yet — this PRD is the +spec future commits implement against. Owner: operator. +**Trigger**: TimeWeb admin panel after first day of v0.3.0 live: +~34K tokens used in a half-day session (mostly bot + some smoke). +At the 6-calls/hour cap that projects to **~480 ₽/month** (101 ₽/M in, +608 ₽/M out for gpt-5.4-mini). Manageable but worth shrinking — most +of the per-call cost is repeated infrastructure tokens, not the +situational signal the model actually uses. + +## Goals + +1. Cut per-advise() input tokens from ~800 → ≤300 (target 250). +2. Preserve correctness: the LLM must still see enough context to + produce a valid `skill_id` from the registry and a useful rationale. +3. Keep all changes transparent to the rest of the runtime — the + public `advise()` / `complete()` surface area doesn't change. + +Non-goals: +- Switching providers. TimeWeb stays. +- Caching the LLM's *responses* (cache key would be situational, too + many misses to be worth the bookkeeping). +- Touching the analytical loops (postmortem / reflect). They're called + less often and need fuller context; cost there is acceptable. + +## Cost breakdown — what we're optimizing + +Measured on live advise() calls (TimeWeb gpt-5.4-mini, single advisor +trigger): + +| Block | tokens (avg) | % of call | +|--------------------------------------|--------------|-----------| +| `skillRegistryPrompt({limit:1800})` | ~450 | 56% | +| System instructions (rules + JSON) | ~200 | 25% | +| User snapshot + threats + need + recent | ~150 | 19% | +| **Total input** | **~800** | **100%** | +| Output (JSON answer) | ~40-50 | — | + +The registry block dominates. It currently lists all 30+ registered +skills with their human titles. The model rarely needs the full list — +most decisions are within 5-8 plausible skills per trigger. + +## Proposed changes + +### 1. Compact registry format (P1, biggest win) + +Drop human titles and the per-skill descriptions; switch to +namespace-grouped, comma-separated id lists. + +**Before** (~450 tokens): +``` +Valid skill ids (USE ONLY THESE for avoid_skill / prefer_skill): + craft: + - craft.bed — Craft bed + - craft.chest — Craft chest + - craft.furnace — Craft furnace + ... + survive: + - survive.acquire-food — Acquire food + - survive.eat — Eat + ... +``` + +**After** (~100 tokens): +``` +Valid skill ids (USE EXACTLY one of these or null): + craft: bed, chest, furnace, planks, sticks, torch, wooden-axe, + wooden-pickaxe, wooden-sword, stone-axe, stone-pickaxe, stone-sword + survive: acquire-food, eat, flee, pillar-up, sleep + gather: logs, stone, wool + recovery: tunnel-out + explore: far, wander + village: build-shelter, choose-base, deposit-surplus, place-chest + farm: wheat + diag: physics, scan, match +``` + +Saving: **~350 tokens/call**. + +Implementation: add `skillRegistryPrompt({ mode: "compact" })` mode in +`runtime/skill-registry.js`. Default mode stays for slow analytical +loops (postmortem / reflect) which can afford the verbose form. + +### 2. Need-scoped registry (P2, additional ~50 token saving) + +When `activeNeed` is set, filter the registry to skills plausibly +relevant to that level + always-available safety skills. + +Relevance table (manually curated, lives in `runtime/manifesto/needs.js`): + +| Need | Relevant skills (in addition to ALWAYS set) | +|-------------------|------------------------------------------------------------| +| alive | survive.flee, survive.eat, recovery.tunnel-out | +| food | survive.acquire-food, survive.eat, farm.wheat | +| tools_wood | gather.logs, craft.planks, craft.sticks, craft.wooden-* | +| shelter_basic | gather.wool, craft.bed, village.build-shelter, village.choose-base | +| tools_stone | gather.stone, craft.sticks, craft.stone-* | +| armor_basic | gather.wool (placeholder) | +| food_security | farm.wheat, survive.acquire-food | +| tools_iron | gather.stone | +| armor_iron | (none — no skill yet) | +| village_seed | craft.chest, village.deposit-surplus, village.build-shelter | +| village_full | (full registry) | +| ALWAYS | survive.flee, survive.pillar-up, recovery.tunnel-out, | +| | explore.far, explore.wander | + +Compact + scoped = **~50 tokens** for the registry block (down from 450). + +Add a `prompt-builder.test.js` checking that: +- `survive.flee` is always present (emergency safety) +- The recommended skill from the previous call would still be in the + scoped registry (regression protection) + +### 3. Snapshot pruning (P3, ~50 tokens) + +The user-prompt snapshot includes fields the LLM rarely consults: +`weather`, `experience`, `dimension`, `biome`, `players[]`. Drop them +from the advise() user-prompt builder. Keep `position`, `hp`, `food`, +`isDay`, `closestHostile`, `activeNeed`, `recent dispatches`, +`hazards.footBlock` (lava detection), top inventory keys. + +### 4. Prompt caching — investigation (P4) + +OpenAI and Anthropic both support implicit prompt caching: when ≥1024 +prefix tokens are identical across consecutive requests, the prefix +is billed once. TimeWeb's docs are silent on this. + +Task: probe whether TimeWeb passes through OpenAI's `prompt_tokens_details.cached_tokens` +field. If yes, *increase* the system prefix length (keep verbose registry) +because cached input is ~10x cheaper than fresh. If no, full optimization +1+2+3 still wins. + +Add a one-off check in `scripts/check-timeweb.js`: print +`payload?.usage?.prompt_tokens_details?.cached_tokens` if present. + +### 5. Telemetry — per-trigger token attribution (P5) + +Today `advisor_recommendations` records `tokens_in` per row but the +operator has no easy view of *which trigger types* are most expensive. + +Extend `scripts/list-improvements.js --stats` to also print per-trigger: +``` +trigger_reason total applied ok fail avg_in avg_out cost_₽ share% +wedged_* 20 18 3 15 280 45 2.1 45% +emergency_* 3 3 2 1 240 50 0.3 6% +repeat_* 8 7 0 7 260 42 0.8 17% +preempt_retry_* 14 12 2 10 290 44 1.5 32% +``` + +`cost_₽` = avg_in × calls × IN_PRICE + avg_out × calls × OUT_PRICE, +with prices read from env (`TIMEWEB_PRICE_IN_RUB_PER_M`, +`TIMEWEB_PRICE_OUT_RUB_PER_M`). + +## Acceptance + +After v0.3.1 lands: +- Re-run `node scripts/check-timeweb.js` probe 3 (`advise()`): + expect `tokens_in` ≤ 300 (was ~800). +- Re-run probe 4 (auto-trigger flow): rationale still references the + registered skill correctly. +- Run live for 1 hour, check `node scripts/list-improvements.js --stats`: + per-trigger `avg_in` ≤ 300. +- Existing 360 tests still green; new prompt-builder tests cover the + scoped registry behaviour. + +## Out of scope (later versions) + +- Tool/function-calling instead of free-form JSON (TimeWeb support unclear). +- Custom model selection per trigger (gpt-5.4-nano for routine repeats, + gpt-5.4-mini for emergencies). Defer until cost/quality data points + exist. +- Embedding-based prior-recommendation similarity check ("we already + told the bot to tunnel-out at this exact wedge 10 minutes ago, skip"). + +## Implementation order + +When this version is greenlit, work in this order on a single branch +`v0.3.1` (one PR per session, per recently-updated workflow memory): + +1. P1 compact registry mode + prompt-builder test +2. P2 need-scoped registry (extend `runtime/manifesto/needs.js` with + `relevantSkills`) +3. P3 snapshot pruning in `fast-advisor.js#buildUserPrompt` +4. P4 caching probe (one-off) +5. P5 cost telemetry in CLI viewer +6. STATUS.md + smoke retest + PR + +All changes are additive; no behaviour regression expected. If +real-world after v0.3.1 shows the LLM giving worse advice with the +compact registry, fall back to default mode by flipping a single +constant in `fast-advisor.js`.