feat(v0.3.0): paradigm shift — TimeWeb-only LLM + persistent advisor trail + improvement queue

This is the rc.4 batch the user requested:

  1. Emergency triggers (low HP + close hostile, lava-under-foot)
     bypass the long cooldown so the LLM is consulted BEFORE the bot
     dies, not after.
  2. Active manifesto need is now included in the advisor user prompt
     — the LLM picks suggestions that satisfy the bot's current
     concrete need (L2 tools_wood → "gather logs nearby" not
     "explore further").
  3. Every advisor recommendation is persisted to SQLite
     (advisor_recommendations table) with full token usage. The
     reflex marks 'applied=1' when it dispatches and updates
     outcome_ok/code when the dispatch completes. Ground truth for
     "is the LLM actually helping" lives in the DB, not in logs.
  4. Pi CLI is OUT of every background loop. coach/postmortem and
     coach/reflect now go through the same TimeWeb endpoint
     fast-advisor uses, via the shared coach/llm-call.js helper.
     Pi is reserved for manual operator commands.
  5. The LLM (postmortem, reflect, advisor) can flag "structural
     gaps" — missing skills/features the operator should implement.
     These land in the new improvement_requests table. Dedup by
     title bumps `votes` instead of inserting duplicates so the
     queue doesn't bloat. Operator views via
     `node scripts/list-improvements.js`.
  6. A deterministic trigger-tuner runs hourly: reads 24h of
     recommendation stats, flags triggers whose success rate is
     below 25% (sample ≥ 5) or whose prompts are expensive (>1000
     input tokens) with mediocre payoff. Improvements get
     source="tuner", category="tuning". No LLM call.

New files:
  runtime/coach/llm-call.js        — askAnalytical() helper
  runtime/coach/trigger-tuner.js   — stats → improvements
  runtime/coach/trigger-tuner.test.js
  scripts/list-improvements.js     — operator CLI

Schema additions:
  advisor_recommendations: id, ts, trigger_reason, planned_skill,
    recommended_skill, action, rationale, active_need, tokens_in,
    tokens_out, latency_ms, applied, outcome_ok, outcome_code, outcome_at
  improvement_requests: id, ts, source, category, title, description,
    context, priority, status, duplicate_of, votes, implemented_at, notes

Renamed env-var consumers:
  Pi-coach drainOnce({ askPi })   → drainOnce({ askAnalyticalFn? })
  Pi-reflect runOnce({ askPi })   → runOnce({ askAnalyticalFn? })
  bot.js attachCoach/attachReflect no longer pass askPi
  attachTuner() added to bot.js spawn handler
  lessons.source 'pi-coach'   → 'timeweb-coach'
  lessons.source 'pi-reflect' → 'timeweb-reflect'

Token cost measured live:
  ~705 input + 45 output = ~750 total per advisor call
  worst case @ 6 calls/hour rate cap = ~108K tokens/day
  OpenAI gpt-5-mini reference price: ~$0.60/month

Operator usage:
  node scripts/list-improvements.js                # open queue
  node scripts/list-improvements.js --stats        # advisor performance
  node scripts/list-improvements.js --done 17 "shipped in 0.3.1"
  node scripts/list-improvements.js --reject 18 "duplicate"

Tests: 360 green (was 332, +28 new).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-27 19:12:48 +03:00
co-authored by Claude Opus 4.7
parent bc381b2a4b
commit 0d97ccccfa
18 changed files with 1188 additions and 197 deletions
+103
View File
@@ -0,0 +1,103 @@
// Trigger tuner — periodic statistical sanity-check over the
// advisor_recommendations table.
//
// Replaces the old Pi-reflect "analyse your own pattern" loop with a
// deterministic local computation: no LLM call, no subscription, just
// SQL. Every TUNE_INTERVAL_MS the tuner reads the last 24h of
// recommendations, groups by trigger_reason, and flags two failure
// modes as improvement_requests for the operator:
//
// 1. low-success trigger: a trigger that fires often (≥ MIN_SAMPLE)
// but lands a successful outcome < SUCCESS_FLOOR of the time.
// The threshold probably needs tuning, or the prompt isn't giving
// the LLM the right hint.
// 2. expensive trigger: trigger averages > EXPENSIVE_TOKENS input
// tokens but its success rate is mediocre. Could mean the prompt
// includes context the LLM doesn't actually use.
//
// The tuner deduplicates via createImprovementRequest's votes mechanism
// — re-flagging the same gap just bumps the counter, not the row count.
import {
isAvailable as knowledgeAvailable,
recommendationStats,
createImprovementRequest,
} from "../knowledge/index.js";
import { info, warn } from "../log.js";
const TUNE_INTERVAL_MS = 60 * 60 * 1000; // 1 hour
const MIN_SAMPLE = 5;
const SUCCESS_FLOOR = 0.25;
const EXPENSIVE_TOKENS = 1000;
const EXPENSIVE_SUCCESS_CEILING = 0.5;
let _timer = null;
export function attach({ intervalMs = TUNE_INTERVAL_MS } = {}) {
if (_timer) {
warn("tuner", "attach called twice; ignoring");
return;
}
_timer = setInterval(() => {
runOnce().catch((e) => warn("tuner", `tick err: ${e?.message ?? e}`));
}, intervalMs);
_timer.unref?.();
info("tuner", `attached; tune every ${Math.round(intervalMs / 60000)} min`);
}
export function detach() {
if (_timer) clearInterval(_timer);
_timer = null;
}
export function runOnce({ stats = null } = {}) {
if (!knowledgeAvailable()) return { ok: false, reason: "knowledge unavailable" };
const rows = stats ?? recommendationStats({ sinceHours: 24 });
if (!rows.length) return { ok: true, flagged: 0, reason: "no data" };
const flagged = [];
for (const row of rows) {
const sample = (row.applied ?? 0);
if (sample < MIN_SAMPLE) continue;
const succ = row.succeeded ?? 0;
const successRate = sample === 0 ? 0 : succ / sample;
// 1. Low success → tune the trigger
if (successRate < SUCCESS_FLOOR) {
const title = `Trigger "${row.trigger_reason}" has low success rate`;
createImprovementRequest({
source: "tuner",
category: "tuning",
title,
description: `Over the last 24h, ${sample} applied recommendations from trigger ${row.trigger_reason} produced only ${succ} successful outcomes (${(successRate * 100).toFixed(0)}%). Consider tightening the trigger condition, improving the prompt, or adjusting the threshold.`,
context: { stats: row },
priority: 2,
});
flagged.push({ kind: "low_success", trigger: row.trigger_reason, sample, succ });
continue;
}
// 2. Expensive prompt with mediocre payoff
const avgIn = row.avg_in ?? 0;
if (avgIn > EXPENSIVE_TOKENS && successRate < EXPENSIVE_SUCCESS_CEILING) {
const title = `Trigger "${row.trigger_reason}" prompt is expensive`;
createImprovementRequest({
source: "tuner",
category: "tuning",
title,
description: `Trigger ${row.trigger_reason} averages ${Math.round(avgIn)} input tokens but lands successful outcomes only ${(successRate * 100).toFixed(0)}% of the time (${succ}/${sample}). The prompt may include context the model doesn't use — consider trimming.`,
context: { stats: row },
priority: 4,
});
flagged.push({ kind: "expensive_prompt", trigger: row.trigger_reason, avgIn });
}
}
if (flagged.length > 0) {
info("tuner", `flagged ${flagged.length} improvement(s) from ${rows.length} trigger group(s)`);
}
return { ok: true, flagged: flagged.length, items: flagged, groups: rows.length };
}
// Test exports
export const __testing = { TUNE_INTERVAL_MS, MIN_SAMPLE, SUCCESS_FLOOR, EXPENSIVE_TOKENS };