feat(runtime): fully autonomous self-healing — no operator approval #10
+54
-37
@@ -189,53 +189,70 @@ The protocol is intentionally tiny — anyone can write a second client
|
|||||||
(a Telegram bridge, a web UI, a one-shot CLI) by reading
|
(a Telegram bridge, a web UI, a one-shot CLI) by reading
|
||||||
`runtime/ipc-protocol.js`.
|
`runtime/ipc-protocol.js`.
|
||||||
|
|
||||||
## Self-improvement loop
|
## Self-improvement loop (fully autonomous)
|
||||||
|
|
||||||
End-to-end and wired. The flow:
|
End-to-end, no operator-in-the-loop. The bot writes proposals when it
|
||||||
|
spots a *real* bug, applies them with Pi headless, and rolls them back
|
||||||
|
if they break things. The flow:
|
||||||
|
|
||||||
```
|
```
|
||||||
1. reflex chain dispatches an action → action returns { ok: false, detail }
|
1. reflex dispatches action → action returns { ok: false, detail }
|
||||||
2. bot.js failure tracker accumulates the failure under its label
|
2. bot.js failure tracker classifies the detail:
|
||||||
3. same label fails 3× in a row → writeProposal() → markdown lands in
|
bug → TypeError / Cannot read / is not defined …
|
||||||
state/<host>/proposals/<ts>-<slug>.md
|
timeout → "timed out after Ns"
|
||||||
4. next IPC STATUS event includes pendingProposals: N
|
feature-gap → "no reachable log", "no food", "no bed" …
|
||||||
5. TUI shows [proposals N, press y] badge
|
other → anything else
|
||||||
6. operator presses y, reads the proposal, presses y again to approve
|
3. 5 consecutive failures with the SAME label, where the run is dominated
|
||||||
7. proposal moves to state/<host>/proposals/approved/
|
by 'bug' or all 'timeout' → writeProposal()
|
||||||
8. operator runs: npm run propose:apply <filename>
|
(feature gaps are SKIPPED — reflex routing solves those, not the LLM)
|
||||||
9. script verifies clean working tree, creates feat/proposal-<slug>
|
4. runtime/auto-improve.js watcher (poll 2s) sees the new file,
|
||||||
branch, spawns `pi -p` with the proposal + repo-conventions prompt
|
debounces 10s, then spawns scripts/auto-patch.js detached
|
||||||
10. Pi commits a patch on that branch (no push, no merge)
|
5. auto-patch.js:
|
||||||
11. operator reviews diff, runs `npm run bot` to smoke-test
|
- refuses on dirty tree
|
||||||
12. operator pushes the branch and opens a PR by hand
|
- moves proposal pending → approved/ (audit trail)
|
||||||
13. supervisor on the running bot picks up runtime/*.js changes and
|
- creates branch auto/<slug> off main
|
||||||
hot-restarts the child the moment they hit disk
|
- runs `pi -p "<patch prompt>"` with 10-min timeout
|
||||||
|
- if Pi committed AND only touched runtime/ → cherry-pick onto main
|
||||||
|
- else → discard branch, exit non-zero
|
||||||
|
6. supervisor's runtime/*.js watcher fires the moment the cherry-pick
|
||||||
|
lands → child restarts on the new code
|
||||||
|
7. if the new code crashes >5 times in 60s AND the last commit on main
|
||||||
|
is younger than 15 min AND it touched runtime/ → supervisor
|
||||||
|
`git reset --hard HEAD~1` and restarts. Up to MAX_ROLLBACKS times
|
||||||
|
per supervisor lifetime, then bails out for manual investigation.
|
||||||
```
|
```
|
||||||
|
|
||||||
Operator is in the loop at three guardrails: approving the proposal,
|
### Rate limits
|
||||||
reviewing Pi's diff, deciding to merge.
|
|
||||||
|
|
||||||
### Triggers (today)
|
- **Proposal cooldown**: 30 min between proposal files of any kind.
|
||||||
|
- **Auto-improve cooldown**: 15 min between finished `auto-patch.js` runs.
|
||||||
|
- **Hourly cap**: max 4 auto-patches per hour, even if cooldown allows.
|
||||||
|
- **Rollback cap**: 3 rollbacks per supervisor lifetime; after that the
|
||||||
|
supervisor exits and waits for human review.
|
||||||
|
|
||||||
Only one detector is wired: "same labelled action fails 3 times in a
|
### What counts as a bug
|
||||||
row" — for example, three back-to-back `flee from zombie` failures.
|
|
||||||
30 min cooldown so the same proposal doesn't multiply when the bot
|
|
||||||
keeps trying.
|
|
||||||
|
|
||||||
More triggers worth adding (each as a small follow-up):
|
`runtime/bot.js` ships two whitelists (`NORMAL_FAILURE_SUBSTRINGS` and
|
||||||
- "Pi auto-escalation fired but the snapshot didn't change in the next
|
`BUG_FAILURE_SUBSTRINGS`). The proposal trigger fires only when:
|
||||||
N ticks" → bot is fundamentally stuck, propose a code change.
|
- the trailing run of same-label failures contains at least one bug
|
||||||
- "death count >K in M minutes at similar coords" → safety regression.
|
(TypeError / ReferenceError / "Cannot read properties" / etc.),
|
||||||
- "operator typed the same chat command twice and the bot couldn't act"
|
- OR every failure in the run is a timeout (and they happened on the
|
||||||
→ missing operator verb.
|
same operation, so it's probably broken not just unreachable).
|
||||||
|
|
||||||
### Why a manual `propose:apply` step
|
Feature gaps like "no reachable log within 32 blocks" are *not* a bug
|
||||||
|
— the autonomous reflex sees that result, sets `noTreesUntil` and
|
||||||
|
switches to wander. If the script can't solve it via reflex routing,
|
||||||
|
that's a design issue the operator fixes by editing `runtime/reflex.js`
|
||||||
|
directly — not by asking Pi to patch around it.
|
||||||
|
|
||||||
Approval inside the TUI is cheap — one keypress. Spawning Pi to write a
|
### Manual escape hatches
|
||||||
patch is not (subscription tokens, multiple minutes). Splitting "I want
|
|
||||||
this addressed" (TUI) from "now actually run the patcher" (CLI) means
|
These still work but should rarely be needed:
|
||||||
you can approve five proposals over a session and dispatch them in a
|
- TUI hotkey `y` opens the latest pending proposal for inspection.
|
||||||
batch when convenient.
|
- `npm run propose:apply <filename>` runs the *attended* version of the
|
||||||
|
patcher — leaves the result on a `feat/proposal-<slug>` branch
|
||||||
|
without cherry-picking, so the operator can review the diff manually.
|
||||||
|
- `npm run stop` kills everything and clears lock/socket.
|
||||||
|
|
||||||
## File layout
|
## File layout
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,118 @@
|
|||||||
|
// Auto-improve watcher: when a new proposal file appears under
|
||||||
|
// state/<host>/proposals/, debounce briefly (in case the writer is still
|
||||||
|
// finishing), then spawn scripts/auto-patch.js as a detached background
|
||||||
|
// process. The patcher creates a branch, runs Pi headless, and cherry-picks
|
||||||
|
// any resulting commit onto main. The supervisor's runtime/*.js watcher
|
||||||
|
// then triggers a child restart picking up the new code.
|
||||||
|
//
|
||||||
|
// One auto-improve in flight at a time; a 15-minute cooldown between
|
||||||
|
// finished runs caps the Pi token burn rate. Failures keep the proposal
|
||||||
|
// in approved/ (already auto-moved by the patcher) so a future run could
|
||||||
|
// retry — but with the cooldown, this isn't a tight loop.
|
||||||
|
|
||||||
|
import { spawn } from "node:child_process";
|
||||||
|
import fs from "node:fs";
|
||||||
|
import path from "node:path";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
|
||||||
|
import { stateDir } from "./config.js";
|
||||||
|
import { info, warn } from "./log.js";
|
||||||
|
|
||||||
|
const __filename = fileURLToPath(import.meta.url);
|
||||||
|
const __dirname = path.dirname(__filename);
|
||||||
|
const REPO_ROOT = path.resolve(__dirname, "..");
|
||||||
|
const PATCH_SCRIPT = path.join(REPO_ROOT, "scripts", "auto-patch.js");
|
||||||
|
const PROPOSALS_DIR = path.join(stateDir, "proposals");
|
||||||
|
|
||||||
|
const DEBOUNCE_MS = 10_000;
|
||||||
|
const COOLDOWN_MS = 15 * 60 * 1000;
|
||||||
|
const MAX_TOTAL_PER_HOUR = 4;
|
||||||
|
|
||||||
|
let inFlight = false;
|
||||||
|
let lastFinishedAt = 0;
|
||||||
|
const recentRuns = []; // timestamps
|
||||||
|
|
||||||
|
function withinHourlyCap() {
|
||||||
|
const now = Date.now();
|
||||||
|
while (recentRuns.length && now - recentRuns[0] > 3600_000) recentRuns.shift();
|
||||||
|
return recentRuns.length >= MAX_TOTAL_PER_HOUR;
|
||||||
|
}
|
||||||
|
|
||||||
|
function listPendingProposals() {
|
||||||
|
try {
|
||||||
|
return fs
|
||||||
|
.readdirSync(PROPOSALS_DIR)
|
||||||
|
.filter((f) => f.endsWith(".md"))
|
||||||
|
.sort();
|
||||||
|
} catch (e) {
|
||||||
|
if (e.code === "ENOENT") return [];
|
||||||
|
throw e;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function spawnPatcher(filename) {
|
||||||
|
info("auto-improve", `spawning auto-patch for ${filename}`);
|
||||||
|
inFlight = true;
|
||||||
|
const child = spawn(process.execPath, [PATCH_SCRIPT, filename], {
|
||||||
|
cwd: REPO_ROOT,
|
||||||
|
stdio: ["ignore", "pipe", "pipe"],
|
||||||
|
env: { ...process.env },
|
||||||
|
detached: false,
|
||||||
|
});
|
||||||
|
let stdoutBuf = "";
|
||||||
|
let stderrBuf = "";
|
||||||
|
child.stdout.on("data", (c) => {
|
||||||
|
stdoutBuf += c.toString();
|
||||||
|
});
|
||||||
|
child.stderr.on("data", (c) => {
|
||||||
|
stderrBuf += c.toString();
|
||||||
|
});
|
||||||
|
child.on("exit", (code) => {
|
||||||
|
inFlight = false;
|
||||||
|
lastFinishedAt = Date.now();
|
||||||
|
recentRuns.push(lastFinishedAt);
|
||||||
|
const tail = (stdoutBuf + "\n" + stderrBuf).trim().split("\n").slice(-3).join(" | ");
|
||||||
|
if (code === 0) info("auto-improve", `patch applied for ${filename}: ${tail}`);
|
||||||
|
else warn("auto-improve", `patch did not apply (code=${code}) for ${filename}: ${tail}`);
|
||||||
|
// We do NOT trigger supervisor restart manually — the supervisor's
|
||||||
|
// fs.watch on runtime/*.js fires the moment the cherry-pick lands.
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function tick() {
|
||||||
|
if (inFlight) return;
|
||||||
|
if (withinHourlyCap()) return;
|
||||||
|
if (Date.now() - lastFinishedAt < COOLDOWN_MS) return;
|
||||||
|
|
||||||
|
const pending = listPendingProposals();
|
||||||
|
if (pending.length === 0) return;
|
||||||
|
|
||||||
|
// Pick the oldest pending proposal (sorted lexically — timestamps in name).
|
||||||
|
const filename = pending[0];
|
||||||
|
|
||||||
|
// Debounce: ensure the file has been still for DEBOUNCE_MS so we don't
|
||||||
|
// race a partial write. Track first-seen timestamp per filename.
|
||||||
|
if (!debounceMap.has(filename)) {
|
||||||
|
debounceMap.set(filename, Date.now());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const firstSeen = debounceMap.get(filename);
|
||||||
|
if (Date.now() - firstSeen < DEBOUNCE_MS) return;
|
||||||
|
|
||||||
|
debounceMap.delete(filename);
|
||||||
|
spawnPatcher(filename);
|
||||||
|
}
|
||||||
|
|
||||||
|
const debounceMap = new Map();
|
||||||
|
let pollTimer = null;
|
||||||
|
|
||||||
|
export function startAutoImprover() {
|
||||||
|
if (pollTimer) return;
|
||||||
|
info("auto-improve", `watching ${PROPOSALS_DIR} (cooldown=${COOLDOWN_MS / 1000}s, max ${MAX_TOTAL_PER_HOUR}/hr)`);
|
||||||
|
pollTimer = setInterval(tick, 2000);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function stopAutoImprover() {
|
||||||
|
if (pollTimer) clearInterval(pollTimer);
|
||||||
|
pollTimer = null;
|
||||||
|
}
|
||||||
+86
-19
@@ -30,6 +30,7 @@ import {
|
|||||||
readProposal,
|
readProposal,
|
||||||
approveProposal,
|
approveProposal,
|
||||||
} from "./state-store.js";
|
} from "./state-store.js";
|
||||||
|
import { startAutoImprover } from "./auto-improve.js";
|
||||||
|
|
||||||
fs.mkdirSync(stateDir, { recursive: true });
|
fs.mkdirSync(stateDir, { recursive: true });
|
||||||
const JOINED_FLAG = path.join(stateDir, "joined-before.flag");
|
const JOINED_FLAG = path.join(stateDir, "joined-before.flag");
|
||||||
@@ -136,7 +137,9 @@ function dispatchAction(fn, label, opts = {}) {
|
|||||||
}
|
}
|
||||||
reflexCtx.busy = true;
|
reflexCtx.busy = true;
|
||||||
reflexCtx.currentActionLabel = label;
|
reflexCtx.currentActionLabel = label;
|
||||||
writeCurrentTask({ label, status: "in_progress", snapshot: lastSnapshot });
|
// current-task is a resume anchor — keep it small. Embedding the full
|
||||||
|
// perception snapshot blows the file up to ~3 KB per write × every action.
|
||||||
|
writeCurrentTask({ label, status: "in_progress", position: lastSnapshot.position });
|
||||||
info("dispatch", `→ ${label}`);
|
info("dispatch", `→ ${label}`);
|
||||||
Promise.resolve()
|
Promise.resolve()
|
||||||
.then(() => fn())
|
.then(() => fn())
|
||||||
@@ -170,14 +173,56 @@ function dispatchAction(fn, label, opts = {}) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ---- failure tracking + proposal detection --------------------------------
|
// ---- failure tracking + proposal detection --------------------------------
|
||||||
|
//
|
||||||
|
// A proposal is a request to the LLM to patch the codebase. They cost tokens
|
||||||
|
// and may produce risky patches that need rolling back. We file them ONLY for
|
||||||
|
// failures that genuinely look like bugs the reflex layer can't handle on its
|
||||||
|
// own. Everything else is a feature gap the script should solve via reflex
|
||||||
|
// chain reordering, cooldowns, or new primitives.
|
||||||
|
|
||||||
const PROPOSAL_THRESHOLD = 3; // same labelled action fails 3+ times in a row
|
const PROPOSAL_THRESHOLD = 5; // raised from 3 to dampen spam
|
||||||
let lastProposalAt = 0;
|
let lastProposalAt = 0;
|
||||||
const PROPOSAL_COOLDOWN_MS = 30 * 60 * 1000; // don't spam proposal files
|
const PROPOSAL_COOLDOWN_MS = 30 * 60 * 1000;
|
||||||
|
|
||||||
|
// Detail substrings that mean "this is a known feature gap, the bot handles
|
||||||
|
// it via reflex routing already". Don't file a proposal — the bot will switch
|
||||||
|
// strategies on its own. If something here is wrong, fix the routing.
|
||||||
|
const NORMAL_FAILURE_SUBSTRINGS = [
|
||||||
|
"no reachable log",
|
||||||
|
"no log within",
|
||||||
|
"no bed in range",
|
||||||
|
"no food in inventory",
|
||||||
|
"no target in reach",
|
||||||
|
"rate-limited",
|
||||||
|
"can't see you nearby",
|
||||||
|
"returned false",
|
||||||
|
"no result",
|
||||||
|
];
|
||||||
|
|
||||||
|
// Detail substrings that look like a real bug — patch-worthy.
|
||||||
|
const BUG_FAILURE_SUBSTRINGS = [
|
||||||
|
"TypeError",
|
||||||
|
"ReferenceError",
|
||||||
|
"Cannot read properties",
|
||||||
|
"is not a function",
|
||||||
|
"is not iterable",
|
||||||
|
"is not defined",
|
||||||
|
"unknown block",
|
||||||
|
"unknown item",
|
||||||
|
];
|
||||||
|
|
||||||
|
function classifyFailure(detail) {
|
||||||
|
const s = String(detail ?? "");
|
||||||
|
if (BUG_FAILURE_SUBSTRINGS.some((sub) => s.includes(sub))) return "bug";
|
||||||
|
if (NORMAL_FAILURE_SUBSTRINGS.some((sub) => s.includes(sub))) return "feature-gap";
|
||||||
|
if (s.includes("timed out")) return "timeout";
|
||||||
|
return "other";
|
||||||
|
}
|
||||||
|
|
||||||
function recordFailure(label, detail) {
|
function recordFailure(label, detail) {
|
||||||
reflexCtx.recentFailures.push({ ts: Date.now(), label, detail });
|
const kind = classifyFailure(detail);
|
||||||
if (reflexCtx.recentFailures.length > 10) reflexCtx.recentFailures.shift();
|
reflexCtx.recentFailures.push({ ts: Date.now(), label, detail, kind });
|
||||||
|
if (reflexCtx.recentFailures.length > 20) reflexCtx.recentFailures.shift();
|
||||||
maybeFileProposal(label);
|
maybeFileProposal(label);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -186,7 +231,7 @@ function clearRecentFailures(label) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function maybeFileProposal(label) {
|
function maybeFileProposal(label) {
|
||||||
// Count consecutive trailing failures with the same label.
|
// Same-label trailing run.
|
||||||
const trailing = [];
|
const trailing = [];
|
||||||
for (let i = reflexCtx.recentFailures.length - 1; i >= 0; i--) {
|
for (let i = reflexCtx.recentFailures.length - 1; i >= 0; i--) {
|
||||||
const f = reflexCtx.recentFailures[i];
|
const f = reflexCtx.recentFailures[i];
|
||||||
@@ -195,37 +240,58 @@ function maybeFileProposal(label) {
|
|||||||
}
|
}
|
||||||
if (trailing.length < PROPOSAL_THRESHOLD) return;
|
if (trailing.length < PROPOSAL_THRESHOLD) return;
|
||||||
if (Date.now() - lastProposalAt < PROPOSAL_COOLDOWN_MS) return;
|
if (Date.now() - lastProposalAt < PROPOSAL_COOLDOWN_MS) return;
|
||||||
|
|
||||||
|
// Only file when the run is dominated by bug-class failures (any single
|
||||||
|
// bug counts) OR persistent timeouts on the same operation. Feature gaps
|
||||||
|
// are skipped — the reflex layer should re-route, not the LLM.
|
||||||
|
const anyBug = trailing.some((f) => f.kind === "bug");
|
||||||
|
const allTimeout = trailing.every((f) => f.kind === "timeout");
|
||||||
|
if (!anyBug && !allTimeout) return;
|
||||||
|
|
||||||
lastProposalAt = Date.now();
|
lastProposalAt = Date.now();
|
||||||
|
|
||||||
const summary = `${label} failed ${trailing.length}× in a row`;
|
const summary = `${label} failed ${trailing.length}× in a row (${anyBug ? "bug" : "persistent timeout"})`;
|
||||||
|
const slimSnapshot = lastSnapshot && {
|
||||||
|
position: lastSnapshot.position,
|
||||||
|
health: lastSnapshot.health,
|
||||||
|
food: lastSnapshot.food,
|
||||||
|
inventory: lastSnapshot.inventory,
|
||||||
|
isDay: lastSnapshot.isDay,
|
||||||
|
closestHostile: lastSnapshot.closestHostile,
|
||||||
|
dimension: lastSnapshot.dimension,
|
||||||
|
};
|
||||||
const body = [
|
const body = [
|
||||||
`# Repeated failure: ${label}`,
|
`# Repeated failure: ${label}`,
|
||||||
"",
|
"",
|
||||||
|
`Class: **${anyBug ? "bug" : "persistent timeout"}**.`,
|
||||||
|
"",
|
||||||
"## What happened",
|
"## What happened",
|
||||||
"",
|
"",
|
||||||
`The reflex layer dispatched \`${label}\` ${trailing.length} times in succession without a single success.`,
|
`The reflex layer dispatched \`${label}\` ${trailing.length} times in succession without a single success.`,
|
||||||
"",
|
"",
|
||||||
"## Most recent failures",
|
"## Most recent failures",
|
||||||
"",
|
"",
|
||||||
...trailing.slice(0, 5).map(
|
...trailing
|
||||||
(f, i) => `${i + 1}. \`${new Date(f.ts).toISOString()}\` — ${JSON.stringify(f.detail).slice(0, 200)}`,
|
.slice(0, 5)
|
||||||
|
.map(
|
||||||
|
(f, i) =>
|
||||||
|
`${i + 1}. \`${new Date(f.ts).toISOString()}\` [${f.kind}] ${JSON.stringify(f.detail).slice(0, 200)}`,
|
||||||
),
|
),
|
||||||
"",
|
"",
|
||||||
"## Snapshot at moment of last failure",
|
"## Slim snapshot",
|
||||||
"",
|
"",
|
||||||
"```json",
|
"```json",
|
||||||
JSON.stringify(lastSnapshot, null, 2),
|
JSON.stringify(slimSnapshot, null, 2),
|
||||||
"```",
|
"```",
|
||||||
"",
|
"",
|
||||||
"## Suggested next step",
|
"## Constraints for the patch",
|
||||||
"",
|
"",
|
||||||
"Operator: review whether the reflex should:",
|
"- Touch only files under `runtime/`. Don't touch `extensions/`, `tui/`, or any docs.",
|
||||||
"- back off (cooldown extension)",
|
"- Don't introduce new npm dependencies.",
|
||||||
"- switch to a different action variant",
|
"- Don't change `.env` or anything in `state/`.",
|
||||||
"- escalate to Pi for situational reasoning",
|
"- Don't push, don't open a PR. Commit on the current branch only.",
|
||||||
"- or whether the underlying primitive in `runtime/actions.js` needs work.",
|
"- Prefer the smallest viable fix. A 3-line guard is better than a 30-line refactor.",
|
||||||
"",
|
"- If the failure is genuinely irrecoverable (server-side, not code), document it in a code comment and exit 1.",
|
||||||
`Approve this proposal (move to \`proposals/approved/\`) and run \`npm run propose:apply <filename>\` to delegate a patch attempt to Pi headless.`,
|
|
||||||
].join("\n");
|
].join("\n");
|
||||||
|
|
||||||
const { filename } = writeProposal({ kind: `repeated-fail-${label}`, summary, body });
|
const { filename } = writeProposal({ kind: `repeated-fail-${label}`, summary, body });
|
||||||
@@ -591,3 +657,4 @@ ipc = createIpcServer({
|
|||||||
});
|
});
|
||||||
connect();
|
connect();
|
||||||
startTickLoop();
|
startTickLoop();
|
||||||
|
startAutoImprover();
|
||||||
|
|||||||
+52
-1
@@ -11,7 +11,7 @@
|
|||||||
// Run via `npm run bot`. Falls back to plain `node runtime/bot.js` via
|
// Run via `npm run bot`. Falls back to plain `node runtime/bot.js` via
|
||||||
// `npm run bot:bare` if you want to skip the supervisor.
|
// `npm run bot:bare` if you want to skip the supervisor.
|
||||||
|
|
||||||
import { spawn } from "node:child_process";
|
import { spawn, spawnSync } from "node:child_process";
|
||||||
import fs from "node:fs";
|
import fs from "node:fs";
|
||||||
import path from "node:path";
|
import path from "node:path";
|
||||||
import { fileURLToPath } from "node:url";
|
import { fileURLToPath } from "node:url";
|
||||||
@@ -21,11 +21,17 @@ import { stateDir } from "./config.js";
|
|||||||
const __filename = fileURLToPath(import.meta.url);
|
const __filename = fileURLToPath(import.meta.url);
|
||||||
const __dirname = path.dirname(__filename);
|
const __dirname = path.dirname(__filename);
|
||||||
const RUNTIME_DIR = __dirname;
|
const RUNTIME_DIR = __dirname;
|
||||||
|
const REPO_ROOT = path.resolve(RUNTIME_DIR, "..");
|
||||||
const BOT_ENTRY = path.join(RUNTIME_DIR, "bot.js");
|
const BOT_ENTRY = path.join(RUNTIME_DIR, "bot.js");
|
||||||
|
|
||||||
export const RELOAD_EXIT_CODE = 42;
|
export const RELOAD_EXIT_CODE = 42;
|
||||||
const WATCH_DEBOUNCE_MS = 800;
|
const WATCH_DEBOUNCE_MS = 800;
|
||||||
const MAX_RESTARTS_PER_MINUTE = 5;
|
const MAX_RESTARTS_PER_MINUTE = 5;
|
||||||
|
// When a recent auto-patch breaks the bot, we roll it back. "Recent" =
|
||||||
|
// landed on main within the last ROLLBACK_FRESHNESS_MS. The signal is
|
||||||
|
// MAX_RESTARTS_PER_MINUTE exceeded — i.e. the patch reliably crashes.
|
||||||
|
const ROLLBACK_FRESHNESS_MS = 15 * 60_000;
|
||||||
|
const MAX_ROLLBACKS = 3; // hard ceiling per supervisor lifetime
|
||||||
|
|
||||||
// Pidfile prevents two supervisors from racing on the same MC nickname. The
|
// Pidfile prevents two supervisors from racing on the same MC nickname. The
|
||||||
// Minecraft server refuses the second login ("Игрок с данным никнеймом уже
|
// Minecraft server refuses the second login ("Игрок с данным никнеймом уже
|
||||||
@@ -37,6 +43,37 @@ const PID_FILE = path.join(stateDir, "supervisor.pid");
|
|||||||
let child = null;
|
let child = null;
|
||||||
let restartingDueToWatch = false;
|
let restartingDueToWatch = false;
|
||||||
const restartTimestamps = [];
|
const restartTimestamps = [];
|
||||||
|
let rollbackCount = 0;
|
||||||
|
|
||||||
|
function lastCommitAgeMs() {
|
||||||
|
const res = spawnSync("git", ["log", "-1", "--format=%ct", "HEAD"], { cwd: REPO_ROOT, encoding: "utf8" });
|
||||||
|
if (res.status !== 0) return Number.POSITIVE_INFINITY;
|
||||||
|
const epochSec = Number.parseInt(res.stdout.trim(), 10);
|
||||||
|
if (!Number.isFinite(epochSec)) return Number.POSITIVE_INFINITY;
|
||||||
|
return Date.now() - epochSec * 1000;
|
||||||
|
}
|
||||||
|
|
||||||
|
function lastCommitTouchedRuntime() {
|
||||||
|
const res = spawnSync("git", ["diff", "--name-only", "HEAD~1..HEAD"], { cwd: REPO_ROOT, encoding: "utf8" });
|
||||||
|
if (res.status !== 0) return false;
|
||||||
|
return res.stdout.split("\n").some((f) => f.startsWith("runtime/"));
|
||||||
|
}
|
||||||
|
|
||||||
|
function rollbackLastCommit() {
|
||||||
|
const sha = spawnSync("git", ["rev-parse", "HEAD"], { cwd: REPO_ROOT, encoding: "utf8" }).stdout.trim();
|
||||||
|
console.log(`[supervisor] rolling back HEAD (${sha.slice(0, 8)})`);
|
||||||
|
const reset = spawnSync("git", ["reset", "--hard", "HEAD~1"], {
|
||||||
|
cwd: REPO_ROOT,
|
||||||
|
encoding: "utf8",
|
||||||
|
stdio: "inherit",
|
||||||
|
});
|
||||||
|
if (reset.status !== 0) {
|
||||||
|
console.error(`[supervisor] git reset failed — operator intervention needed`);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
rollbackCount++;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
function nowMs() {
|
function nowMs() {
|
||||||
return Date.now();
|
return Date.now();
|
||||||
@@ -97,6 +134,20 @@ function spawnChild() {
|
|||||||
restartTimestamps.push(now);
|
restartTimestamps.push(now);
|
||||||
while (restartTimestamps.length && now - restartTimestamps[0] > 60_000) restartTimestamps.shift();
|
while (restartTimestamps.length && now - restartTimestamps[0] > 60_000) restartTimestamps.shift();
|
||||||
if (restartTimestamps.length > MAX_RESTARTS_PER_MINUTE) {
|
if (restartTimestamps.length > MAX_RESTARTS_PER_MINUTE) {
|
||||||
|
// Crash loop. If the last commit is young AND touched runtime/, it
|
||||||
|
// probably broke us — roll it back and try once more.
|
||||||
|
const ageMs = lastCommitAgeMs();
|
||||||
|
if (
|
||||||
|
ageMs < ROLLBACK_FRESHNESS_MS &&
|
||||||
|
lastCommitTouchedRuntime() &&
|
||||||
|
rollbackCount < MAX_ROLLBACKS &&
|
||||||
|
rollbackLastCommit()
|
||||||
|
) {
|
||||||
|
console.log(`[supervisor] auto-rollback ${rollbackCount}/${MAX_ROLLBACKS} applied; restart counters reset`);
|
||||||
|
restartTimestamps.length = 0;
|
||||||
|
setTimeout(spawnChild, 500);
|
||||||
|
return;
|
||||||
|
}
|
||||||
console.error(`[supervisor] too many restarts (${restartTimestamps.length} in 60s) — giving up`);
|
console.error(`[supervisor] too many restarts (${restartTimestamps.length} in 60s) — giving up`);
|
||||||
process.exit(1);
|
process.exit(1);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,180 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
// auto-patch.js <proposal-filename>
|
||||||
|
//
|
||||||
|
// Unattended sibling of propose-apply.js. Picks an *unapproved* proposal,
|
||||||
|
// moves it to approved/, branches off main, runs `pi -p` headless, and if Pi
|
||||||
|
// commits something — cherry-picks the commit back into main. The point is
|
||||||
|
// to close the self-improvement loop with no operator interaction.
|
||||||
|
//
|
||||||
|
// Exit codes:
|
||||||
|
// 0 patch applied cleanly (commit on main)
|
||||||
|
// 1 pi spawned but produced no commit (no change to repo)
|
||||||
|
// 2 preflight failed (dirty tree, missing proposal, etc.)
|
||||||
|
// 3 pi exited non-zero
|
||||||
|
// 4 cherry-pick conflict — left in unresolved state on a branch
|
||||||
|
//
|
||||||
|
// Designed to be launched by runtime/auto-improve.js as a detached child.
|
||||||
|
// We deliberately avoid touching anything outside the repo and don't push.
|
||||||
|
|
||||||
|
import { spawn, spawnSync } from "node:child_process";
|
||||||
|
import fs from "node:fs";
|
||||||
|
import path from "node:path";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
|
||||||
|
const __filename = fileURLToPath(import.meta.url);
|
||||||
|
const __dirname = path.dirname(__filename);
|
||||||
|
const REPO_ROOT = path.resolve(__dirname, "..");
|
||||||
|
|
||||||
|
function log(level, msg) {
|
||||||
|
const line = `${new Date().toISOString()} [auto-patch] [${level}] ${msg}`;
|
||||||
|
if (level === "error" || level === "warn") console.error(line);
|
||||||
|
else console.log(line);
|
||||||
|
}
|
||||||
|
|
||||||
|
function git(args, opts = {}) {
|
||||||
|
return spawnSync("git", args, { cwd: REPO_ROOT, encoding: "utf8", ...opts });
|
||||||
|
}
|
||||||
|
|
||||||
|
function exit(code, reason) {
|
||||||
|
log(code === 0 ? "info" : "warn", `exit ${code}: ${reason}`);
|
||||||
|
process.exit(code);
|
||||||
|
}
|
||||||
|
|
||||||
|
const filenameArg = process.argv[2];
|
||||||
|
if (!filenameArg) exit(2, "usage: auto-patch.js <proposal-filename>");
|
||||||
|
|
||||||
|
function findProposal(filename) {
|
||||||
|
const stateRoot = path.join(REPO_ROOT, "state");
|
||||||
|
if (!fs.existsSync(stateRoot)) return null;
|
||||||
|
for (const host of fs.readdirSync(stateRoot)) {
|
||||||
|
const pending = path.join(stateRoot, host, "proposals", filename);
|
||||||
|
const approved = path.join(stateRoot, host, "proposals", "approved", filename);
|
||||||
|
if (fs.existsSync(pending)) return { path: pending, host, status: "pending" };
|
||||||
|
if (fs.existsSync(approved)) return { path: approved, host, status: "approved" };
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const proposal = findProposal(filenameArg);
|
||||||
|
if (!proposal) exit(2, `proposal not found: ${filenameArg}`);
|
||||||
|
|
||||||
|
// Refuse on dirty tree — we'd lose the operator's WIP.
|
||||||
|
const dirty = git(["status", "--porcelain"]).stdout.trim();
|
||||||
|
if (dirty) exit(2, `working tree dirty: ${dirty.split("\n")[0]}`);
|
||||||
|
|
||||||
|
// Move pending → approved so we don't try to apply the same proposal twice.
|
||||||
|
if (proposal.status === "pending") {
|
||||||
|
const approvedDir = path.join(path.dirname(proposal.path), "approved");
|
||||||
|
fs.mkdirSync(approvedDir, { recursive: true });
|
||||||
|
const dst = path.join(approvedDir, filenameArg);
|
||||||
|
const content = fs.readFileSync(proposal.path, "utf8").replace(/^approved: false/m, "approved: true (auto)");
|
||||||
|
fs.writeFileSync(dst, content);
|
||||||
|
fs.unlinkSync(proposal.path);
|
||||||
|
proposal.path = dst;
|
||||||
|
log("info", `moved pending → approved: ${filenameArg}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const proposalText = fs.readFileSync(proposal.path, "utf8");
|
||||||
|
|
||||||
|
// Capture current main HEAD so we can roll back to it if cherry-pick fails.
|
||||||
|
const baseSha = git(["rev-parse", "HEAD"]).stdout.trim();
|
||||||
|
|
||||||
|
const slug = filenameArg
|
||||||
|
.replace(/\.md$/, "")
|
||||||
|
.replace(/[^a-zA-Z0-9-]+/g, "-")
|
||||||
|
.slice(0, 60);
|
||||||
|
const branch = `auto/${slug}`;
|
||||||
|
|
||||||
|
// Delete the branch if it exists from a previous failed attempt.
|
||||||
|
git(["branch", "-D", branch]); // ignore error if absent
|
||||||
|
|
||||||
|
const checkout = git(["checkout", "-b", branch]);
|
||||||
|
if (checkout.status !== 0) exit(2, `cannot create branch ${branch}: ${checkout.stderr}`);
|
||||||
|
|
||||||
|
const prompt = [
|
||||||
|
"You are patching the pepa-pi-bot repo to address an automatically-detected failure.",
|
||||||
|
"This is an UNATTENDED run — no operator will review your output before it lands on main.",
|
||||||
|
"Be conservative. Prefer guard clauses and small surgical edits.",
|
||||||
|
"",
|
||||||
|
"## The proposal",
|
||||||
|
"",
|
||||||
|
proposalText,
|
||||||
|
"",
|
||||||
|
"## Hard rules (non-negotiable)",
|
||||||
|
"",
|
||||||
|
"1. Touch only files under `runtime/`. Do NOT modify `tui/`, `extensions/`, `scripts/`, `docs/`, `package.json`, or anything in `state/`.",
|
||||||
|
"2. Do not introduce npm dependencies.",
|
||||||
|
"3. Do not push, do not open a PR. Commit on the current branch only.",
|
||||||
|
"4. Use a conventional commit message: `fix(runtime/<file>): <one-line summary>`.",
|
||||||
|
"5. If you can't safely fix the issue, write a short comment in the relevant runtime file explaining why and stop — do NOT make a speculative change.",
|
||||||
|
"6. Make exactly ONE commit. If you find multiple issues, focus on the one the proposal describes.",
|
||||||
|
"",
|
||||||
|
"After you commit, your job is done.",
|
||||||
|
].join("\n");
|
||||||
|
|
||||||
|
log("info", `spawning pi -p (timeout 10 min)`);
|
||||||
|
const pi = spawn("pi", ["-p", prompt], {
|
||||||
|
cwd: REPO_ROOT,
|
||||||
|
stdio: ["ignore", "pipe", "pipe"],
|
||||||
|
});
|
||||||
|
|
||||||
|
let piStdout = "";
|
||||||
|
let piStderr = "";
|
||||||
|
pi.stdout.on("data", (chunk) => {
|
||||||
|
piStdout += chunk.toString();
|
||||||
|
});
|
||||||
|
pi.stderr.on("data", (chunk) => {
|
||||||
|
piStderr += chunk.toString();
|
||||||
|
});
|
||||||
|
|
||||||
|
const PI_TIMEOUT_MS = 10 * 60 * 1000;
|
||||||
|
const timer = setTimeout(() => {
|
||||||
|
log("warn", "pi timeout — killing subprocess");
|
||||||
|
pi.kill("SIGTERM");
|
||||||
|
}, PI_TIMEOUT_MS);
|
||||||
|
|
||||||
|
pi.on("exit", (code) => {
|
||||||
|
clearTimeout(timer);
|
||||||
|
log("info", `pi exited code=${code}; stdout=${piStdout.length}B stderr=${piStderr.length}B`);
|
||||||
|
|
||||||
|
if (code !== 0) {
|
||||||
|
// Pi crashed or timed out — return to main, drop the branch.
|
||||||
|
git(["checkout", "main"]);
|
||||||
|
git(["branch", "-D", branch]);
|
||||||
|
exit(3, `pi exited ${code}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const newHead = git(["rev-parse", "HEAD"]).stdout.trim();
|
||||||
|
if (newHead === baseSha) {
|
||||||
|
// Pi did not commit anything. Clean up.
|
||||||
|
git(["checkout", "main"]);
|
||||||
|
git(["branch", "-D", branch]);
|
||||||
|
exit(1, "pi made no commit");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify the commit touched only runtime/.
|
||||||
|
const filesChanged = git(["diff", "--name-only", `${baseSha}..HEAD`]).stdout.trim().split("\n").filter(Boolean);
|
||||||
|
const outside = filesChanged.filter((f) => !f.startsWith("runtime/"));
|
||||||
|
if (outside.length > 0) {
|
||||||
|
log("error", `commit touched files outside runtime/: ${outside.join(", ")} — discarding`);
|
||||||
|
git(["checkout", "main"]);
|
||||||
|
git(["branch", "-D", branch]);
|
||||||
|
exit(2, "patch touched off-limits files");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cherry-pick onto main.
|
||||||
|
git(["checkout", "main"]);
|
||||||
|
const cherry = git(["cherry-pick", newHead]);
|
||||||
|
if (cherry.status !== 0) {
|
||||||
|
log("error", `cherry-pick failed: ${cherry.stderr}`);
|
||||||
|
// Leave the branch around for operator inspection; abort the failed
|
||||||
|
// cherry-pick so main is clean.
|
||||||
|
git(["cherry-pick", "--abort"]);
|
||||||
|
exit(4, `cherry-pick conflict — see branch ${branch}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Success — delete the feature branch (the commit is on main now).
|
||||||
|
git(["branch", "-D", branch]);
|
||||||
|
log("info", `patch applied to main as ${git(["rev-parse", "HEAD"]).stdout.trim().slice(0, 8)}`);
|
||||||
|
exit(0, `applied ${filenameArg}`);
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user