daemon-runtime

Imported from kunchenguid/no-mistakes (.agents/skills/daemon-runtime/SKILL.md). Install upstream with npx skills add kunchenguid/no-mistakes --skill daemon-runtime. Copyright stays with the author.

Daemon Singleton Lock (internal/daemon/lock.go)

• Only one live daemon may own an NM_HOME: an exclusive OS file lock on <NM_HOME>/daemon.lock is acquired as the very first action in RunWithOptions, strictly before stale-run recovery and socket bind, and held for the process lifetime. The kernel releases it on any process death, so a held lock always means a live holder and no staleness heuristic is needed. Without it, a second daemon stole the socket and ran global crash recovery against the live daemon's runs and worktrees.
• Process launch is not readiness: the PID record is published after the singleton lock and before exclusive recovery, while startup succeeds only after a real IPC health response. The 45s production budget covers cold environment setup and recovery; early exits fail promptly, timeout cleanup reaps detached children before fallback or rollback, and managed plus detached failures retain both causes. Regressions: TestStartDetachedDaemonDetectsChildExitPromptly, TestStartDetachedDaemonTimeoutKillsAndReapsChild, TestStartPreservesManagedAndDetachedFallbackErrors, TestColdDetachedStartupProductionGateCardinality.
• A successful stop means the daemon process is gone, not merely that IPC health has disappeared, because only process exit releases the singleton lock. Capture the daemon instance before requesting shutdown, and close the shutdown client before waiting because the daemon drains in-flight handlers during exit. See waitForDaemonStop and stopDetachedDaemon; regressions: e2e TestDaemonStopLeavesNoDaemonProcessOwningTheRoot, TestDaemonRestartReplacesTheDaemonWithExactlyOneOwner.
• Independent layers: internal/ipc listen() dials the socket before unlinking it and refuses to steal a live one; client probes bound the dial with daemon_connect_timeout and fail fast on a dead or wedged socket instead of starting a replacement daemon (EnsureDaemon surfaces the error with a daemon start recovery hint; the health RPC itself is bounded separately by ipc.DefaultDialTimeout). Exception: a refused socket whose PID file names a gone or start-time-mismatched daemon is cleared and reported not running by clearDeadDaemonArtifacts, which inspects the PID before taking the singleton lock and holds it only for the PID-file re-check and removal; acquireSingletonLock retries for singletonLockRetry (250ms) so that short hold never makes a starting daemon exit, while a live daemon still fails it. update snapshots whether daemon artifacts existed before its first probe and restarts the daemon even after that probe cleared them.
• Daemon execution is explicit-only (no-mistakes daemon run --root); never let inherited environment reinterpret probes like --version or status as daemon workers.
• Startup worktree cleanup is DB-aware: never remove a worktree whose run row is pending or running; startRun inserts the run row before creating the worktree, so a no-row directory is safe to remove immediately. That no-row rule holds only inside <NM_HOME>/worktrees, which is discovered by walking because no-mistakes owns it; a configured worktree root is the operator's directory, so cleanup and eject there act on exactly the recorded run worktrees and never enumerate anything else.
• The user-facing model lives in docs/src/content/docs/concepts/daemon.md; the lock rationale lives in the internal/daemon/lock.go and daemon.go comments. Regressions: TestAcquireSingletonLock_*, TestRunWithResources_SecondDaemonForSameRootFailsWithoutStealingSocket, TestRunWithOptions_RequiresSingletonLockBeforeRecovery, TestRecoverOnStartup_DoesNotDeleteActiveRunWorktree, TestServe_SecondListenerForLiveSocketDoesNotStealIt, TestDialConnectTimeoutFailsFastAndNamesSocket, TestIsRunningFailsFastWhenSocketAcceptsButDoesNotRespond, TestIsRunningSurfacesExistingDeadSocket, TestIsRunningRefusedSocketWithGoneDaemonIsNotRunning, TestIsRunningRefusedSocketWithLiveOrUnverifiableDaemonStillRefuses, TestIsRunningStaleProbeDoesNotBlockAStartingDaemon, TestIsRunningStaleProbeKeepsArtifactsOfADaemonThatJustStarted, TestDaemonRunRootFromArgs_EnvDoesNotForceDaemonModeForProbes, TestValidateDaemonPIDFallback_RefusesToKillOwnProcess.

Bounded Daemon Logging and Event-Driven AXI Runs

• internal/logstore owns all daemon-process byte and retention bounds. Lifecycle output uses logs/daemon.log, managed Rovo Dev/OpenCode stdout and stderr use logs/managed-server.log, and service bootstrap/direct crash output uses logs/daemon-bootstrap.log. Rotation snapshots backups and truncates the current inode in place so held service and child descriptors keep writing to the bounded current file. Regressions: internal/logstore/rotate_test.go, TestDetachedDaemonUsesBoundedDedicatedLogSinks, TestManagedServerOutputIsSeparatedFromLifecycleFailureSummary.
• Successful read-only IPC methods are DEBUG; mutations and stream starts are INFO; every request failure is WARN. AXI run driving is subscribe-first and internal/cli/run_reconciler.go is the sole owner of event reconciliation, reconnect, duplicate-event coalescing, and the slow lost-event heartbeat. Do not reintroduce fixed-interval get_run polling. A pre-drive get_active_run or get_run state read that misses its per-attempt deadline is a slow reply, not a dead daemon: classify the timeout, health-probe, and retry. axi run/axi respond default --wait 8m independently so the hold cannot sit on a 10-minute harness cap, and subscription acknowledgement must honor that context. Regressions: TestSuccessfulReadRequestsDoNotLogAtInfo, TestRequestLoggingKeepsMutationsAndFailuresVisible, TestDriveRun_HealthyWaitStaysWithinRequestBudget, TestDriveRun_SlowGetRunRetriesAfterHealthProbe, TestAxiRun_SlowActiveRunReadRetriesInsteadOfStartingAnotherRun, TestAxiRespond_SlowInitialRunReadRetries, TestAxiRun_WaitInterruptsSubscriptionAcknowledgement, TestAxiRun_WaitElapsedAgainstLiveIdleDaemon, TestRunReconciler_*.

Bounded Loss-Aware Event Subscriptions

• internal/ipc/events.go (ClassOf) is the single event taxonomy: activity is droppable, state is not, control is broker-generated, and an unrecognized type fails safe to state. Brokers and consumers must read loss tolerance from it rather than re-listing event names.
• internal/daemon/eventmailbox.go is the single overflow owner: a per-subscriber ring bounded by 64 events and 1 MiB, non-blocking publish (the executor is never stalled), activity as the only evictable class, and everything else folded into one sticky coalescing stream_gap that drains ahead of queued payload. A reserved slot is not enough - it fails at the second simultaneous transition - and producer-side channel receives race the reader, which is why the queue is a ring under a mutex.
• Every state event and every get_run snapshot carries a monotonic StateRev; runSnapshot samples it before the DB read, which is sound only because every producer writes state and then emits. Consumers apply a delta only when its revision is newer, so a delta queued before a snapshot cannot regress state after it. Every subscription opens gapped, so attach and reconnect always reconcile first.
• The fix-review working-tree diff is the only gate context that is never persisted, so it is served on demand by ipc.MethodGetStepDiff (RunManager.StepDiff, bounded at 512 KiB) instead of riding the stream: it was the only unbounded payload, and one frame past the 1 MiB transport line limit ends the subscription and hides every later event.
• Regressions: internal/daemon/eventmailbox_test.go (A1-A13 plus the byte/count ceilings), TestRunSnapshot_*, TestStepDiff_*, TestExecutor_StateEventsAreEmittedAfterTheirDatabaseWrite, TestClassOfUnknownEventFailsSafeToState, TestRunReconciler_StreamGapForcesOneAuthoritativeRead, TestSubscribeOversizedFrameEndsTheStreamAndHidesLaterEvents, internal/tui/overflow_contract_test.go.

Destructive Daemon Lifecycle Guard (internal/lifecycle/guard.go)

• daemon stop, daemon restart, daemon uninstall, update, and a daemon start that would restart a running daemon on service-definition drift (daemon.StartGuarded's beforeRestart, run before the definition is rewritten) refuse by default while pending/running runs exist (the daemon is machine-wide, so stopping it can fail every active pipeline), list the runs via the shared lifecycle.ActiveRuns/lifecycle.RunList helpers, and require an explicit --force. update -y answers only the different-executable prompt and deliberately does not bypass this guard.
• Every invocation of these commands (for daemon start, only one that reaches the restart guard) is logged with caller attribution (PID, PPID, parent command line) via logLifecycleInvocation to <NM_HOME>/logs/cli.log; this is the incident forensic trail, do not remove or weaken it.
• Regressions: TestDaemonStopRefusesWithActiveRunsAndListsThem, TestDaemonStopForceOverridesActiveRunGuard, TestDaemonRestartRefusesWithActiveRuns, TestDaemonStartRefusesToRestartForDriftWithActiveRuns, TestStartGuarded_RefusalLeavesDriftedServiceAndDaemonAlone, TestLifecycleCommandsWriteCallerAttributionToCLILog (internal/cli/daemon_lifecycle_test.go), TestUpdaterRunRefusesWithActiveRunsAndListsThem, TestUpdaterActiveRunGuardAllowsForce (internal/update).

Daemon Login-Shell Environment (internal/shellenv)

• The managed service definition's PATH (managedServicePath, internal/daemon/service_launchd.go / service_systemd.go) is the deterministic bootstrap list, never a probe result; the effective environment comes from shellenv.ApplyToProcessWithShellRetryExcept at daemon startup (prepareDaemonEnvironment) and is what every step and agent subprocess inherits. Changing the bootstrap list does not supply version-manager directories; the startup probe determines the effective PATH.
• The probe ($SHELL -l -i -c 'env -0') runs in its own session (detachFromTerminal, Setsid): a Setpgid child of a terminal's foreground process is stopped by SIGTTIN as soon as interactive zsh initializes job control, so a foreground daemon run used to burn the whole timeout and degrade. Do not fold this back into ConfigureShellCommand; Setpgid and Setsid cannot be combined.
• Only a missing shell binary (shellBinaryMissing: fork/exec ENOENT or an unresolvable name) is waited for, and only at startup (DefaultShellRetryWindow): on macOS with nix-darwin the login shell lives under /run/current-system, which the activate-system daemon recreates after the Nix store mounts, so a RunAtLoad launch agent can race it (#143). A shell that exists but fails is never waited for. The environment is resolved once per daemon lifetime; restart the daemon to pick up a login shell that appeared or changed later.
• Regressions: internal/shellenv/shellenv_retry_test.go (incl. TestDefaultShellCommandOutput_InteractiveShellFromForegroundTerminal, which re-executes the test binary under script(1) to own a pty).