fix: overhaul container lifecycle — recovery, health, uninstall, UI state

Container recovery:
- Health monitor: MAX_RESTART_ATTEMPTS 3→10, interval 60s→120s
- Dependency-aware restarts: won't restart services before their deps
- Reset dependent counters when a dependency recovers
- Handle "created" state containers (were invisible to health monitor)
- Added IndeedHub, mempool-api, mysql to tier system
- Crash recovery: podman start timeout 30s→120s with retry
- Podman client: socket timeout 5s→30s, added restart policy

UI state representation:
- Exit code 0 shows "stopped" (gray), not "crashed" (red)
- Exit code 137 shows "killed (OOM)"
- Non-zero exit shows "crashed" (red)
- Added exit_code field to PackageDataEntry

Install/uninstall fixes:
- Install returns error when container doesn't start (was silent success)
- Post-install hooks awaited instead of fire-and-forget tokio::spawn
- Uninstall: graceful rm before force, volume prune, network cleanup
- Uninstall returns error on partial failure (was 200 OK)

Config consistency:
- DB passwords read from /var/lib/archipelago/secrets/ (was hardcoded)
- Bitcoin: added ZMQ ports 28332/28333 for LND block notifications
- IndeedHub port 7777→8190 (was conflicting with strfry)
- Marketplace versions: LND 0.17.4→0.18.4, Mempool 2.5.0→3.0.0

Performance:
- Metrics collector interval 60s→300s (was duplicating health monitor)
- Podman client: proper error propagation instead of unwrap_or_default

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

.claude/memory/MEMORY.md

@@ -34,6 +34,9 @@
 ## Deploy & Container Fixes
 - [project_deploy_session_2026_03_22.md](project_deploy_session_2026_03_22.md) — Fleet deploy fixes: credential mismatches, restart storms, rootless port 80, deploy script hardening
 
+## Gamepad Navigation
+- [project_gamepad_nav.md](project_gamepad_nav.md) — Controller nav system, key files, patterns, Chromium gotchas
+
 ## Completed Work
 - [project_mesh_198_issue.md](project_mesh_198_issue.md) — Mesh .198: 3 bugs fixed and deployed
 - [project_indeedhub_arch3_fix.md](project_indeedhub_arch3_fix.md) — IndeedHub Arch 3: corrupted combined tarball fixed

.claude/memory/feedback_asset_workflow.md

@@ -0,0 +1,16 @@
+---
+name: Asset workflow - designer makes images
+description: User is a designer — never generate PNG/JPEG/SVG assets, only provide specs. TUI/text animations are Claude's job.
+type: feedback
+---
+
+Never generate PNG, JPEG, or SVG image assets. The user is a designer and will always create these manually.
+
+**Claude's job:** TUI text, animations, shell scripts, code
+**User's job:** PNG, JPEG, SVG, any visual/graphic assets
+
+When images are needed, provide clear specs (dimensions, format, constraints, where they go) and let the user create them.
+
+**Why:** User is a professional designer. Auto-generated pixel art looks generic compared to their actual brand artwork.
+
+**How to apply:** When boot splash, logos, icons, or any visual assets are needed, output a spec sheet with dimensions/format/constraints. Never run image generation scripts as part of the build.

.claude/memory/feedback_deploy_patterns.md

@@ -0,0 +1,19 @@
+---
+name: Deploy container patterns
+description: Hard-won deploy patterns — rootless port 80, credential sync, health checks, image export
+type: feedback
+---
+
+Container deploy patterns learned from fleet-wide deploy sessions.
+
+**Rootless port 80:** Containers binding port 80 MUST use `--user 0:0`. `NET_BIND_SERVICE` cap doesn't work in rootless Podman.
+
+**Why:** Discovered across multiple containers (FileBrowser, Nextcloud, Vaultwarden, Jellyfin) that `--cap-add NET_BIND_SERVICE` is silently ignored in rootless mode. Only `--user 0:0` works.
+
+**Credential sync:** MariaDB/Postgres only read env vars on FIRST init. If deploy generates new random passwords in `secrets/` but the DB data dir already exists, the DB keeps the OLD password. Fix: either wipe data dir + reinit, or `ALTER USER` to sync.
+
+**Image export:** Always export custom images as INDIVIDUAL tarballs (`podman save -o name.tar`). Combined tarballs corrupt image IDs.
+
+**Health checks:** Every container should have `--health-cmd`. Currently 25+ containers have them.
+
+**How to apply:** Check these patterns in any deploy script changes or new container additions.

.claude/memory/feedback_gamepad_unfinished.md

@@ -0,0 +1,12 @@
+---
+name: Gamepad navigation unfinished
+description: Gamepad/controller nav rewrite (aada1975) shipped but has issues — needs further work
+type: feedback
+---
+
+Gamepad navigation rewrite was committed (aada1975) and included in CI ISO builds, but user reports it's not working correctly. Issues:
+- Can't exit input fields with up/down arrow keys when other elements are available
+- Navigation behavior not right (unspecified details — need to investigate)
+
+**Why:** The rewrite was a major change to `useControllerNav.ts` and focus management. Shipped in beta but needs polish.
+**How to apply:** When touching gamepad/controller nav, treat as unfinished work. Test arrow key behavior on inputs, focus trap logic, and spatial navigation thoroughly.

.claude/memory/feedback_logo_ascii.md

@@ -0,0 +1,19 @@
+---
+name: Archipelago ASCII logo — never change
+description: The block-letter ASCII art logo for Archipelago is locked in. Use this exact design everywhere.
+type: feedback
+---
+
+The Archipelago ASCII block-letter logo is finalized. Never change it.
+
+```
+█▀█ █▀▄ █▀▀ █ █ █ █▀█ █▀▀ █   █▀█ █▀▀ █▀█
+█▀█ █▀▄ █   █▀█ █ █▀▀ ██▀ █   █▀█ █ █ █ █
+▀ ▀ ▀ ▀ ▀▀▀ ▀ ▀ ▀ ▀   ▀▀▀ ▀▀▀ ▀ ▀ ▀▀▀ ▀▀▀
+```
+
+Uses ▀ ▄ █ block characters. 45 chars wide, fits in any 52+ col box.
+Render in Bitcoin orange (`\033[38;5;208m`) by default.
+
+**Why:** User explicitly approved this logo and said "save that never change."
+**How to apply:** Use this for all TUI contexts — install screens, MOTD, menu banners, boot displays. Replace the old spaced-out `a r c h i p e l a g o` text with this wherever a banner is needed.

.claude/memory/project_app_registry.md

@@ -0,0 +1,19 @@
+---
+name: App Registry Setup
+description: Archipelago app container registry at 80.71.235.15:3000 (Gitea) — marketplace images mirrored there
+type: project
+---
+
+Archipelago app registry running on Gitea at `80.71.235.15:3000`, org `archipelago`.
+
+**Why:** Self-hosted container registry so Archipelago nodes pull app images from our infrastructure instead of Docker Hub/ghcr.io. Critical for unbundled ISO installs where apps are downloaded on-demand.
+
+**How to apply:**
+- Registry URL: `80.71.235.15:3000/archipelago/<app>:<version>`
+- HTTP only (insecure) — nodes need `registries.conf` with `insecure = true`
+- ISO build bakes the insecure registry config into `/home/archipelago/.config/containers/registries.conf`
+- Marketplace data in `neode-ui/src/views/marketplace/marketplaceData.ts` uses `REGISTRY` constant
+- 34 images pushed from .228 on 2026-03-26
+- NOT pushed yet: Thunderhub, Penpot (not on .228)
+- Gitea instance deployed via Portainer on `80.71.235.15:9443`
+- Login: podman login 80.71.235.15:3000 (credentials set up on .228)

.claude/memory/project_cicd_setup.md

@@ -0,0 +1,20 @@
+---
+name: CI/CD Setup
+description: Gitea Actions CI/CD — runner on .228, workflow builds unbundled ISO on push to main
+type: project
+---
+
+CI/CD pipeline using Gitea Actions on git.tx1138.com.
+
+**Why:** Automatic ISO builds on every push to main. ISOs copied to FileBrowser /Builds/ for download.
+
+**How to apply:**
+- Gitea repo: `git.tx1138.com/lfg2025/archy`
+- Runner: .228 registered as `archipelago-builder` with label `ubuntu-latest:host`
+- Runner service: `gitea-runner.service` (systemd, runs as archipelago user)
+- Runner config: `~/.runner` on .228
+- Workflow: `.gitea/workflows/build-iso.yml` — unbundled ISO only
+- Uses `https://git.tx1138.com/actions/checkout@v4` (NOT github.com actions)
+- Builds: backend (cargo), frontend (npm), then ISO with `UNBUNDLED=1`
+- Output: copied to `/var/lib/archipelago/filebrowser/Builds/`
+- act_runner v0.2.11 installed at `/usr/local/bin/act_runner`

.claude/memory/project_container_orchestration.md

@@ -0,0 +1,20 @@
+---
+name: Container Orchestration Hardening
+description: Container orchestration overhaul — stop grace periods, pull retry, persistent restart tracking, scheduled remediation, failsafe install, boot reconciliation
+type: project
+---
+
+Container orchestration hardening implemented on dev-iso branch (2026-03-28).
+
+**Why:** Gitea issue requesting true orchestration. Containers were unreliable — 10s stop timeout risked Bitcoin Core UTXO corruption, image pulls failed silently, restart counters reset on process restart enabling infinite loops, doctor/reconcile scripts only ran manually.
+
+**What was done (7 changes):**
+1. Per-container stop grace periods (600s bitcoin, 330s lnd, 300s electrs, 120s databases, 60s btcpay, 30s default) + systemd TimeoutStopSec=660
+2. Image pull retry with exponential backoff (3 attempts: 5s/15s/45s) + post-pull verification + stacks.rs error propagation instead of silent swallow
+3. Resolved container/health_monitor.rs TODO (documented as orchestrator-level responsibility)
+4. Persistent restart tracking to restart-tracker.json (survives process restarts, seeded on startup)
+5. Scheduled systemd timers: container-doctor every 30min, reconcile-containers every 6h
+6. Failsafe install: post-pull image verify, rollback on start failure, 30s post-start health check with crash diagnosis
+7. Boot reconciliation: runs reconcile-containers.sh after crash recovery completes
+
+**How to apply:** These changes affect beta reliability. The other programmer is working on custom base ISO on the same branch — coordinate on build-auto-installer-iso.sh changes.

.claude/memory/project_gamepad_nav.md

@@ -0,0 +1,22 @@
+---
+name: Gamepad Navigation System
+description: Controller/gamepad navigation architecture, key decisions, known issues, and the nav map doc location
+type: project
+---
+
+Gamepad/controller navigation is a core feature of Archipelago — the UI runs on a kiosk with Xbox-style controller input.
+
+**Why:** Archipelago runs on dedicated hardware with a TV/monitor + gamepad. Every page must be fully navigable without a mouse.
+
+**How to apply:** When modifying any page's interactive elements, check that `data-controller-container` and `tabindex` are set correctly. Read `neode-ui/docs/GAMEPAD-NAV-MAP.md` for the full per-page navigation spec and implementation notes.
+
+## Key files
+- `neode-ui/src/composables/useControllerNav.ts` — all navigation logic
+- `neode-ui/docs/GAMEPAD-NAV-MAP.md` — full nav spec with per-page tables, implementation notes, and Chromium gotchas
+
+## Critical patterns
+- Cards on grid pages: `glass-card transition-all hover:-translate-y-1` + `data-controller-container tabindex="0"`
+- Settings page is a MIXED page (containers + standalone buttons) — nav searches both together
+- ToggleSwitch has `tabindex="-1"` + `data-controller-ignore` so gamepad skips it
+- Focus glow uses blurred box-shadow, NOT `0 0 0 Npx` spread (Chromium compositor bug with translateZ(0))
+- `outline: none !important` on all containers to kill browser default focus rings

.claude/memory/project_iso_size_reduction.md

@@ -0,0 +1,29 @@
+---
+name: ISO Size Reduction Plan
+description: Plan to reduce ISO from 3.9GB — prioritized phases for post-beta
+type: project
+---
+
+Current ISO: ~3.9GB (unbundled). Target: <1.5GB.
+
+**Why:** Debian Live base (~800MB) + rootfs with kiosk/Podman/firmware (~2.1GB) + squashfs overhead.
+
+**Phase 1 — Quick wins (post-beta, ~500MB-1GB savings):**
+- Strip unused firmware blobs (WiFi chipsets, GPU)
+- Remove build-only packages from rootfs (not needed at runtime)
+- `--no-install-recommends` in all apt installs
+- Strip debug symbols from binaries
+- Remove man pages, docs, locale data (`localepurge`)
+
+**Phase 2 — Minimal base (~1-1.5GB savings):**
+- Replace Debian Live ISO with custom `debootstrap --variant=minbase` live image
+- Make kiosk (X11 + Chromium ~400MB) optional / separate overlay
+- Alpine-based rootfs alternative
+
+**Phase 3 — Long term (<1GB target):**
+- Custom kernel with only needed modules
+- A/B read-only root partition (no live boot infrastructure)
+- Network installer variant (tiny ISO, needs internet)
+- Reproducible builds with exact dep trees
+
+**How to apply:** Each phase is independent. Phase 1 is safe to do anytime. Phase 2 requires testing the boot chain. Phase 3 is architectural.

.claude/memory/project_session_20260328.md

@@ -0,0 +1,42 @@
+---
+name: ISO Session 2026-03-28 Handoff
+description: Session handoff — branding overhaul, ISOLINUX config updated, terminal banners redesigned, UEFI still broken
+type: project
+---
+
+## Session State (2026-03-28 ~latest)
+
+### Branding Overhaul (this session)
+
+**ISOLINUX boot menu:**
+- Config updated: menu centered (HSHIFT 28, WIDTH 26), title "Bitcoin Node OS"
+- Selection: white on dark, hotkeys in Bitcoin orange (#fb923c)
+- Tab message: "Press TAB to edit | https://archipelago.sh"
+- MENU RESOLUTION kept at 1024x768 (uses GRUB background.png)
+- Three options: Install Archipelago, Install (verbose), Boot from local disk
+
+**Terminal banners — unified design across all screens:**
+- Name: "A R C H I P E L A G O" (uppercase, spaced, bold white)
+- Separator: orange line
+- Subtitle: dim text (varies by context)
+- Colors: basic ANSI (works on bare-metal console, not 256-color)
+- Width: fits 80-col terminals (no overflow/clipping)
+- Build script auto-install.sh: centered + adaptive-width boxes
+- Standalone scripts: fixed 52-char boxes
+
+**Files changed:**
+- build-auto-installer-iso.sh: ISOLINUX config, colors (256 to basic ANSI), case, header + completion
+- build/debian-iso/custom/etc/profile.d/z99-archipelago.sh: full rewrite
+- build/debian-iso/custom/archipelago/auto-start.sh: full rewrite
+- archipelago-scripts/archipelago-menu.sh: full rewrite
+- build/debian-iso/custom/isolinux/stdmenu.cfg, menu.cfg, live.cfg: updated
+- branding/generate-isolinux-splash.py: new file (640x480 splash generator, optional)
+
+### Outstanding Issues
+- UEFI boot broken — drops to grub> prompt, only Legacy BIOS works
+- ISOLINUX resolution kept at 1024x768, may clip on some hardware
+- Install + onboarding logs confirmed present on .198 (5 log files)
+- Need to review actual log content from .198
+
+### Target Machine
+- Dell on .198, Legacy BIOS, password: archipelago