AI System Guardian

AI System Guardian is a self-healing monitoring daemon for multi-agent AI architectures. It runs a 10-second patrol loop that executes 13 runtime health checks (PM2 process status, HTTP probes, pipeline stall detection, disk space, budget burn rate) and a 24-hour deep code security scan with 21 patterns (eval, exec, pickle, shell injection, unauthorized network calls). When it detects a failure, it automatically restarts processes, rolls back stuck tasks, or escalates to a human operator.

When a health check fails — say a PM2 process crashes — the sentinel issues a `pm2 restart` command automatically. A 5-minute cooldown prevents restart storms. If the process crashes again within cooldown, the sentinel writes an escalation task and sends a Discord/Telegram alert. For pipeline stalls, it rolls back the stuck task and reassigns it to a different worker. For bloated logs, it rotates them automatically.

The scanner checks AI-generated scripts for: eval()/exec() (arbitrary code execution), __import__/importlib (dynamic imports), os.system/popen/exec*/spawn* (shell injection), subprocess.Popen, ctypes (native code loading), pickle/shelve (insecure deserialization), shutil.rmtree (recursive deletion), input() (blocking I/O), direct .key/.env/token file access, and outbound HTTP calls to non-allowlisted domains.

PM2 is recommended because the sentinel monitors PM2-managed processes and uses pm2 restart for auto-healing. However, the health check patterns (HTTP probes, disk monitoring, log rotation, code scanning) work independently. You could adapt it to systemd, Docker health checks, or Kubernetes liveness probes.

Yes. Each health check is a standalone function that returns (ok, message). To add a custom check, write a function following the same pattern and add it to the shadow_sentinel_scan() main loop. The sentinel will include it in the dashboard JSON and daily digest automatically.

The sentinel tracks API costs logged by the task pipeline. It calculates hourly and daily burn rates and projects monthly spend. If the projected spend exceeds your configured threshold, it sends an alert. This catches runaway retry loops or infinite task generation that can drain API credits in hours.