Add project documentation

docs/architecture.md

@@ -0,0 +1,177 @@
+# Architecture
+
+## High-Level Design
+
+Agentic OS is a multi-service Docker stack:
+
+```mermaid
+flowchart LR
+  browser[Browser] --> frontend[Next.js Frontend]
+  frontend --> backend[FastAPI Backend]
+  backend --> postgres[(Postgres)]
+  backend --> redis[(Redis)]
+  worker[Worker] --> redis
+  worker --> agent[LangGraph Agent]
+  agent --> litellm[LiteLLM Gateway]
+  litellm --> ollama[Ollama on Host]
+  litellm --> cloud[Gemini DeepSeek OpenRouter]
+  worker --> mcp[MCP Sidecars]
+  mcp --> devices[Vessel Devices]
+  agent --> memory[Project Memory MCP]
+  agent --> obsidian[Obsidian Vault Git]
+```
+
+## Services
+
+| Service | Location | Responsibility |
+|---|---|---|
+| `frontend` | `frontend/` | Next.js operator console, task UX, dashboard, rules, skills, models, inventory. |
+| `backend` | `backend/app/` | FastAPI API, auth, schemas, inventory, tasks, MCP admin, balance polling. |
+| `worker` | `backend/app/worker.py` | Consumes task queue, owns MCP manager, runs LangGraph task agent. |
+| `postgres` | `docker-compose.yml` | Users, inventory, tasks, task events, approvals, balances, audit logs. |
+| `redis` | `docker-compose.yml` | Task queue, live event pub/sub, cancel/escalate flags, MCP command queue/status. |
+| `litellm` | `litellm/config.yaml` | Unified API for local Ollama and cloud providers. |
+| MCP sidecars | `mcp-servers/` and cloned repos | Device-facing tool servers. |
+
+## Backend Modules
+
+| Path | Purpose |
+|---|---|
+| `backend/app/api/` | REST and WebSocket endpoints. |
+| `backend/app/agent/` | LangGraph workflow, device profiles, playbooks, tool execution, MCP development. |
+| `backend/app/mcp_manager/` | MCP repository sync, subprocess lifecycle, command queue, tool discovery. |
+| `backend/app/services/` | Rules, skills, balance, model routing, report formatting, memory/Obsidian integrations, audit. |
+| `backend/app/models/` | SQLAlchemy models. |
+| `backend/alembic/` | Database migrations. |
+
+## Frontend Modules
+
+| Path | Purpose |
+|---|---|
+| `frontend/app/page.tsx` | Operations dashboard. |
+| `frontend/app/tasks/page.tsx` | Task list and guided task creation. |
+| `frontend/app/tasks/[id]/page.tsx` | Live task page, approvals, report, artifacts, run history. |
+| `frontend/app/inventory/page.tsx` | Vessel and device management. |
+| `frontend/app/mcp/page.tsx` | MCP server status and controls. |
+| `frontend/app/rules/page.tsx` | Troubleshooting rules editor. |
+| `frontend/app/skills/page.tsx` | Procedural skills editor/preview. |
+| `frontend/app/models/page.tsx` | LLM routing configuration. |
+| `frontend/components/` | Shared UI, task status, task report panels, app shell. |
+| `frontend/lib/` | API client, auth context, shared task types. |
+
+## Task Lifecycle
+
+```mermaid
+flowchart TD
+  create[Create Task] --> preview[Preview Scope]
+  preview --> queue[Queue in Redis]
+  queue --> worker[Worker Picks Task]
+  worker --> context[Context Search]
+  context --> triage[Triage and Scope]
+  triage --> diagnose[Run MCP Diagnostics]
+  diagnose --> reason[LLM Reasoning Cascade]
+  reason --> report[Structured Report]
+  report --> approvals[Approval Requests]
+  report --> memory[Project Memory]
+  report --> obsidian[Obsidian Note]
+  report --> dashboard[Dashboard Summary]
+```
+
+### 1. Create and Preview
+
+The frontend calls `POST /api/tasks/preview` with title, issue, and vessel. The backend uses `select_devices_for_issue()` from `backend/app/services/troubleshooting_rules.py` so preview and execution share the same routing logic.
+
+### 2. Queue
+
+`POST /api/tasks` inserts a `Task`, sets status `queued`, emits a `log` event, and pushes the task ID to Redis list `agentic:task_queue`.
+
+### 3. Worker Execution
+
+The worker consumes the queue, loads MCP servers, and calls `run_task_agent()` in `backend/app/agent/graph.py`.
+
+### 4. LangGraph Pipeline
+
+| Node | Responsibility |
+|---|---|
+| `context_step` | Search project-memory and Obsidian for related past investigations. |
+| `triage_step` | Use local LLM plus rules/skills to plan checks and severity. |
+| `diagnose_step` | Run device-specific or generic MCP diagnostics. |
+| `reason_step` | Use tiered LLM cascade to interpret findings. |
+| `report_step` | Build structured report, create approvals, save artifacts. |
+
+### 5. Live Events
+
+Events are sent through `publish_event()` in `backend/app/services/events.py`.
+
+- Persisted events go to `task_events` and Redis pub/sub.
+- Ephemeral high-frequency events such as `progress` and `tool_start` are streamed live but not stored.
+- The frontend WebSocket `/api/tasks/{id}/stream` replays recent persisted events and then streams live ones.
+
+### 6. Report Shape
+
+Reports are JSON stored on the `tasks.report` column. New reports include:
+
+- `executive_summary`;
+- `scope_checked`;
+- `findings`;
+- `planned_steps`;
+- `actions_taken`;
+- `tools_run`;
+- `diagnostics` with formatted evidence and raw metadata;
+- `memory_id`, `memory_project_id`;
+- `obsidian_path`, `obsidian_push_error`;
+- LLM usage and cost fields.
+
+Backward-compatible fields such as `summary`, `steps`, `devices_checked`, `root_cause`, and `resolution` remain available.
+
+## Persistence
+
+### Postgres
+
+Stores durable application state:
+
+- users and roles;
+- vessels and devices;
+- tasks, events, reports, approvals;
+- balance snapshots;
+- audit logs.
+
+Database migrations live under `backend/alembic/`.
+
+### Redis
+
+Stores operational state:
+
+- task queue;
+- live task event channels;
+- MCP command queue and status cache;
+- task cancellation and LLM escalation flags.
+
+### Project Memory
+
+Task reports and durable operational facts are saved through an HTTP MCP server configured by `MEMORY_MCP_URL`. Task outputs use the project ID configured by `MEMORY_TASK_PROJECT_ID`.
+
+### Obsidian
+
+The backend clones the configured vault repository into `/runtime/obsidian-vault`, writes task notes under `OBSIDIAN_TASKS_FOLDER`, and commits/pushes when `OBSIDIAN_AUTO_PUSH=true`.
+
+## MCP Integration Boundary
+
+The application does not talk directly to infrastructure devices. It calls MCP tools. MCP servers hide the details of pfSense REST, Proxmox API/SSH, Asterisk shell/docker access, FortiGate/FortiSwitch APIs, and generic SSH.
+
+This boundary makes the system extensible:
+
+1. Add or clone an MCP server.
+2. Add a device catalog entry if needed.
+3. Add rules/skills so the agent knows when to use it.
+4. Add diagnostics formatting for readable reports.
+
+## Security Boundaries
+
+- Device secrets are encrypted at rest.
+- Real `.env` values must not be committed.
+- MCP write tools are disabled by default unless each MCP explicitly enables writes.
+- MCP self-development requires approval by default.
+- Config-changing proposals are represented as approval requests before execution.
+- Project memory and Obsidian should store references and summaries, not secret values.
+

docs/developer-operations.md

@@ -0,0 +1,201 @@
+# Developer and Operations Guide
+
+## Local Stack
+
+Agentic OS runs with Docker Compose.
+
+```bash
+docker compose up -d --build
+```
+
+Default service ports:
+
+| Service | Port | Purpose |
+|---|---:|---|
+| Frontend | `3000` | Operator UI |
+| Backend | `8000` | FastAPI API |
+| LiteLLM | `4000` | LLM gateway |
+| Postgres | `5432` | Application and LiteLLM databases |
+
+Ollama runs on the macOS host, not inside Docker, so it can use Apple Silicon acceleration. Containers reach it through `host.docker.internal:11434`.
+
+## Configuration
+
+Use `.env.example` as the source of truth for configuration names. Copy it to `.env` and fill real values locally.
+
+Never commit `.env`.
+
+Important groups:
+
+- **Security**: `SECRET_KEY`, `CREDENTIAL_ENCRYPTION_KEY`, first admin credentials.
+- **Database/cache**: `DATABASE_URL`, `REDIS_URL`.
+- **CORS**: `CORS_ALLOW_ORIGINS`.
+- **LLM routing**: local, Gemini, DeepSeek, OpenRouter, LiteLLM settings.
+- **MCP development**: enable/approval/push policy.
+- **Gitea**: clone/push base URL, username, token, MCP repos.
+- **Obsidian**: vault repo, task folder, auto-push.
+- **Project memory**: HTTP MCP URL, token, project IDs.
+- **Rules/skills/templates**: mounted paths for agent guidance and baselines.
+- **Device defaults**: per-device credentials and endpoint defaults.
+
+## Database
+
+Models live in `backend/app/models/`.
+
+Alembic migrations live in `backend/alembic/`.
+
+The backend initializes/upgrades the database on startup. Avoid ad-hoc schema changes in application startup code; prefer migrations.
+
+## Tests
+
+Run backend tests inside the container:
+
+```bash
+docker compose exec -T backend pytest -q
+```
+
+Run a focused subset:
+
+```bash
+docker compose exec -T backend pytest tests/test_troubleshooting_rules.py tests/test_agent.py -q
+```
+
+Build the frontend to validate TypeScript and production rendering:
+
+```bash
+docker compose build frontend
+```
+
+Known caveat: some environment-sensitive tests can fail when the Compose `.env` overrides test-specific environment expectations. Keep test configuration isolated when adding new tests.
+
+## Key Extension Points
+
+### Add a Device Type
+
+1. Add or update a catalog entry in `backend/app/inventory_catalog.py`.
+2. Add environment defaults in `backend/app/services/device_defaults.py`.
+3. Add an MCP server or select an existing one.
+4. Add routing keywords/rules in `rules/troubleshooting.yaml`.
+5. Add a device profile or playbook under `backend/app/agent/`.
+6. Add report formatting in `backend/app/services/diagnostic_format.py`.
+7. Add tests.
+
+### Add an MCP Server
+
+MCP servers can be local folders under `mcp-servers/` or cloned from Gitea by name.
+
+Configuration:
+
+- `MCP_REPOS`: comma-separated repo names.
+- `MCP_LOCAL_DIR`: local mounted MCP folder.
+- `MCP_START_<REPO_NAME>`: optional custom launch command.
+
+The worker starts servers and exposes tool status through the MCP manager.
+
+### Add a Troubleshooting Rule
+
+Edit `rules/troubleshooting.yaml`.
+
+Rules should define:
+
+- matching keywords and optional `all_of`;
+- target devices;
+- diagnostic order;
+- severity hint;
+- human-readable hints and steps.
+
+Rules should be specific enough to avoid accidental full-stack sweeps.
+
+### Add a Skill
+
+Create or edit `skills/<id>/SKILL.md`.
+
+Skills add deep procedural guidance. They should not decide device scope; rules do that. A good skill describes the investigation method, known pitfalls, commands/tools to prefer, and what evidence proves or disproves the cause.
+
+### Add Diagnostic Formatting
+
+Raw MCP output is noisy. Prefer compact formatted summaries in `backend/app/services/diagnostic_format.py`.
+
+For each new diagnostic output:
+
+- parse JSON/text using structured logic where possible;
+- render short tables or bullet summaries;
+- preserve metadata such as `output_chars`, `raw_available`, and `truncated`;
+- keep raw output behind evidence expansion, not in the default report view.
+
+## Task Agent Development
+
+The main workflow is in `backend/app/agent/graph.py`.
+
+Be careful with changes there:
+
+- `context_node` affects memory/Obsidian reuse.
+- `triage_node` affects device scope and severity.
+- `diagnose_node` affects MCP calls and approval risk.
+- `reason_node` affects model cost and report interpretation.
+- `report_node` affects persistence, approvals, Memory, and Obsidian.
+
+For device behavior, prefer smaller profile modules such as:
+
+- `pfsense_profiles.py`
+- `proxmox_profiles.py`
+- `asterisk_profiles.py`
+- generic `playbooks.py`
+
+## Frontend Development
+
+Frontend pages live under `frontend/app/`.
+
+Important shared pieces:
+
+- `frontend/lib/api.ts`: REST and WebSocket helpers.
+- `frontend/lib/auth.tsx`: auth context and roles.
+- `frontend/lib/task-types.ts`: shared task and event helpers.
+- `frontend/components/TaskLiveStatus.tsx`: phase/timeline rendering.
+- `frontend/components/task-detail-panels.tsx`: report, scope, timeline, artifacts, cost, history.
+- `frontend/components/ui.tsx`: common UI primitives.
+
+Design principle: default screens should show summaries and attention items; raw evidence should be expandable.
+
+## Operational Checks
+
+After changing backend or worker code:
+
+```bash
+docker compose build backend worker
+docker compose up -d backend worker
+```
+
+After changing frontend code:
+
+```bash
+docker compose build frontend
+docker compose up -d frontend
+```
+
+After changing MCP servers:
+
+1. Use the MCP page to reload/upgrade when possible.
+2. Check worker logs if server status is `error`.
+3. Confirm the tool list changed as expected.
+
+## Troubleshooting the Platform
+
+| Symptom | First checks |
+|---|---|
+| Task stuck queued | Worker running, Redis healthy, no stuck in-flight task. |
+| No MCP tools | Worker logs, MCP repo cloned/local path exists, start command works. |
+| Empty Proxmox VM list | API token has `VM.Audit` on `/vms`, node permissions, SSL verify setting. |
+| Obsidian note missing | Task `obsidian_path`, backend logs, vault git status/push, local Obsidian pull. |
+| Balance missing | `/api/balance`, provider keys, DNS from backend container. |
+| Report too noisy | Add/adjust diagnostic formatter and report fields instead of dumping raw output. |
+
+## Security Practices
+
+- Do not commit `.env`, tokens, keys, or device credentials.
+- Reference secret names, not values, in docs and memory.
+- Keep MCP write flags disabled unless intentionally testing writes.
+- Keep MCP source patch approval enabled for normal operation.
+- Avoid broad CORS origins in deployments.
+- Treat task reports and Obsidian notes as operational records; do not include full credentials.
+

docs/operator-guide.md

@@ -0,0 +1,128 @@
+# Operator Guide
+
+This guide explains how an operator uses Agentic OS day to day.
+
+## Dashboard
+
+The dashboard at `/` is the operations view. Use it to answer:
+
+- What is running now?
+- What needs approval or attention?
+- Were recent tasks saved to memory and Obsidian?
+- Are cloud model balances healthy?
+- Are MCP servers loaded?
+
+Main sections:
+
+- **Active Tasks**: queued, running, and waiting approval tasks.
+- **Attention Needed**: failed tasks, pending approvals, Obsidian push failures, and low/errored API balances.
+- **Recent Findings**: compact summaries of completed reports.
+- **Budgets**: OpenRouter, DeepSeek, and other configured cloud balances.
+- **Platform Readiness**: MCP server status and tool counts.
+
+## Inventory
+
+Inventory describes vessels and onboard devices.
+
+1. Open `/inventory`.
+2. Create or edit a vessel with public IP/site notes.
+3. Enable device slots such as `proxmox`, `pfsense`, `docker_vm`, `asterisk_geneseasx`, `fortigate`, or `fortiswitch`.
+4. Add address, port, username, and secret only when they differ from environment defaults.
+
+Secrets are encrypted before being stored. Do not put plaintext credentials in vessel notes.
+
+## Creating a Task
+
+1. Open `/tasks`.
+2. Select **New task**.
+3. Pick a quick template or enter a title and issue manually.
+4. Select the vessel.
+5. Review **What will be checked**.
+6. Choose:
+   - **Create & watch live** to go to the task page.
+   - **Run in background** to return to the dashboard.
+
+The preview is important: it shows which devices and check groups the agent will run. If the preview says no devices match, rewrite the issue with explicit device names or ask for a full stack health check.
+
+Example issue text:
+
+```text
+check status of proxmox and pfsense
+```
+
+Expected preview:
+
+- Proxmox: VM/LXC status, guest configuration, host health.
+- pfSense: system status, interfaces, gateways, firewall/NAT when requested.
+
+## Understanding Live Status
+
+The task page shows the run as phases:
+
+1. **Context** — searching prior memory and Obsidian notes.
+2. **Plan** — deciding scope and diagnostic approach.
+3. **Connect** — connecting to MCP/device targets.
+4. **Diagnose** — running checks.
+5. **Analyze** — reasoning about results.
+6. **Report** — writing final report and artifacts.
+
+The timeline is intentionally concise. Expand command or evidence details only when you need raw output.
+
+## Approvals
+
+Some actions require approval:
+
+- MCP source patches proposed by MCP development.
+- Config-changing tool calls.
+- Git pushes for generated MCP changes when auto-push is disabled.
+
+When a task enters `waiting_approval`, open it and review the approval card. Approve only if the proposed change matches the intended fix and the risk is acceptable.
+
+## Reports
+
+Reports are structured to be readable first and evidential second.
+
+Default sections:
+
+- **Scope**: what was requested and what was actually checked.
+- **Report**: summary, scope checked, findings, recommendations.
+- **Evidence**: collapsed per-tool output and formatted tables.
+- **Saved Artifacts**: memory ID, Obsidian path, push status.
+- **Models & Cost**: LLM routing and cost summary.
+- **Run History**: follow-up runs on the same task.
+
+Important rule: a report should not imply a device was checked if it was not in `scope_checked` or `devices_checked`.
+
+## Continuing or Restarting Tasks
+
+Use **Continue investigation** when the first run is incomplete or you want targeted follow-up checks. The agent receives prior findings and your follow-up note.
+
+Use **Restart from scratch** when you want to discard run history and rerun the original task.
+
+Use **Stop task** for stuck or no-longer-needed queued/running tasks.
+
+## Memory and Obsidian
+
+At task completion:
+
+- Project memory receives a durable summary under `MEMORY_TASK_PROJECT_ID`.
+- Obsidian receives a markdown note under `OBSIDIAN_TASKS_FOLDER`.
+- The task report shows IDs/paths and push errors if any.
+
+If Obsidian push fails, the note may still exist in the backend runtime vault clone. Fix git/reachability and rerun or inspect logs before assuming the report was lost.
+
+## Rules and Skills
+
+Rules and skills guide the agent.
+
+- `/rules`: controls device selection, priority, severity hints, and standard checklist hints.
+- `/skills`: adds deeper procedures for known scenarios such as VoIP, DNS/captive portal, or safe pfSense changes.
+
+Use rules for **which devices to check**. Use skills for **how to investigate** a class of problem.
+
+## MCP Page
+
+Use `/mcp` to inspect MCP server health, tools, reload/upgrade servers, or trigger controlled development for MCP bugs.
+
+If a tool fails because of infrastructure permissions, fix the target system. If a tool fails because the MCP is missing a capability or has a code exception, MCP development may propose a patch.
+

docs/project-overview.md

@@ -0,0 +1,69 @@
+# Project Overview
+
+## Purpose
+
+Agentic OS is a Docker-based operations and troubleshooting platform for vessels and homelab infrastructure. It lets an operator describe an issue in plain language, select a vessel, and have an agent decide which onboard devices to inspect, run diagnostics through MCP tools, reason over the results, and produce a clean report.
+
+The project is built around GeneseasX-style deployments, but the architecture is intentionally modular: Proxmox, pfSense, Asterisk, Debian/Docker hosts, FortiGate, FortiSwitch, and other device families are represented as inventory slots and routed to matching MCP integrations.
+
+## Core Capabilities
+
+- **Vessel inventory**: stores vessels/locations and onboard device records, including encrypted credentials.
+- **Guided task creation**: previews which devices and checks will run before starting a task.
+- **Agentic troubleshooting**: uses a LangGraph workflow to gather context, triage, diagnose, reason, and report.
+- **MCP diagnostics**: launches or connects to device-specific MCP servers for read-only checks and approved write paths.
+- **Tiered LLM routing**: starts with local Ollama and escalates to Gemini, DeepSeek, or OpenRouter tiers when needed.
+- **Approvals**: gates config-changing actions and MCP self-development behind support/admin approval.
+- **Live task status**: streams progress, current phase, device/tool activity, approvals, and final artifacts.
+- **Concise reporting**: stores structured reports with findings, scope checked, evidence references, cost, and saved artifacts.
+- **Persistent knowledge**: searches and writes to project-memory and an Obsidian vault so future tasks can reuse prior findings.
+- **Operations dashboard**: highlights active tasks, attention items, recent findings, budgets, and MCP readiness.
+
+## Primary Users
+
+| User | Role in the system |
+|---|---|
+| `admin` | Full system administration, users, models, inventory, approvals, and task operations. |
+| `support` | Creates/runs tasks, manages vessel inventory, approves safe actions. |
+| `readonly` | Observes dashboards, inventory, tasks, timelines, and reports. |
+
+## What Agentic OS Is Not
+
+- It is not a replacement for the underlying firewalls, PBX, or hypervisor management UI.
+- It does not automatically apply unsafe config changes without explicit approval.
+- It should not store secrets in documentation, reports, memory entries, or committed files.
+- It does not make unreachable infrastructure reachable; MCPs still require network access and valid credentials.
+
+## Main Functional Areas
+
+### Dashboard
+
+The dashboard is the operator's first view. It summarizes:
+
+- active tasks and their status;
+- tasks needing approval or attention;
+- failed tasks and artifact push failures;
+- recent findings and saved reports;
+- cloud API balances;
+- MCP server readiness.
+
+### Inventory
+
+Inventory models vessels and onboard devices. Each device has a catalog key, device type, address, port, MCP server, username, and encrypted secret. Device defaults can be populated from environment variables so operators do not need to re-enter common credentials.
+
+### Tasks
+
+Tasks are the main unit of work. A task contains a title, issue text, target vessel, run history, live events, approvals, and final report. The task engine chooses devices from the issue text using troubleshooting rules and then runs device-specific diagnostics.
+
+### Rules and Skills
+
+Troubleshooting rules decide the device scope and standard hints. Skills add deeper procedure text for matching scenarios such as VoIP, DNS/captive portal, or pfSense change safety.
+
+### MCP Management
+
+The worker starts MCP servers as subprocess sidecars. MCP repos can be local folders or cloned from Gitea. The MCP page lets operators inspect loaded tools and perform maintenance actions such as reload, upgrade, and controlled MCP development.
+
+### Reports and Knowledge
+
+Reports are stored in Postgres, written to project-memory, and exported to Obsidian. The default UI shows concise findings; raw evidence remains available behind expandable sections.
+