Add project documentation
This commit is contained in:
177
docs/architecture.md
Normal file
177
docs/architecture.md
Normal file
@@ -0,0 +1,177 @@
|
||||
# Architecture
|
||||
|
||||
## High-Level Design
|
||||
|
||||
Agentic OS is a multi-service Docker stack:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
browser[Browser] --> frontend[Next.js Frontend]
|
||||
frontend --> backend[FastAPI Backend]
|
||||
backend --> postgres[(Postgres)]
|
||||
backend --> redis[(Redis)]
|
||||
worker[Worker] --> redis
|
||||
worker --> agent[LangGraph Agent]
|
||||
agent --> litellm[LiteLLM Gateway]
|
||||
litellm --> ollama[Ollama on Host]
|
||||
litellm --> cloud[Gemini DeepSeek OpenRouter]
|
||||
worker --> mcp[MCP Sidecars]
|
||||
mcp --> devices[Vessel Devices]
|
||||
agent --> memory[Project Memory MCP]
|
||||
agent --> obsidian[Obsidian Vault Git]
|
||||
```
|
||||
|
||||
## Services
|
||||
|
||||
| Service | Location | Responsibility |
|
||||
|---|---|---|
|
||||
| `frontend` | `frontend/` | Next.js operator console, task UX, dashboard, rules, skills, models, inventory. |
|
||||
| `backend` | `backend/app/` | FastAPI API, auth, schemas, inventory, tasks, MCP admin, balance polling. |
|
||||
| `worker` | `backend/app/worker.py` | Consumes task queue, owns MCP manager, runs LangGraph task agent. |
|
||||
| `postgres` | `docker-compose.yml` | Users, inventory, tasks, task events, approvals, balances, audit logs. |
|
||||
| `redis` | `docker-compose.yml` | Task queue, live event pub/sub, cancel/escalate flags, MCP command queue/status. |
|
||||
| `litellm` | `litellm/config.yaml` | Unified API for local Ollama and cloud providers. |
|
||||
| MCP sidecars | `mcp-servers/` and cloned repos | Device-facing tool servers. |
|
||||
|
||||
## Backend Modules
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `backend/app/api/` | REST and WebSocket endpoints. |
|
||||
| `backend/app/agent/` | LangGraph workflow, device profiles, playbooks, tool execution, MCP development. |
|
||||
| `backend/app/mcp_manager/` | MCP repository sync, subprocess lifecycle, command queue, tool discovery. |
|
||||
| `backend/app/services/` | Rules, skills, balance, model routing, report formatting, memory/Obsidian integrations, audit. |
|
||||
| `backend/app/models/` | SQLAlchemy models. |
|
||||
| `backend/alembic/` | Database migrations. |
|
||||
|
||||
## Frontend Modules
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `frontend/app/page.tsx` | Operations dashboard. |
|
||||
| `frontend/app/tasks/page.tsx` | Task list and guided task creation. |
|
||||
| `frontend/app/tasks/[id]/page.tsx` | Live task page, approvals, report, artifacts, run history. |
|
||||
| `frontend/app/inventory/page.tsx` | Vessel and device management. |
|
||||
| `frontend/app/mcp/page.tsx` | MCP server status and controls. |
|
||||
| `frontend/app/rules/page.tsx` | Troubleshooting rules editor. |
|
||||
| `frontend/app/skills/page.tsx` | Procedural skills editor/preview. |
|
||||
| `frontend/app/models/page.tsx` | LLM routing configuration. |
|
||||
| `frontend/components/` | Shared UI, task status, task report panels, app shell. |
|
||||
| `frontend/lib/` | API client, auth context, shared task types. |
|
||||
|
||||
## Task Lifecycle
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
create[Create Task] --> preview[Preview Scope]
|
||||
preview --> queue[Queue in Redis]
|
||||
queue --> worker[Worker Picks Task]
|
||||
worker --> context[Context Search]
|
||||
context --> triage[Triage and Scope]
|
||||
triage --> diagnose[Run MCP Diagnostics]
|
||||
diagnose --> reason[LLM Reasoning Cascade]
|
||||
reason --> report[Structured Report]
|
||||
report --> approvals[Approval Requests]
|
||||
report --> memory[Project Memory]
|
||||
report --> obsidian[Obsidian Note]
|
||||
report --> dashboard[Dashboard Summary]
|
||||
```
|
||||
|
||||
### 1. Create and Preview
|
||||
|
||||
The frontend calls `POST /api/tasks/preview` with title, issue, and vessel. The backend uses `select_devices_for_issue()` from `backend/app/services/troubleshooting_rules.py` so preview and execution share the same routing logic.
|
||||
|
||||
### 2. Queue
|
||||
|
||||
`POST /api/tasks` inserts a `Task`, sets status `queued`, emits a `log` event, and pushes the task ID to Redis list `agentic:task_queue`.
|
||||
|
||||
### 3. Worker Execution
|
||||
|
||||
The worker consumes the queue, loads MCP servers, and calls `run_task_agent()` in `backend/app/agent/graph.py`.
|
||||
|
||||
### 4. LangGraph Pipeline
|
||||
|
||||
| Node | Responsibility |
|
||||
|---|---|
|
||||
| `context_step` | Search project-memory and Obsidian for related past investigations. |
|
||||
| `triage_step` | Use local LLM plus rules/skills to plan checks and severity. |
|
||||
| `diagnose_step` | Run device-specific or generic MCP diagnostics. |
|
||||
| `reason_step` | Use tiered LLM cascade to interpret findings. |
|
||||
| `report_step` | Build structured report, create approvals, save artifacts. |
|
||||
|
||||
### 5. Live Events
|
||||
|
||||
Events are sent through `publish_event()` in `backend/app/services/events.py`.
|
||||
|
||||
- Persisted events go to `task_events` and Redis pub/sub.
|
||||
- Ephemeral high-frequency events such as `progress` and `tool_start` are streamed live but not stored.
|
||||
- The frontend WebSocket `/api/tasks/{id}/stream` replays recent persisted events and then streams live ones.
|
||||
|
||||
### 6. Report Shape
|
||||
|
||||
Reports are JSON stored on the `tasks.report` column. New reports include:
|
||||
|
||||
- `executive_summary`;
|
||||
- `scope_checked`;
|
||||
- `findings`;
|
||||
- `planned_steps`;
|
||||
- `actions_taken`;
|
||||
- `tools_run`;
|
||||
- `diagnostics` with formatted evidence and raw metadata;
|
||||
- `memory_id`, `memory_project_id`;
|
||||
- `obsidian_path`, `obsidian_push_error`;
|
||||
- LLM usage and cost fields.
|
||||
|
||||
Backward-compatible fields such as `summary`, `steps`, `devices_checked`, `root_cause`, and `resolution` remain available.
|
||||
|
||||
## Persistence
|
||||
|
||||
### Postgres
|
||||
|
||||
Stores durable application state:
|
||||
|
||||
- users and roles;
|
||||
- vessels and devices;
|
||||
- tasks, events, reports, approvals;
|
||||
- balance snapshots;
|
||||
- audit logs.
|
||||
|
||||
Database migrations live under `backend/alembic/`.
|
||||
|
||||
### Redis
|
||||
|
||||
Stores operational state:
|
||||
|
||||
- task queue;
|
||||
- live task event channels;
|
||||
- MCP command queue and status cache;
|
||||
- task cancellation and LLM escalation flags.
|
||||
|
||||
### Project Memory
|
||||
|
||||
Task reports and durable operational facts are saved through an HTTP MCP server configured by `MEMORY_MCP_URL`. Task outputs use the project ID configured by `MEMORY_TASK_PROJECT_ID`.
|
||||
|
||||
### Obsidian
|
||||
|
||||
The backend clones the configured vault repository into `/runtime/obsidian-vault`, writes task notes under `OBSIDIAN_TASKS_FOLDER`, and commits/pushes when `OBSIDIAN_AUTO_PUSH=true`.
|
||||
|
||||
## MCP Integration Boundary
|
||||
|
||||
The application does not talk directly to infrastructure devices. It calls MCP tools. MCP servers hide the details of pfSense REST, Proxmox API/SSH, Asterisk shell/docker access, FortiGate/FortiSwitch APIs, and generic SSH.
|
||||
|
||||
This boundary makes the system extensible:
|
||||
|
||||
1. Add or clone an MCP server.
|
||||
2. Add a device catalog entry if needed.
|
||||
3. Add rules/skills so the agent knows when to use it.
|
||||
4. Add diagnostics formatting for readable reports.
|
||||
|
||||
## Security Boundaries
|
||||
|
||||
- Device secrets are encrypted at rest.
|
||||
- Real `.env` values must not be committed.
|
||||
- MCP write tools are disabled by default unless each MCP explicitly enables writes.
|
||||
- MCP self-development requires approval by default.
|
||||
- Config-changing proposals are represented as approval requests before execution.
|
||||
- Project memory and Obsidian should store references and summaries, not secret values.
|
||||
|
||||
Reference in New Issue
Block a user