# Agentic OS — troubleshooting decision rules # Edit this file or use Admin → Rules in the UI. Reloaded on each task run. # # Fields per rule (only these are read by the engine): # id, name, priority (ties only; matching is by keyword specificity), severity # match.keywords — any keyword match scores the rule (multi-word phrases score 2x) # match.all_of — optional; every term must appear or the rule is skipped # devices — catalog_key list, or [all] for every onboard device # order — optional diagnose sequence (subset of devices) # hints — shown verbatim to triage/reasoning LLM # steps — suggested investigation checklist (also shown to the LLM) # # NOTE: hints/steps are the ONLY place free-text knowledge reaches the agent. # Topology values below are the TYPICAL GeneseasX golden-lab pattern; field # vessels differ (different subnets, container named "voip" vs "asterisk", # FortiGate in the trunk path). Always confirm live — never assume the IPs. version: 1 broad_triggers: # Phrases only — avoid generic words like "check" or "status" that appear in # scoped tasks (e.g. "check VM status on Proxmox"). keywords: - health check - full stack - all systems - all devices - everything - full audit - stack audit - nothing works - whole ship device_keywords: proxmox: [proxmox, pve, hypervisor, qemu, lxc, vm, virtual machine, node] pfsense: [pfsense, firewall, nat, vpn, openvpn, wireguard, gateway, captive, dns, portal, snat, hairpin, reflection] docker_vm: [docker, geneseas, geneseasx, compose, container, shipportal, ship-dns, radius] asterisk_geneseasx: [asterisk, voip, sip, pjsip, phone, trunk, registration, dial, rtp, iax, iax2, dialplan, codec, opus, ulaw, extension, originate, yealink] asterisk_satbox: [satbox, tmgeneseas, native asterisk] fortigate: [fortigate, fortios] fortiswitch: [fortiswitch, switch port] rules: - id: no-registration name: Phone / SIP registration failure priority: 90 severity: high match: keywords: [register, registration, unregistered, "401", "403", timeout, sip, "cannot register", "won't register, not registering"] devices: [pfsense, asterisk_geneseasx] order: [pfsense, asterisk_geneseasx] hints: - "Typical GeneseasX path: phones do NOT register directly to Asterisk. They hit a pfSense SIP NAT redirect (e.g. 192.168.0.254:5060 -> Asterisk 10.20.30.222:5060). Verify that port-forward + the matching pass rule exist before blaming Asterisk." - "Yealink red flags seen before: SIP server set to the pfSense VIP instead of the Asterisk IP; NAT keepalive enabled (AccountNATEnableUDPUpdate=1) while rport disabled (AccountNATRport=0). That mismatch destabilises registration/media." - "On Asterisk, registration = pjsip AOR/contact + qualify. A failing 'qualify' (unreachable) usually means a NAT/firewall path problem, not credentials." steps: - "pfsense: confirm SIP NAT port-forward (UDP 5060 -> Asterisk) and the pass rule on the phone VLAN (Business/CREW)" - "pfsense: check the SIP redirect VIP and outbound NAT for the phone subnet" - "asterisk: pjsip show registrations" - "asterisk: pjsip show endpoints / pjsip show aors / pjsip show contacts" - "asterisk: pjsip show endpoint — check qualify status and contact address" - id: one-way-audio name: One-way audio / RTP path priority: 85 severity: high match: keywords: ["one-way", one way, rtp, no audio, "can't hear", cant hear, "only one side", asymmetric, audio, silence, "no sound"] devices: [pfsense, asterisk_geneseasx] order: [pfsense, asterisk_geneseasx] hints: - "MOST COMMON ROOT CAUSE here: pfSense Business interface (vtnet3 / opt2) is missing a pass rule for the RTP range (UDP 10000-10029) toward shipPortal. Result: phone->Asterisk RTP is dropped while Asterisk->phone passes, so the REMOTE party cannot hear the phone. Which direction is silent tells you which way is blocked." - "Proven fix pattern: on pfSense add 'pass udp 10000-10029 to shipPortal' on the phone VLAN AND set outbound SNAT for RTP to the shipPortal VIP (e.g. 192.168.0.254). Back up config.xml before any write." - "Asterisk side: pjsip.conf local_net must include BOTH the phone subnet (e.g. 192.168.0.0/24) and the VoIP subnet (e.g. 10.20.30.0/24); external_media_address must be the Asterisk reachable IP (e.g. 10.20.30.222). Missing local_net causes wrong NAT mangling of SDP." - "Confirm with evidence, not theory: check pfSense filter.log for blocked phone->Asterisk :100xx flows, and Asterisk RTP RX/TX counters during a live call (both should increment roughly equally)." steps: - "pfsense: list firewall rules on the phone VLAN for UDP 10000-10029 toward Asterisk/shipPortal" - "pfsense: check outbound NAT (SNAT) for the RTP range" - "pfsense: inspect filter.log during a live call for blocked RTP (src phone -> Asterisk :100xx)" - "asterisk: pjsip show endpoint — verify local_net / external_media_address" - "asterisk: rtp set debug on (or core show channels) during a live call — confirm bidirectional RTP RX/TX" - id: outbound-trunk-failure name: Outbound call failure / IAX trunk priority: 87 severity: high match: keywords: ["call out", "cant call out", "can't call out", "cannot call out", "no outbound", outbound, "dial out", congestion, "fast busy", trunk, iax, iax2, "subscriber absent", "no route"] devices: [asterisk_geneseasx, pfsense, fortigate] order: [asterisk_geneseasx, pfsense, fortigate] hints: - "Outbound on GeneseasX typically leaves via an IAX2 trunk (UDP 4569) to an upstream PBX (e.g. tmvoip.satlite.eu). Start by proving the trunk is up: iax2 show peers / iax2 show registry (qualify latency)." - "Dialplan transforms matter: e.g. an outbound pattern _7XXXX strips the leading 7 before sending to the trunk. Confirm the dialed number actually matches a route." - "If the trunk qualifies OK and the call still fails fast with congestion / 'Subscriber absent' (cause 20), the problem is the trunk ROUTE or CODEC, not pfSense. Known gotcha: IAX trunk negotiated opus-only while PJSIP endpoints are ulaw — fix the allow= codec list / transcoding." - "Firewall is rarely the cause once qualify is OK. On field vessels the IAX trunk is SNAT'd out via a FortiGate policy; in the golden lab the trunk path is pfSense-only. Only dig into firewall if iax2 qualify is failing." steps: - "asterisk: iax2 show peers and iax2 show registry — trunk reachability + latency" - "asterisk: dialplan show — confirm the dial pattern and route for the failing number" - "asterisk: live test — originate/dial the number and watch core show channels + verbose for the hangup cause" - "asterisk: compare codec allow= on the trunk vs the endpoint (opus vs ulaw mismatch)" - "pfsense/fortigate: only if qualify fails — confirm UDP 4569 passes outbound to the trunk peer" - id: container-down name: Docker / Asterisk container down priority: 80 severity: high match: keywords: [container, "docker down", crash, crashed, restart, restarting, "voip down", shipportal, "container down", exited, unhealthy] devices: [proxmox, docker_vm, asterisk_geneseasx] order: [proxmox, docker_vm, asterisk_geneseasx] hints: - "Work bottom-up: Proxmox guest running -> Docker VM reachable -> target container healthy." - "Container name varies by deployment: golden lab uses 'asterisk', some field vessels use 'voip'. Check docker ps output for the actual name before docker exec." - "A container in a restart loop usually has a config/volume error — check 'docker logs' for the crash reason rather than just restarting it." steps: - "proxmox: list VMs / LXC and confirm the Docker VM guest is running" - "docker_vm: docker ps -a — find the container state and real name" - "docker_vm: docker logs --tail 100 — crash/restart reason" - "asterisk: docker exec asterisk -rx 'core show version' once it's up" - id: crew-dns-slow name: Crew DNS slow / slow internet priority: 78 severity: medium match: all_of: [dns] keywords: ["slow dns", "dns slow", "dns latency", "slow internet", slow, latency, "ship-dns", upstream, resolve, resolving, "3 seconds"] devices: [docker_vm, pfsense] order: [docker_vm, pfsense] hints: - "Known root cause: the ship-dns container's PRIMARY upstream is unreachable from the Docker VM, so every query waits the full timeout before falling back. Seen with primary 1.1.1.1 blocked -> ~3s per lookup; 8.8.8.8/9.9.9.9 answered in <0.1s." - "Proven fix: set ship-dns primary=8.8.8.8, secondary=9.9.9.9, dns_timeout=2 (and raise resolver worker concurrency). Measure per-upstream timing from the Docker VM before/after." - "Architecture rule: do NOT hand the Docker management IP (e.g. 10.20.30.222) directly to crew/business as DNS — pfSense is the DNS gateway and redirects per segment." - "PERSISTENCE WARNING: in-place edits inside the ship-dns container revert on image recreate. Fold fixes into the image, don't just docker cp." steps: - "docker_vm: docker ps — ship-dns container health" - "docker_vm: time per-upstream lookups from the Docker VM (dig @1.1.1.1 / @8.8.8.8 / @9.9.9.9 example.com)" - "docker_vm: check ship-dns dns_settings (primary/secondary/timeout) and worker concurrency" - "pfsense: verify the crew/business DNS redirect rules point at pfSense, not the Docker IP" - id: dns-captive-portal name: DNS / captive portal access priority: 75 severity: medium match: keywords: [captive, portal, "captive portal", crew, redirect, coovachilli, radius, "cant login", "can't login", "no internet", authentication] devices: [pfsense, docker_vm] order: [pfsense, docker_vm] hints: - "GeneseasX crew access = pfSense captive portal + RADIUS auth back to the Docker stack (e.g. 10.20.30.222). Crew is isolated via floating rules + CP, so a missing pass rule blocks even authenticated traffic." - "Portal page is reached via hairpin / NAT reflection: crew -> pfSense:443 -> Docker portal. If the portal won't load, check NAT reflection and the 443 redirect, not just the container." - "CREW often has an auto redirect but may LACK explicit pass rules if a new service (e.g. VoIP) was deployed there — verify pass rules exist for the new service." steps: - "pfsense: captive portal zone status and the crew interface rules (floating + interface)" - "pfsense: confirm RADIUS reachability to the Docker stack (auth/acct ports) and the 443 portal redirect / NAT reflection" - "docker_vm: docker ps — portal + RADIUS containers healthy" - "docker_vm: docker logs for the portal/RADIUS container on failed auth" - id: proxmox-vm-inventory name: Proxmox VM / LXC status and config priority: 86 severity: low match: keywords: [proxmox, qemu, lxc, hypervisor, pve, guest, vmid] devices: [proxmox] order: [proxmox] hints: - Proxmox focus — list QEMU/LXC guests and fetch per-VM status + config when requested - Include actual VM data in the report; do not suggest checking the UI instead - If other devices are named in the issue, check those too with their own diagnostics steps: - proxmox list nodes - proxmox list qemu and lxc on each node - proxmox get_qemu_status and get_qemu_config for each guest (especially running) - id: vm-unreachable name: VM / host unreachable priority: 70 severity: high match: keywords: [unreachable, "vm down", "host down", "ping fail", offline, "no route"] devices: [proxmox, pfsense] order: [proxmox, pfsense] hints: - Proxmox guest status and networking first, then pfSense routes/NAT steps: - proxmox list nodes and VM status - pfsense list gateways and interface status - id: list-firewall-rules name: List pfSense firewall / NAT policies priority: 88 severity: low match: keywords: [list, firewall, rule, rules, policy, policies, nat, "port forward", filter] devices: [pfsense] order: [pfsense] hints: - Call pfsense_list_firewall_rules, port_forwards, outbound NAT — include raw data in report - "Typical interface->role map (confirm live): vtnet0=WAN1, vtnet1=WAN2, vtnet2=CREW, vtnet3=Business, vtnet4=Management. Management is usually pass-all; Business is restrictive (DNS/HTTPS/SIP); CREW is captive-portal isolated." steps: - pfsense list interfaces - pfsense list firewall rules (all interfaces) - pfsense list port forwards and outbound NAT - id: asterisk-health name: Asterisk health check priority: 60 severity: medium match: keywords: [asterisk, voip, pjsip, "voip health"] all_of: [health] devices: [asterisk_geneseasx, docker_vm] order: [docker_vm, asterisk_geneseasx] hints: - Confirm the Asterisk container is healthy on the Docker VM (name 'asterisk' or 'voip') before deep CLI - Cover the three pillars — endpoint registrations, active channels, and trunk state steps: - "docker_vm: docker ps — Asterisk container status" - "asterisk: pjsip show registrations and pjsip show endpoints" - "asterisk: iax2 show peers — trunk reachability" - "asterisk: core show channels — active calls" default: severity: medium devices: [] hints: - No specific rule matched — diagnose only devices mentioned in the issue text - Do not run unrelated MCPs; if scope is unclear, say so in the report