FaultIQ — Graph-Based Fault Analysis & SOP Automation

  

FaultIQ is an open-source platform that detects faults in distributed systems, identifies the root cause through graph-based traversal, and guides operators through structured remediation with automated verification.

The problem: When payment-api fails, it cascades to 4 other services. Every tool fires a separate alert. Your operator spends 45 minutes manually tracing which service actually caused the failure.

FaultIQ's answer: BFS traversal across your service dependency graph identifies the single root cause service in milliseconds. A structured SOP playbook sequences the fix steps. The system auto-verifies recovery when healthy signals arrive.

---

Demo in 30 Seconds

┌─────────────────────────────────────────────────────────────────┐
│  1. Inject fault: ledger-service starts timing out              │
│  2. Detection engine: BFS traversal → root cause = ledger-svc   │
│  3. SOP playbook: "Check recent deploy" → "Inspect DB pool"     │
│  4. Fix deployed → healthy signal → auto-resolved               │
└─────────────────────────────────────────────────────────────────┘

Try it live:

git clone https://github.com/theprodsde/faultiq && cd faultiq
cp .env.example .env
docker compose up --build
First build takes ~6 min (13 images + Keycloak realm import). After that:
  docker compose up -d        # ~60-90s to a usable UI
Open http://localhost:4001
Sign in: super / superpass
Go to /dashboard/demo → "All healthy — continue", then "Fault ledger-service"

No terminal needed after startup — the demo page has a one-click fault injection button. On first boot the stack seeds the demo tenants and projects from scripts/service-map.yaml and builds the service graph automatically, so all services report 2xx without any manual SQL.

Detection is not instant: the health poller runs every 30s and the incident needs a few poll cycles before the root cause crosses the confidence threshold and the SOP playbook is generated. Measured over repeated runs on a clean stack, budget 1-3 minutes from injecting a fault to an incident reaching TRIAGING, and a similar window for auto-resolution. The result is deterministic (same root cause and confidence every run); only the latency varies with where the injection lands relative to the poll cycle. The demo page shows each phase as it advances, so progress is visible rather than looking stalled.

---

Screenshots

Dashboard — Service Graph

Every service shows live health projected onto the graph by the detection engine.

!Dashboard

Incident Detail — RCA + SOP Playbook

Ranked root-cause candidates, then the SOP phase bar and generated playbook.

!Incident Detail

!SOP Playbook

Demo Page — One-Click Fault Injection

!Demo Page

Fault Injected → Auto-Resolution

ledger-service goes red at 100% error rate while its callers stay healthy:

!Fault Detected

Healing the service auto-resolves the incident and closes every SOP phase:

!Resolved

---

Demo GIF

!Demo

_{Real capture, time-compressed 4x. Detection and auto-verification genuinely take ~90s each, gated on the 30s health-poller interval.}

To regenerate the screenshots and GIF:

docker compose up -d        # stack must be healthy first
npm install                 # installs Playwright (dev-only)
npx playwright install chromium
npm run capture             # or: node scripts/capture-demo.js

The script logs in through Keycloak SSO, resets the simulated services, injects a fault, and waits for the detection engine to actually reach TRIAGING before shooting. It identifies the new incident by snapshotting the pre-existing incident IDs rather than comparing timestamps, since the health poller runs on its own 30s cadence and "when I injected" does not line up with "when the incident was recorded". It then screenshots the RCA and SOP views, heals the service, waits for auto-resolution, and encodes the GIF. Tune GIF_SPEED, GIF_FPS and GIF_WIDTH via the environment.

---

How It Works

Health poller          Detection engine         SOP engine
(polls /health)   →    (BFS + evidence)    →    (phases + playbook)
     ↓                       ↓                        ↓
Signal: ledger-service   Root cause: ledger-service   TRIAGING → FIXING
timeout, err=0.42        confidence: 0.47             Step 1: Check deploy
                         7 signals accumulated        Step 3: AUTO-VERIFY ←── healthy signal

Phases: DETECTING → NARROWING → CONFIRMED → TRIAGING → FIXING → VERIFYING → RESOLVED

Every phase transition is driven by real signal conditions, not timers or manual clicks.

---

Quick Start

git clone https://github.com/[your-org]/FaultIQ
cd FaultIQ
cp .env.example .env   # edit with your values
docker compose up --build

Open http://localhost:4001 · Login: super / superpass

Inject a demo fault in 3 commands:

1. Set ledger-service to timeout
curl -X PUT http://localhost:8091/admin/services/ledger-service/status \
  -H "Content-Type: application/json" \
  -d '{"statusClass":"timeout","errorRate":0.42,"latencyP95":4800}'
2. Watch detection (automatic via health poller, or inject manually)
curl -X POST http://localhost:8085/api/v1/signals \
  -H "Content-Type: application/json" \
  -d '{"tenant":"acme-corp","project":"proj-payments-prod","service":"svc_ledger_service","statusClass":"timeout","errorRate":0.42,"latencyP95":4800}'
3. Restore — system auto-resolves
curl -X PUT http://localhost:8091/admin/services/ledger-service/status \
  -H "Content-Type: application/json" \
  -d '{"statusClass":"2xx","errorRate":0.001,"latencyP95":45}'

Or use the interactive demo at http://localhost:4001/dashboard/demo — no terminal needed.

---

Architecture

| Layer | Technology | Role | |-------|-----------|------| | Service graph | Neo4j + Go | Stores topology + CALLS edges | | Signal ingestion | Go + Redis pub/sub | Receives health signals, rate-limited | | Detection engine | Go (BFS + pgxpool) | Traverses graph, accumulates evidence, runs SOP phases | | RCA ranking | Go (PageRank-style scoring) | Ranks root cause candidates by evidence + propagation | | SOP playbooks | Go + Postgres | Ordered steps with auto-verify conditions | | Code awareness | Go (git clone + AST) | Links incidents to recent commits | | Frontend | Next.js 14 + Cytoscape | Graph visualization, incident detail, SOP panel | | Auth | Keycloak (OIDC) | Multi-tenant JWT auth |

12 Go services. 3 SQL migration files. 1 docker-compose.yml.

---

Onboarding a Project

Define your service topology in a YAML file (supports 15+ services in under 30 seconds):

1. Start FaultIQ
docker compose up --build
2. Import your project via UI:
   → http://localhost:4001/dashboard/onboarding
   → Choose "Import YAML" → paste/upload your service-map.yaml
   → Graph builds automatically
3. Or use a built-in template (E-Commerce, SaaS Analytics, Fintech Payments)

For Kubernetes: annotate your services and let k8s-connector auto-discover:

metadata:
  annotations:
    faultiq.io/health-path: "/health"
    faultiq.io/calls: "svc_ledger,svc_auth"
    faultiq.io/repo: "https://github.com/org/payment-service"

See docs/SERVICE_MAP.md for full format reference including auth patterns (OAuth2, static keys, TCP checks).

---

Features

• Graph-based root cause analysis — BFS traversal with PageRank-style propagation scoring
• SOP state machine — 7-phase pipeline with signal-condition auto-verification
• Playbook learning — step reordering based on historical success rates
• Code-level RCA — links faults to specific git commits via code-indexer
• Deployment health gates — 5-min observation window after CI/CD deploy
• Multi-tenant — full isolation per tenant via Keycloak JWT
• Signal history + SLO tracking — 7-day retention, real uptime% computation
• OTel receiver — auto-builds service graph from distributed traces
• K8s service discovery — via annotations or in-cluster service account

Running Tests

End-to-end test suite (requires running stack)
bash scripts/test-e2e.sh
Go unit tests
for svc in detection-engine api-gateway signal-ingestion rca-ranker; do
  cd services/$svc && go test ./... && cd ../..
done
Frontend TypeScript
cd frontend && npx tsc --noEmit

---

Project Templates

Three ready-to-import templates for common architectures:

| Template | Services | Use case | |----------|---------|----------| | E-Commerce | 12 | Cart, orders, payments, inventory | | SaaS Analytics | 8 | Data pipeline, aggregation, reporting | | Fintech Payments | 11 | Payment processor, fraud detection, ledger |

---

Documentation

| Doc | What it covers | |-----|---------------| | docs/LOCAL_DEMO.md | Running locally and demo walkthrough | | docs/SERVICE_MAP.md | service-map.yaml format + auth patterns | | docs/CODE_INDEXER.md | Git repo integration for code-level RCA | | docs/SECRETS.md | Production secrets management | | docs/REDIS_HA.md | Redis Sentinel high availability |

---

Comparison

| | FaultIQ | Datadog APM | Dynatrace | Causely | |--|---------|------------|-----------|---------| | Graph-based root cause | ✅ | ❌ (per-service alerts) | ✅ (black box) | ✅ (K8s only) | | SOP state machine | ✅ | ❌ | ❌ | ❌ | | Reasoning transparency | ✅ Full chain | ❌ | ❌ | ❌ | | Code commit correlation | ✅ | ✅ ($$$) | ❌ | ❌ | | Open source / self-hosted | ✅ | ❌ | ❌ | ❌ | | Pricing | Free | $100k+/yr | $80k+/yr | Enterprise |

---

Support the Project

If FaultIQ has saved you debugging time, consider buying me a coffee — it helps keep the project alive.


---

Contributing

See CONTRIBUTING.md. We welcome bug reports, feature requests, SOP playbook improvements, and pull requests.

Security

See SECURITY.md for responsible disclosure guidelines.

License

Apache 2.0 — see LICENSE. This means you can use FaultIQ commercially, modify it, and distribute it, as long as you include the license notice.