Request Flows

This document traces three core operations through the entire ThreatWeaver stack, from the frontend UI action to the database write and back.

Flow 1: Start a Scan

This is the most complex flow in ThreatWeaver. It spans the AssessmentWizard UI, the AppSec routes, the PentestCoordinator orchestrator, 56+ attack agents, AI validation, and SSE streaming.

Route Entry Point

From backend/src/routes/appsec.routes.ts, the scan start endpoint is:

POST /api/appsec/assessments/:id/start

This route requires authentication and the appsec module to be enabled (enforced by requireModule('appsec') middleware).

End-to-End Sequence

Pipeline Phases in Detail

The coordinator manages the assessment lifecycle through these states:

queued -> profiling -> planning -> attacking -> validating -> chaining -> completed

From pentestCoordinator.service.ts, each phase maps to specific services:

Phase	State	Service	Purpose
0	Bootstrap	`ResourceBootstrapper`	Budget allocation, auth, target validation
1	Profiling	`targetProfiler.profile()`	Endpoint discovery, crawling
2	Planning	`pentestAiAdapter`	AI attack plan generation
3	Attacking	56+ agent singletons	Parallel vulnerability testing
4	Validating	`validationEngine.validate()`	AI verification, dedup, enrichment
4.5		Sensitive data scan	AI scans HTTP responses for PII/secrets
4.6		Finding analysis	AI enriches critical/high findings
4.7		Remediation	AI generates framework-specific fixes
5	Chaining	`chainBuilder.analyze()`	Exploit chain discovery
5.5		Report narrative	AI executive summary
6	Completed	Finalization	Stats update, webhook notification

SSE Events

The coordinator extends EventEmitter and emits these event types (defined in AssessmentEvent):

type: 'phase_change' | 'progress' | 'agent_start' | 'agent_complete' |
      'agent_error' | 'finding_new' | 'chain_discovered' |
      'assessment_complete' | 'assessment_error' | 'warning' |
      'auth_test_user' | 'auth_test_users_complete' | 'resource_bootstrap'

The frontend subscribes to these via an SSE endpoint and renders live scan progress.

Agent Metrics

After each agent completes, the coordinator collects AgentMetrics:

export interface AgentMetrics {
    agentName: string
    requestsSent: number
    findingsCreated: number
    findingsValidated: number       // True positives
    findingsFalsePositive: number
    findingsDuplicate: number
    durationMs: number
    endpointsTested: number
}

Flow 2: View Dashboard

The dashboard flow is a read-heavy path that aggregates vulnerability data into KPIs, trends, and breakdowns.

Route Entry Point

From backend/src/routes/dashboard.routes.ts:

GET /api/dashboard/kpis
GET /api/dashboard/widget-data  (POST also supported)

Dashboard routes require authentication and the exposure_management module.

End-to-End Sequence

Dashboard widgets use a validated request schema (from dashboard.routes.ts):

const widgetDataSchema = z.object({
    dataSource: z.enum([
        'vulnerabilities', 'assets', 'risk_scores', 'compliance',
        'remediation', 'sla', 'trends', 'appsec'
    ]),
    chartType: z.enum([
        'kpi', 'bar', 'line', 'doughnut', 'gauge', 'table',
        'matrix', 'stacked_bar', 'area', 'pie', 'number'
    ]).optional(),
    groupBy: z.string().max(50).optional(),
    filters: z.record(z.string(), z.unknown()).optional(),
    sort: z.object({
        field: z.string().max(50),
        order: z.enum(['ASC', 'DESC']),
    }).optional(),
    limit: z.number().int().min(1).max(1000).optional(),
})

Rate Limiting

Dashboard endpoints have their own per-user rate limiters, and the general /api/dashboard/* path is excluded from the global rate limiter to support high-traffic polling.

Flow 3: Tenable Sync

The Tenable sync flow imports vulnerability and asset data from Tenable.io using the Export API v2. This is the core data pipeline for the Vulnerability Management module.

Route Entry Point

From backend/src/routes/sync.routes.ts:

POST /api/sync/trigger    -- Start a new sync
GET  /api/sync/status     -- Get current sync status

Sync routes require authentication.

End-to-End Sequence

Sync Status Polling

The frontend polls GET /api/sync/status to track sync progress. The response includes ETA calculation (from sync.routes.ts):

// Only compute ETA after 15% progress -- earlier estimates are too volatile
if (progress > 15 && progress < 100) {
    const elapsedMs = Date.now() - startedAt.getTime()
    const fractionComplete = progress / 100
    const totalEstimatedMs = elapsedMs / fractionComplete
    const remainingMs = totalEstimatedMs - elapsedMs
    const etaSeconds = Math.ceil(remainingMs / 1000)
}

Data Flow Summary

Tenable.io Cloud
  |
  | Export API v2 (paginated chunks)
  v
syncService (normalize + transform)
  |
  | UPSERT (ON CONFLICT)
  v
PostgreSQL
  |
  | Aggregation queries
  v
aggregationService (KPIs, WeaverScore, trends)
  |
  | JSON responses
  v
Frontend Dashboard (React + Recharts)

Key Entities Involved

Entity	Table	Role
`SyncJob`	`sync_jobs`	Tracks sync status, progress, timestamps
`Vulnerability`	`vulnerabilities`	Normalized vulnerability records from Tenable
`Asset`	`assets`	Discovered assets (servers, workstations, etc.)
`VulnerabilityStats`	`vulnerability_stats`	Pre-computed aggregation statistics

Flow 4: Webhook Notification

When a scan completes, ThreatWeaver fires outbound webhook notifications to any registered receivers.

Trigger

Emitted from pentestCoordinator.service.ts after Phase 5 completes:

assessment status → 'completed'
        │
        ▼
sendScanCompleteNotification(assessmentId)

Sequence

Payload Schema

{
  event: 'assessment.completed',
  assessmentId: string,         // UUID
  targetUrl: string,
  scanType: 'blackbox' | 'graybox' | 'whitebox',
  status: 'completed' | 'failed',
  findingsCount: number,
  criticalCount: number,
  highCount: number,
  completedAt: string,          // ISO 8601
}

Signature Verification (receiver side)

import hmac, hashlib
expected = hmac.new(webhook_secret.encode(), payload_bytes, hashlib.sha256).hexdigest()
assert request.headers['X-TW-Signature'] == expected

Flow 5: Error Handling

ThreatWeaver uses a layered error handling strategy — validation at the boundary, structured errors in services, and a global error middleware that sanitizes responses.

Layer 1 — Input Validation (Zod)

All mutating endpoints validate the request body with Zod before any business logic runs:

// Example from dashboard.routes.ts
const widgetDataSchema = z.object({
  dataSource: z.enum(['vulnerabilities', 'assets', ...]),
  limit: z.number().int().min(1).max(1000).optional(),
})
// Invalid input → 400 Bad Request { field, message } — no stack trace exposed

Layer 2 — Auth & Tenant Errors

JWT missing/expired   → 401 Unauthorized
Role insufficient     → 403 Forbidden
Tenant suspended      → 403 { error: 'Tenant account suspended' }
Tenant deprovisioned  → 410 Gone
Tenant not found      → 404 Not Found
TLM unavailable       → falls back to local tenant_user_routing table
DB fallback fails     → 502 Bad Gateway

Layer 3 — Service Errors

Services throw typed errors that the global error handler catches:

Layer 4 — Tenant Schema Errors

If the DB schema is unavailable (e.g., schema not yet provisioned):

// set-tenant-schema.ts
} catch (err: any) {
  console.error('Failed to set tenant schema:', err.message)
  return res.status(503).json({ error: 'Database schema not available' })
}

Error Response Format

All error responses follow this shape — no stack traces, no internal paths, no DB details:

{
  error: string,           // Human-readable message (safe to display)
  field?: string,          // Only present for validation errors
  code?: string,           // Optional error code for programmatic handling
}

Request Lifecycle Summary

Every API request in ThreatWeaver passes through this middleware chain (in order):

Helmet (security headers)
Permissions-Policy header
CORS
detectBaseUrl
Rate limiter (global)
Auth-specific rate limiters (login, MFA, etc.)
Body parser (JSON, 1MB limit)
Cookie parser
Body error handler (malformed JSON, too large)
JSON depth limiter (max 20 levels)
Decryption middleware (double encryption)
CSRF Origin validation (mutating requests)
Multi-tenant middleware chain:
    a. authenticate (JWT verification)
    b. resolveTenant (lookup schema from TLM)
    c. setTenantSchema (SET search_path on QueryRunner)
    d. enforceTenancy (license validation)
    e. tenantRateLimiter (per-tenant limits)
    f. usageMeter (request counting)
setupModeGuard (redirect if setup incomplete)
Route handler

Flow 1: Start a Scan​

Route Entry Point​

End-to-End Sequence​

Pipeline Phases in Detail​

SSE Events​

Agent Metrics​

Flow 2: View Dashboard​

Route Entry Point​

End-to-End Sequence​

Widget Data Schema​

Rate Limiting​

Flow 3: Tenable Sync​

Route Entry Point​

End-to-End Sequence​

Sync Status Polling​

Data Flow Summary​

Key Entities Involved​

Flow 4: Webhook Notification​

Trigger​

Sequence​

Payload Schema​

Signature Verification (receiver side)​

Flow 5: Error Handling​

Layer 1 — Input Validation (Zod)​

Layer 2 — Auth & Tenant Errors​

Layer 3 — Service Errors​

Layer 4 — Tenant Schema Errors​

Error Response Format​

Request Lifecycle Summary​

Flow 1: Start a Scan

Route Entry Point

End-to-End Sequence

Pipeline Phases in Detail

SSE Events

Agent Metrics

Flow 2: View Dashboard

Route Entry Point

End-to-End Sequence

Widget Data Schema

Rate Limiting

Flow 3: Tenable Sync

Route Entry Point

End-to-End Sequence

Sync Status Polling

Data Flow Summary

Key Entities Involved

Flow 4: Webhook Notification

Trigger

Sequence

Payload Schema

Signature Verification (receiver side)

Flow 5: Error Handling

Layer 1 — Input Validation (Zod)

Layer 2 — Auth & Tenant Errors

Layer 3 — Service Errors

Layer 4 — Tenant Schema Errors

Error Response Format

Request Lifecycle Summary