Module Architecture & Data Flows

This document is the definitive reference for how ThreatWeaver's backend is structured internally. It covers the global request lifecycle that every API call passes through, then dedicates a full section to each of the 12 major modules: what it does, which entities it owns, which external systems it touches, and how it connects to neighbouring modules.

Audience

This page is written for backend engineers, technical leads, and security reviewers. If you want the high-level picture first, start with Architecture Diagrams. If you want individual request traces, see Request Flows.

---

How ThreatWeaver's Modular Architecture Works

ThreatWeaver is a monolithic Express application split into functional modules. Each module maps to:

• One or more route files (backend/src/routes/*.routes.ts)
• One or more service files (backend/src/services/)
• A set of TypeORM entities it "owns" (primary read/write responsibility)
• Optional external API dependencies
• Optional module entitlement gating (e.g., requireModule('appsec'))

Modules are not micro-services — they share the same Node.js process, the same PostgreSQL connection pool, and the same TypeORM DataSource. What isolates them is:

1. Entitlement middleware — routes check whether the authenticated tenant's license includes the module before processing any request.
2. Database schema isolation — all DB queries go through getTenantRepository(), which sets search_path to the tenant's private PostgreSQL schema before any query executes.
3. Rate limiting per module — different modules have different rate limits (e.g., AppSec reads allow 2,000 req/min vs. 100 req/min global).

The result is a system where modules can be independently enabled, independently rate-limited, and independently audited — while still sharing infrastructure for efficiency.

---

Global Request Lifecycle

Every authenticated API request passes through this middleware stack before any route handler runs. This is defined in backend/src/index.ts and applies globally.

Full Middleware Order (from index.ts)

Step	Middleware	Purpose
1	Trust Proxy	Enable req.ip behind Render's load balancer
2	Helmet	CSP, HSTS, X-Frame-Options, referrer-policy
3	Permissions-Policy	Restrict browser feature access
4	CORS	Origin allowlist (frontend + agent origins)
5	Global Rate Limiter	100 req/min per IP — DoS protection
6	Auth-specific rate limiters	Login: 10/15 min; MFA: 5/15 min
7	Body Parser (JSON)	1 MB payload cap; reject malformed JSON
8	Cookie Parser	Parse HttpOnly JWT cookies
9	Body Error Handler	Return 400 on malformed JSON or oversized body
10	JSON Depth Limiter	Reject objects nested > 20 levels (DoS protection)
11	Decryption Middleware	Decrypt E2EE payloads (double-encrypted from frontend)
12	CSRF Validation	Block state-mutating requests without valid Origin header
13a	authenticate	Verify JWT from cookie or Authorization header
13b	resolveTenant	Resolve tenant ID → schema name from JWT claim
13c	setTenantSchema	SET search_path = <tenant_schema> on query runner
13d	enforceTenancy	Validate license + module entitlement
13e	tenantRateLimiter	Per-tenant request caps from entitlement config
13f	usageMeter	Increment usage counter for billing/audit
14	setupModeGuard	Redirect to setup wizard if initial setup is incomplete
15	Route Handler	Module-specific logic executes

Exceptions: Routes under /auth/*, /internal/*, and /agent/* skip steps 13a–13f. The /auth/* routes have their own tighter rate limits. /internal/* uses X-Internal-API-Key timing-safe comparison. /agent/* uses mutual TLS + certificate authentication.

---

Module 1: Authentication

What It Does

The Authentication module handles all identity operations: user login, token issuance, refresh, logout, MFA enrollment and verification, and session management. It is the entry point for all human users before any other module is accessible. It maintains an IP allowlist/blocklist evaluated on every login attempt, and records every authentication event to a security audit log for compliance.

Owned DB Entities

Entity	Purpose
User	User accounts, bcrypt-hashed passwords, role assignments, MFA secrets
TokenBlacklist	Revoked JWTs (stored until natural expiry to prevent reuse)
IpWhitelist	Trusted IPs that bypass rate limiting
IpBlacklist	Blocked IPs (manual + automatic after brute-force detection)
SecurityAuditLog	Immutable record of every auth event (login, logout, MFA, password change)

External Dependencies

None. Authentication is fully self-contained within ThreatWeaver. SSO (Microsoft Entra ID) lives in the SSO module, which delegates token exchange back to this module's session system.

Key Security Considerations

• All passwords use bcrypt with a minimum cost factor of 12.
• JWTs are issued as HttpOnly; Secure; SameSite=Strict cookies — not accessible to JavaScript.
• Tokens contain tenantId, userId, role, and a jti (unique token ID) for revocation tracking.
• After 5 consecutive failed logins, the source IP is automatically added to the blocklist.
• The /auth/* path is excluded from the multi-tenant middleware chain (unauthenticated users have no tenant context yet).

Cross-Module Dependencies

• SSO Module delegates user provisioning and session issuance back to Auth.
• Security Audit Module queries SecurityAuditLog (written by Auth) for the admin security dashboard.
• Admin Panel manages User and IpWhitelist/Blacklist entities.

---

Module 2: Sync Engine (Tenable Integration)

What It Does

The Sync Engine is ThreatWeaver's data ingestion pipeline. It connects to the Tenable.io Cloud API (Export API v2), streams vulnerability and asset data in paginated chunks, normalizes the Tenable schema into ThreatWeaver's internal schema, and upserts the results into PostgreSQL. After each successful sync, it triggers the aggregation service to recompute all KPIs and trend data. Syncs are tracked as SyncJob records with full progress visibility via SSE.

Owned DB Entities

Entity	Purpose
SyncJob	Tracks every sync run — status, start/end time, record counts, error messages
Audit	General audit log entries written during sync for compliance

External Dependencies

• Tenable.io Cloud API (Export API v2) — requires TENABLE_ACCESS_KEY and TENABLE_SECRET_KEY environment variables stored encrypted in AppConfig.
• Tenable API keys are decrypted at runtime by the API Config module.

Key Security Considerations

• Sync is admin-only. The usageMeter middleware tracks each sync against the tenant's monthly quota.
• tenantRateLimiter prevents sync abuse (e.g., triggering dozens of concurrent exports).
• Tenable API credentials are never stored in plaintext — they live in AppConfig encrypted with AES-256.

Cross-Module Dependencies

• API Config Module — Sync reads Tenable credentials from AppConfig.
• Dashboard Module — Sync feeds data to Vulnerability and Asset tables that the Dashboard reads.
• Assets Module — Upserted Asset records become queryable via the Assets API.
• Vulnerabilities Module — Upserted Vulnerability records power all vulnerability views.
• Progress/SSE Module — Progress events are streamed back to the UI via the SSE endpoint.

---

Module 3: Dashboard

What It Does

The Dashboard module is the primary read path for the Vulnerability Management workspace. It aggregates vulnerability and asset data into KPIs (total vulns, critical count, WeaverScore, trend direction), serves widget data for customizable charts, and provides trend snapshots for time-series analysis. It is a read-only module — it never writes vulnerability or asset data directly.

Owned DB Entities

Entity	Purpose
TrendSnapshot	Point-in-time snapshots of KPIs for trend charts (saved after each sync)

Read Entities (owned by other modules)

Vulnerability, Asset (owned by their respective modules — Dashboard reads only).

External Dependencies

None. Dashboard is a pure internal aggregation layer.

Key Security Considerations

• requireModule('vulnerability_dashboard') — tenants without this module entitlement receive 403.
• Cache headers are set on KPI responses to reduce DB load on high-traffic deployments.
• Widget data queries use parameterized queries via TypeORM; the groupBy field is validated against an allowlist before being interpolated.

Cross-Module Dependencies

• Sync Engine — must run first to populate Vulnerability and Asset tables.
• Vulnerabilities Module — shares the Vulnerability table.
• Assets Module — shares the Asset table.
• Export Module — the Dashboard's aggregated data can be exported via the Export module.

---

Module 4: AppSec Scanner

What It Does

The AppSec Scanner is ThreatWeaver's automated penetration testing engine and the largest, most complex module in the codebase. It manages the full assessment lifecycle: target definition, authentication profiling, six-phase scanning (Phase 0 bootstrap through Phase 5 chaining + reporting), 56+ attack agents, AI-powered finding validation, and PDF report generation. It exposes SSE streams for real-time progress, supports CI/CD integration via API key authentication, and can send webhook callbacks when scans complete.

Owned DB Entities

Entity	Purpose
PentestAssessment	Top-level scan container — target, config, status, phase, timing, stats
PentestFinding	Discovered vulnerability: type, severity, CVSS, CWE, evidence refs, validation status
PentestTarget	Application/API being tested: URL, tech stack, scan mode (black/gray/white box)
PentestEvidence	Raw request/response payloads proving a finding (stored encrypted)
PentestAuthProfile	Authentication credentials and session config per target
PentestReport	Generated PDF/HTML pentest reports (S3 or DB storage)
PentestExploitChain	Multi-step attack chains linking related findings
PentestFindingObservation	FP heuristic results and per-probe confidence data
PentestAgentLog	Per-agent execution logs (timing, endpoint count, errors)
PentestCrawlResult	Discovered endpoints from Phase 1 crawling
Phase0Session	Interactive Phase 0 Q&A session data (sector, context hints)
PentestAgentConfig	Per-agent tuning (timeouts, payload lists, thresholds)

External Dependencies

• AI Providers (Claude/GPT) — used for finding validation, CVSS enrichment, remediation generation, executive summary narrative, and exploit chain analysis. Configured via AiProvider entity.
• OOB Callback Server (optional) — SSRF/OAST probes use an out-of-band DNS/HTTP callback server (INTERACTSH_URL env var).

Key Security Considerations

• Read endpoints: 2,000 req/min. Write endpoints: 120 req/min. Prevents scan flooding.
• requireModule('appsec') — only tenants with the AppSec module can create assessments.
• All outbound HTTP probe requests go through validateOutboundUrl() — prevents SSRF from the scanner itself being weaponized.
• Evidence payloads (raw HTTP request/response) may contain secrets from the target; they are stored with restricted access and stripped of known-sensitive headers before display.

Cross-Module Dependencies

• CI/CD Scan Module — delegates scan creation and start to AppSec entities.
• Delta Scanning Module — reads AppSec assessment data read-only.
• Webhook Receivers — GitHub/GitLab pushes trigger AppSec assessments.
• Reports Module — reads PentestFinding and PentestAssessment for compliance report generation.
• Notifications — scan completion triggers notification events.

---

Module 5: CI/CD Scan Integration

What It Does

The CI/CD Scan module provides a headless, session-free interface for triggering AppSec scans from CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins). Unlike the main AppSec module which requires JWT-authenticated browser sessions, this module authenticates entirely via a static X-API-Key header validated with a timing-safe comparison. It exposes minimal endpoints: create a scan, check status, retrieve findings summary — exactly what a CI job needs.

Owned DB Entities

PentestAssessment, PentestTarget (shared with AppSec — CI/CD creates assessments in the same tables).

External Dependencies

None. Delegates all scanning logic to the AppSec module's coordinator.

Key Security Considerations

• No session cookies or JWTs involved — purely API key based for CI compatibility.
• requireApiKey uses crypto.timingSafeEqual() to prevent timing side-channel attacks.
• API keys are stored hashed (SHA-256) in the database — never retrievable in plaintext after creation.
• The tenant is derived from the API key itself (keys are scoped per-tenant at creation time).

Cross-Module Dependencies

• AppSec Scanner — CI/CD Scan is a thin facade over AppSec's coordinator.
• Webhook Receivers — complementary entry point (GitHub push → webhook → this module for automated trigger).

---

Module 6: Webhook Receivers

What It Does

The Webhook Receivers module handles inbound webhooks from external source code platforms — primarily GitHub and GitLab. When a push event arrives (e.g., a merge to main), the module verifies the webhook signature, identifies the associated PentestTarget, creates a new assessment, and triggers the AppSec scanner. This enables fully automated security scanning on every code push with zero human intervention.

Owned DB Entities

PentestTarget (read — maps repo URL to target), PentestAssessment (creates new assessment per push).

External Dependencies

• GitHub — sends push, pull_request, release events signed with HMAC-SHA256.
• GitLab — sends equivalent events with a plain X-Gitlab-Token secret header.

Key Security Considerations

• GitHub: HMAC-SHA256 signature verification against X-Hub-Signature-256 header. Rejects any request that doesn't match.
• GitLab: Plain token comparison using timing-safe equality.
• Event payloads are never logged in plaintext (may contain source code metadata).
• Only registered PentestTarget repos can trigger scans — unknown repos are silently ignored with 200 OK (prevents enumeration).

Cross-Module Dependencies

• AppSec Scanner — creates assessments and starts scans using the AppSec coordinator.
• CI/CD Scan Module — similar purpose; Webhook Receivers is the push-triggered entry point, CI/CD Scan is the pull/polling entry point.

---

Module 7: SSO (Microsoft Entra ID)

What It Does

The SSO module implements OAuth2 authentication via Microsoft Entra ID (formerly Azure AD). When a user clicks "Sign in with Microsoft," this module handles the authorization code exchange, validates the Microsoft-issued JWT, provisions the user in ThreatWeaver (or updates their profile if they already exist), and issues a ThreatWeaver session cookie via the Auth module. It also manages tenant membership — assigning SSO users to the correct tenant based on their Azure tenant ID.

Owned DB Entities

Entity	Purpose
User	Created/updated on first SSO login (shared with Auth module)
tenant_memberships	Maps SSO users to their ThreatWeaver tenant
tenant_configs	Per-tenant SSO configuration (client ID, tenant ID, redirect URIs)

External Dependencies

• Microsoft Entra ID — OAuth2 authorization server. Requires AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID per-tenant configuration stored in tenant_configs.

Key Security Considerations

• PKCE (code_verifier / code_challenge) is used — protects against authorization code interception.
• state parameter is a cryptographically random nonce stored in session — validated on callback to prevent CSRF.
• resolveLimiter rate limiter on the callback endpoint prevents token flooding.
• Microsoft JWTs are validated against Microsoft's JWKS endpoint (signature + audience + expiry).

Cross-Module Dependencies

• Auth Module — SSO delegates final session issuance (JWT cookie creation) to the Auth module.
• Admin Panel — admins manage tenant_configs for SSO setup.
• Setup Module — SSO configuration can be completed via the Setup wizard.

---

Module 8: Admin Panel

What It Does

The Admin Panel module exposes all tenant administration capabilities: user management (create, update, deactivate, assign roles), global settings, role and permission management, tenant configuration, and archived data management. Every route requires at least JWT authentication plus requirePermission() middleware for fine-grained authorization — no admin route is accessible to non-admin users.

Owned DB Entities

Entity	Purpose
User	User accounts managed via admin UI (shared with Auth)
Role	Named role definitions (admin, analyst, viewer, etc.)
Settings	Tenant-level configuration (feature flags, notification prefs, branding)
ArchivedData	Soft-deleted records preserved for audit trail

External Dependencies

None. Admin is a fully internal management layer.

Key Security Considerations

• Zod strict validation on all write endpoints — extra fields are rejected (not ignored), preventing mass-assignment attacks.
• requirePermission() is granular — a user with manage_users cannot access manage_settings without that separate permission.
• All admin mutations are written to SecurityAuditLog.
• Self-demotion protection: admins cannot remove their own admin role via the API.

Cross-Module Dependencies

• Auth Module — manages User entity also used by Auth.
• SSO Module — admin configures tenant_configs for SSO.
• API Config Module — admin manages Tenable API credentials stored in AppConfig.
• Security Audit Module — admin actions are logged to SecurityAuditLog.

---

Module 9: API Configuration

What It Does

The API Config module manages external API credentials — primarily Tenable API keys — stored with AES-256 encryption. Credentials are never stored in plaintext. The module provides endpoints to save, test, and retrieve (masked) credential configuration. The encryption key is derived from a server-side secret, meaning database compromise alone cannot expose credentials.

Owned DB Entities

Entity	Purpose
AppConfig	Key-value store for tenant configuration; sensitive values stored AES-256 encrypted

External Dependencies

• Tenable API — used only for the "test connection" endpoint.

Key Security Considerations

• Encryption key is stored in APP_ENCRYPTION_KEY environment variable — never in the database.
• AES-256-GCM provides both confidentiality and integrity (authenticated encryption).
• Credentials are only decrypted in-memory, at the moment they are needed (e.g., during sync), never written decrypted.
• GET endpoints for credential retrieval return only masked values (e.g., ****abcd).

Cross-Module Dependencies

• Sync Engine — primary consumer; reads Tenable credentials at sync time.
• Admin Panel — exposes the UI for managing AppConfig entries.

---

Module 10: Security Audit

What It Does

The Security Audit module provides a read-only window into the security event log for administrators and security reviewers. It surfaces all authentication events, admin actions, permission changes, and anomalous access patterns from SecurityAuditLog. It also exposes management interfaces for the IP allowlist and blocklist — the dynamic security controls that Auth evaluates on every login.

Owned DB Entities

Entity	Purpose
SecurityAuditLog	Immutable event log (written by Auth, read by Security Audit module)
IpWhitelist	Trusted IPs that bypass auth rate limits
IpBlacklist	Blocked IPs (can be manually added/removed here)

External Dependencies

None. Security Audit is a pure read/management layer over the security event log.

Key Security Considerations

• All read endpoints require view_security permission — not all admin users have this.
• SecurityAuditLog entries are append-only by design — there is no DELETE endpoint for log entries.
• IP allowlist/blocklist changes are themselves logged to SecurityAuditLog.

Cross-Module Dependencies

• Auth Module — writes all events that Security Audit reads.
• Admin Panel — provides the UI surfacing Security Audit data.

---

Module 11: Scanner/Sensor (Distributed Agents)

What It Does

The Scanner/Sensor module manages distributed Docker-based scan sensors — remote agents that can be deployed in isolated network segments to reach targets not accessible from ThreatWeaver's cloud backend. Sensors establish a persistent WebSocket tunnel to the backend's /ws/agent endpoint, authenticate with mutual TLS plus a certificate, and receive scan tasks. Results stream back through the same tunnel. The module handles agent enrollment, heartbeat monitoring, task dispatch, and reconnection logic.

Owned DB Entities

Entity	Purpose
ScannerConfig	Global scanner configuration (concurrency, timeouts, allowed probe types)
ScanSensor	Registered remote sensor record (hostname, network segment, enrollment status)
ScanSensorHeartbeat	Health check records — used to detect offline sensors
ScannerEnrollmentToken	One-time tokens issued for sensor enrollment
ScannerAgent	Agent definition records (name, category, capabilities, enabled flag)
ScanAgentTask	Individual task records assigned to agents

External Dependencies

None external. Sensors communicate directly with the backend over WebSocket.

Key Security Considerations

• mTLS: sensors must present a valid client certificate signed by ThreatWeaver's internal CA.
• antiReplay middleware: each WebSocket message includes a monotonically increasing sequence number; replayed messages are rejected.
• authenticateAgent validates both the certificate and the enrollment token before admitting a sensor.
• Sensor-to-backend communication is WebSocket over TLS only — no plain HTTP fallback.

Cross-Module Dependencies

• AppSec Scanner — the coordinator dispatches tasks to remote sensors and receives findings through this module's gateway.
• Scan Management Module — users manage sensors and view sensor health via the Scan Management UI.

---

Module 12: Reporting

What It Does

The Reporting module handles scheduled and on-demand report generation. Reports can be compliance-focused (PCI-DSS, SOC2, ISO 27001 mapping) or operational (vulnerability trends, asset risk summaries). Reports are generated as PDF or HTML documents, stored, and optionally emailed. The module supports scheduling (e.g., weekly PDF to CISO) and ad-hoc generation from the UI. XSS stripping is applied to all user-supplied text before it enters the report template.

Owned DB Entities

Entity	Purpose
ReportSchedule	Recurring report definitions (cron schedule, template, recipients)
ReportJob	Individual report generation jobs with status, output location, error info

External Dependencies

• Email service — SMTP-based email delivery for scheduled report distribution (configured via Settings).

Key Security Considerations

• requireModule('compliance_reporting') — reports are a licensed feature.
• XSS stripping on all user-provided strings before template rendering prevents stored-XSS in generated documents.
• Generated PDFs are stored with tenant-scoped paths — cross-tenant access to report files is prevented at the storage layer.
• Report download URLs are short-lived (signed URLs or session-gated endpoints).

Cross-Module Dependencies

• AppSec Scanner — reads PentestFinding and PentestAssessment for pentest reports.
• Vulnerabilities Module — reads Vulnerability and Asset for operational reports.
• Compliance entities — reads ComplianceMapping for compliance-focused reports.
• Admin Panel — report schedule management is accessible via Admin UI.

---

Cross-Module Data Flows

The following diagram shows how data flows between modules over the lifetime of a typical enterprise deployment.

Key Cross-Module Integration Points

Source Module	Target Module	Data Transferred	Trigger
Sync Engine	Dashboard	Vulnerability, Asset records	After each sync completes
Sync Engine	Vulnerabilities	Vulnerability upserts	Continuous during sync
AppSec Scanner	Reports	PentestFinding, PentestAssessment	On-demand or scheduled
AppSec Scanner	VFP	PentestFinding enriched with CVSS	After scan completion
Auth	Security Audit	SecurityAuditLog events	Every auth operation
Webhook Receivers	AppSec Scanner	New PentestAssessment	On push event receipt
CI/CD Scan	AppSec Scanner	New PentestAssessment	On API call from CI
API Config	Sync Engine	Decrypted Tenable credentials	At sync start
SSO	Auth	Provisioned User record	On first SSO login

---

Tenant Isolation: Schema-Per-Tenant

Every data-accessing module uses getTenantRepository() instead of the global TypeORM repository. This single function call is the enforcement point for ThreatWeaver's schema-per-tenant isolation model.

How It Works

1. The JWT contains a tenantId claim.
2. The resolveTenant middleware maps tenantId → schemaName (e.g., tenant_abc123).
3. The setTenantSchema middleware executes SET search_path = tenant_abc123, public on the PostgreSQL connection before any query runs.
4. All TypeORM queries on that connection automatically resolve to the tenant's private schema.
5. There is no need to add WHERE tenant_id = ? to queries — isolation is enforced at the connection level.

What Is Shared vs. Isolated

Data	Isolation Level	Location
User vulnerability data	Full schema isolation	tenant_abc123.vulnerabilities
AppSec findings	Full schema isolation	tenant_abc123.pentest_findings
Assets	Full schema isolation	tenant_abc123.assets
Tenant definitions	Shared (public schema)	public.tenants
Module entitlements	Shared (public schema)	public.tenant_entitlements
AI provider configs	Shared (public schema)	public.ai_providers
Global scanner configs	Shared (public schema)	public.scanner_agents

Bypassed Paths

The multi-tenant middleware chain (steps 13a–13f) is skipped for:

• /auth/* — user has no tenant context yet
• /internal/* — internal service-to-service calls using X-Internal-API-Key
• /agent/* — sensor WebSocket connections authenticated by mTLS, not JWT

These paths use their own authentication mechanisms and explicitly scope queries to the tenant derived from their respective auth tokens.

---

Additional Modules (Summary Reference)

The following modules follow the same patterns as the 12 detailed above. They are documented here for completeness.

Module	Route File	Primary Purpose	Auth
Assets	assets.routes.ts	Asset inventory management; mirrors Tenable asset data	JWT + requireModule('vulnerability_dashboard')
Vulnerabilities	vulnerabilities.routes.ts	Vulnerability CRUD, state transitions, bulk operations	JWT + requireModule + readLimiter / writeLimiter
Anomaly Detection	anomaly.routes.ts	AI-powered anomaly events and exclusion management	JWT + requireModule('intelligence')
Asset Ownership	assetOwnership.routes.ts	CSV-driven asset ownership assignment	JWT + file upload validation
OS Category Rules	osCategoryRules.routes.ts	Custom rules mapping OS strings to categories	JWT + XSS strip
Delta Scanning	delta.routes.ts	Read-only diff view of assessment changes between runs	JWT + requireModule('appsec')
Progress/SSE	progress.routes.ts	Real-time sync progress stream	JWT + SSE (setMaxListeners(50))
Export	export.routes.ts	Streaming CSV/JSON exports of vuln and asset data	JWT + requireModule + streaming response
AI Security	aiSecurity.routes.ts	AI tool inventory, risk tracking, policy enforcement	JWT + requireModule('ai_security')
Setup/License	setup.routes.ts	Initial setup wizard and license validation (TLM)	Public for setup; JWT for license updates
Internal API	internal.routes.ts	Service-to-service calls (provisioning, webhooks)	X-Internal-API-Key timing-safe comparison
Scan Management	scan.routes.ts	Scan scheduler, credential vault, job history	JWT + requireModule('scanner_management') + validateOutboundUrl