Architecture & Technology Stack

This page describes the high-level design of the OffloadSecurity CSPM platform. The system is built as a modular monolith using FastAPI and React, supported by a robust asynchronous task engine, a multi-tenant MongoDB data layer, and an autonomous compliance engine.

System Architecture Overview

The platform follows a traditional 3-tier architecture (Frontend, Backend, Database) but extends it with a Docker-in-Docker (DinD) scanning pattern and an asynchronous Celery/Redis pipeline for heavy security workloads.

High-Level Component Diagram

The following diagram illustrates the flow from the user interface through the API layer to the scanning engines and autonomous compliance components, mapping natural language components to specific code entities.

graph TD
    subgraph "Frontend_React"
        UI["MainContent"]
        State["AppStateContext"]
        API_Client["apiService"]
    end

    subgraph "Backend_FastAPI"
        Server["Server"]
        Router["Route Registration"]
        Auth["Auth System"]
        DB_Mgr["CSPMDatabaseManager"]
        DI["ServiceContainer"]
    end

    subgraph "Async_Pipeline"
        Redis[("Redis Broker")]
        Celery["Celery Workers"]
        Scheduler["UnifiedLifecycleScheduler"]
    end

    subgraph "Compliance_Engine"
        ACE["AutonomousComplianceEngine"]
        Drift["ComplianceDriftDetectionService"]
        SCF["CommonControlRegistryService"]
    end

    subgraph "Scanning_Engine_DinD"
        Prowler["UnifiedProwlerService"]
        Tools["Native Tools (ZAP/Trivy/Nmap)"]
    end

    UI --> API_Client
    API_Client --> Server
    Server --> Router
    Router --> Auth
    Server --> DB_Mgr
    Server --> DI
    Server --> Redis
    Redis --> Celery
    Celery --> Prowler
    Celery --> Tools
    Celery --> DB_Mgr
    Celery --> ACE
    ACE --> Drift
    ACE --> SCF
    SCF --> DB_Mgr

Backend: FastAPI & Modular Routing

The backend is a FastAPI application designed for high concurrency using Python's asyncio.

Key Architectural Patterns:

Centralized Route Registration: Instead of declaring all routes in a single entrypoint, the platform uses register_all_routes to mount modular routers. This ensures that a failure in one security module does not prevent the entire platform from starting.
Service Container (DI): The ServiceContainer class manages service dependencies and provides clean dependency injection. It initializes repositories using RepositoryFactory and business logic services with proper repository dependencies.
Structured Logging: Production logs are emitted as JSON via setup_structured_logging to facilitate ingestion into ELK or Loki.
Native Tool Verification: On startup, the system performs a comprehensive check for required binary tools (Docker, Nmap, Trivy) via verify_native_tools_or_fail to ensure the scanning environment is viable.
Observability: The platform exposes Prometheus-compatible metrics via the /metrics endpoint, tracking request latency, error rates, and scan durations.

Data Layer: MongoDB Multi-Database Design

The platform utilizes a Multi-Database Architecture managed by the CSPMDatabaseManager. This provides logical isolation for different security domains and simplifies scaling.

Database Registry

The CSPMDatabaseManager manages connections to distinct databases for each functional module:

Module/Database	Environment Variable	Default Name
Core/Platform	`DB_NAME`	`cspm_platform`
Cloud Security	`CLOUD_SECURITY_DB_NAME`	`cspm_cloud_security`
Risk Management	`RISK_MANAGEMENT_DB_NAME`	`cspm_risk_management`
Assessments	`ASSESSMENT_DB_NAME`	`cspm_assessments`
Container Security	`CONTAINER_SECURITY_DB_NAME`	`cspm_container_security`
AI Governance	`AI_GOVERNANCE_DB_NAME`	`cspm_ai_governance`
Security Scans	`SECURITY_SCANS_DB_NAME`	`cspm_security_scans`
Threat Intelligence	`THREAT_INTELLIGENCE_DB_NAME`	`cspm_threat_intelligence`
Orchestration	`ORCHESTRATION_DB_NAME`	`cspm_orchestration`

Tenancy and Cross-Module References

The architecture enforces strict team-based data isolation. For example, team_control_state stores per-team implementation status for global SCF controls. Cross-module relationships are handled via the create_cross_module_reference helper in the core database.

Autonomous Compliance Engine

The AutonomousComplianceEngine continuously monitors compliance posture by auto-updating SCF (Secure Controls Framework) control status based on scan findings and assessment answers.

Compliance Mapping & Drift

The engine utilizes a sophisticated mapping system to bridge the gap between technical findings and regulatory controls:

Keyword Indexing: Findings are tokenized and matched against control keywords using stem-style prefix matching.
Drift Detection: The ComplianceDriftDetectionService compares current state against snapshots to detect regressions, expired evidence, or threshold crossings.
Cross-Framework Mapping: The ControlMappingEngine allows an answer in one framework (e.g., ISO 27001) to auto-satisfy requirements in others (e.g., SOC 2) via the unified SCF registry.

graph LR
    subgraph "Data_Sources"
        Scan["Scan Findings"]
        Assess["Assessment Answers"]
    end

    subgraph "Mapping_Logic"
        KWM["Keyword Matcher"]
        FWM["Framework Mapper"]
    end

    subgraph "Registry"
        SCF["Common Control Registry"]
        State["Team Control State"]
    end

    Scan --> KWM
    Assess --> FWM
    KWM --> SCF
    FWM --> SCF
    SCF --> State
    State --> Drift["Drift Detection"]

Async Task System: Redis & Celery

The platform relies on Celery for long-running security scans and periodic maintenance tasks.

Event Loop Management

A critical architectural pattern in the platform is the use of CeleryDatabaseManager. Because Celery workers use a ForkPoolWorker strategy, global database singletons become stale. The CeleryDatabaseManager ensures fresh connections are created for every task execution.

Scan Orchestration

The ScanOrchestrationService coordinates between web, container, and cloud scanning modules. It utilizes Redis for caching scan status and Celery for distributed execution.

Microservices Migration Roadmap

The platform is architected to facilitate a transition from a modular monolith to a distributed microservices environment:

Database Decoupling: Each security module already possesses its own database instance in MongoDB, managed via CSPMDatabaseManager.
Service Abstraction: Business logic is encapsulated in services (e.g., WebSecurityService, CloudSecurityService) that are injected via the ServiceContainer, making them easy to move into standalone processes.
Modular API Design: Routers are prefixed and tagged by domain (e.g., /common-controls, /compliance-engine), allowing for easy routing at the API gateway level in the future.

System Architecture Overview​

High-Level Component Diagram​

Backend: FastAPI & Modular Routing​

Key Architectural Patterns:​

Data Layer: MongoDB Multi-Database Design​

Database Registry​

Tenancy and Cross-Module References​

Autonomous Compliance Engine​

Compliance Mapping & Drift​

Async Task System: Redis & Celery​

Event Loop Management​

Scan Orchestration​

Microservices Migration Roadmap​