# noobaa/noobaa-core architecture

> Architecture diagram of the GitHub repository noobaa/noobaa-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noobaa/noobaa-core
- Repository: https://github.com/noobaa/noobaa-core

## Overview

This project – NooBaa – is a highly modular, cloud data service that offers an S3 object-store interface over a variety of storage backends such as public clouds (AWS S3, GCS, Azure Blob), on-premise filesystems, and more. The system is designed to layer cloud management with dynamic data placement, tiering, and replication across heterogeneous storage silos. In other words, it abstracts various storage resources behind a unified S3-compatible API while also including management, cluster, and background service layers.

##### High-Level Architecture and Main Components
- Client Interfaces and CLI: This includes the NooBaa CLI which is used for deploying, monitoring, and managing the system (as indicated by the README’s “install,” “status,” and “uninstall” commands).
- API/Endpoint Layer: A number of API modules exist (e.g., the “src/api” directory, “src/endpoint” for S3, IAM, and lambda endpoints) which expose the S3 interface and service endpoints.
- Data Services and Storage Abstraction: At the core, there is an abstraction layer that connects to various storage backends (“src/agent”, “src/sdk”, and “src/native”). This module incorporates different block storage services (e.g., for Azure, Google, Filesystems, etc.) and provides flexible data placement, deduplication, and encryption.
- Background and System Services: The project includes several “bg_services” under “src/server/bg_services” and “system_services” components in “src/server/system_services”. These handle asynchronous tasks (replication, bucket log aggregation, scrubbing, and shard/replica management) and cluster-level functions.
- Management and Orchestration: The “src/server” directory comprises modules for handling cluster management, node orchestration, replication, and licensing.
- Native and Performance-Critical Code: There is significant native code (in “src/native”), offering optimized functionality (e.g., for chunking, compression, encryption, and hashing) to increase performance.
- Testing and Deployment Support: The file structure shows dedicated directories for tests (unit, system, QA), deployment scripts (Dockerfiles, Kubernetes related configs in “src/deploy”), and upgrade/migration scripts (in “src/upgrade”).

##### Interactions and Data Flows
- How the API endpoints (S3, IAM, Lambda, etc.) forward certain requests to the backend components – for example, agents that transform and write data to underlying storage systems.
- Flows that cover background replication and lifecycle tasks (triggered by components in bg_services and system_services) and monitor health/status data.
- Native performance modules with the data path, indicating that they are “plugged in” at the lower layers to handle intensive tasks like content chunking, encryption and fast file/storage operations.
- Communication between the management/control plane and the distributed storage agents responsible for interacting with local filesystems or third-party cloud storage services.

##### Architectural Patterns and Design Principles
- Microservices / Modular design: The structure clearly separates functionality by concerns (API, agent, background tasks, storage SDK, system upgrades, etc.) which allows components to be scaled and maintained independently.
- Abstraction Layer: The design decouples the S3 API layer from the actual storage backends, allowing data placement to be controlled by policies.
- Asynchronous and Event-Driven Processing: Background services and job/task queues (as seen in “bg_services” and “func_services”) indicate a strong emphasis on asynchronous processing.
- Extensibility and Pluggability: The existence of various modules (for different cloud providers, native libraries, and test harnesses) shows how the system is built to integrate new storage types or functionalities without significant re-engineering.

- Annotations: For key connections (e.g., “Policy-driven data placement”, “Event-based replication”, “CLI Deployment & Operator Integration”), add annotations to further explain the design.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client & CLI
    subgraph "Client & CLI"
        CLI_READ["NooBaa README"]:::external
        CLI_DOC["CLI Docs"]:::external
        CLI_SCRIPTS["CLI Scripts"]:::external
    end

    %% API / Endpoint Layer
    subgraph "API / Endpoint Layer"
        API_MOD["API Modules"]:::api
        ENDPOINT["Endpoint Implementations"]:::api
        S3_API["S3 API Gateway"]:::api
        IAM["IAM Endpoint"]:::api
        LAMBDA["Lambda Endpoint"]:::api
    end

    %% Data Services & Storage Abstraction
    subgraph "Data Services & Storage Abstraction"
        AGENT["Storage Agents"]:::internal
        SDK["SDK Interface"]:::internal
        NATIVE["Native Functionality"]:::performance
    end

    %% Background & System Services
    subgraph "Background & System Services"
        BG["Background Services"]:::background
        SYS["System Services"]:::background
        FUNC["Function Services"]:::background
    end

    %% Management & Orchestration
    subgraph "Management & Orchestration"
        MGMT["Cluster Management & Control"]:::management
    end

    %% Testing & Deployment Support
    subgraph "Testing & Deployment Support"
        TEST["Tests"]:::testing
        DEPLOY["Deployment Scripts"]:::testing
        UPGRADE["Upgrade & Migration Scripts"]:::testing
    end

    %% Connections
    CLI_SCRIPTS -->|"request"| S3_API

    API_MOD -->|"implements"| S3_API
    API_MOD -->|"implements"| IAM
    API_MOD -->|"implements"| LAMBDA
    API_MOD -->|"integrates"| ENDPOINT

    S3_API -->|"forwards"| AGENT
    S3_API -->|"triggers"| BG
    S3_API -->|"management"| MGMT

    AGENT -->|"utilizes"| SDK
    SDK -->|"accelerates"| NATIVE

    MGMT -->|"orchestrates"| AGENT

    SYS -->|"reports"| BG
    BG -->|"executes"| FUNC

    %% Click Events
    click CLI_READ "https://github.com/noobaa/noobaa-core/blob/master/README.md"
    click CLI_DOC "https://github.com/noobaa/noobaa-core/blob/master/docs/NooBaaNonContainerized/NooBaaCLI.md"
    click CLI_SCRIPTS "https://github.com/noobaa/noobaa-core/tree/master/src/cmd"

    click API_MOD "https://github.com/noobaa/noobaa-core/tree/master/src/api"
    click ENDPOINT "https://github.com/noobaa/noobaa-core/tree/master/src/endpoint"

    click AGENT "https://github.com/noobaa/noobaa-core/tree/master/src/agent"
    click SDK "https://github.com/noobaa/noobaa-core/tree/master/src/sdk"
    click NATIVE "https://github.com/noobaa/noobaa-core/tree/master/src/native"

    click BG "https://github.com/noobaa/noobaa-core/tree/master/src/server/bg_services"
    click SYS "https://github.com/noobaa/noobaa-core/tree/master/src/server/system_services"
    click FUNC "https://github.com/noobaa/noobaa-core/tree/master/src/server/func_services"

    click MGMT "https://github.com/noobaa/noobaa-core/tree/master/src/server"

    click TEST "https://github.com/noobaa/noobaa-core/tree/master/src/test"
    click DEPLOY "https://github.com/noobaa/noobaa-core/tree/master/src/deploy"
    click UPGRADE "https://github.com/noobaa/noobaa-core/tree/master/src/upgrade"

    %% Styles
    classDef external fill:#AEDFF7,stroke:#333,stroke-width:2px;
    classDef api fill:#FFC107,stroke:#333,stroke-width:2px;
    classDef internal fill:#B8E986,stroke:#333,stroke-width:2px;
    classDef background fill:#F3A4B5,stroke:#333,stroke-width:2px;
    classDef management fill:#D1C4E9,stroke:#333,stroke-width:2px;
    classDef performance fill:#FFCDD2,stroke:#333,stroke-width:2px;
    classDef testing fill:#C5CAE9,stroke:#333,stroke-width:2px;
```
