# alcionai/corso architecture

> Architecture diagram of the GitHub repository alcionai/corso, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alcionai/corso
- Repository: https://github.com/alcionai/corso

## Overview

##### Project Type and Purpose
- Corso is an open‑source, command‑line data‑protection tool for Microsoft 365 (Exchange, OneDrive, SharePoint, Teams).
- It provides an intuitive CLI for administrators to back up, export, restore, and purge M365 data into storage of their choice.

##### High‑Level File Structure Analysis
- Top‑level folders:
– src/cli – CLI commands and flags
– src/internal – core engine: M365 connectors, Kopia integration, orchestrators, utilities
– src/pkg – shared libraries: account management, config, storage adapters, API clients
– cmd/… – specialized binaries and test drivers (sanity, longevity, graph PowerShell helper)
– build/, docker/, .github/ – build scripts and CI/CD workflows
– website/ – Docusaurus‑based documentation site
- Key patterns:
– Adapter and factory patterns for M365 services and storage backends
– “operations” orchestration layer managing backup, export, restore flows
– Wrapper layer around Kopia (the backup engine) for snapshot management
– Streamstore abstraction for writing incremental data streams
– Internal “converters” to normalize M365 payloads (EML, ICS, VCF) to a unified model

##### Core Components
A. CLI Frontend
– Entry point (corso.go) → dispatches to subcommands: backup, export, restore, purge, debug
B. Controller / Orchestrator Layer
– internal/controller.go
– internal/operations (backup.go, export.go, restore.go, retention_config.go)
– Factories and dependency injection (cmd/factory)
C. M365 Service Adapters
– pkg/services/m365/api/graph and pkg/services/m365/service/* for Exchange, OneDrive, SharePoint, Teams
– internal/m365/collection and internal/m365/service layers
D. Data Conversion
– internal/converters (EML, ICS, VCF)
– internal/model for unified schema
E. Backup Engine Integration
– internal/kopia wrapper, inject, merge, retention
– pkg/store/model_store and pkg/store/wrapper for storing Kopia snapshots
F. Storage Backends
– pkg/storage/filesystem and pkg/storage/s3 adapters
– cli/repo for filesystem or S3 repositories
G. StreamStore & Metrics
– internal/streamstore for streaming data to Kopia
– internal/stats and pkg/metrics for telemetry
H. Utilities & Common
– internal/common (config, limiters, retry, crash handling, logging)

##### Component Interactions & Data Flows
1) User invokes `corso backup` → CLI layer parses flags and options.
2) CLI creates an operations.Backup instance via factory.
3) Backup operation calls M365 connectors → paginates via Graph API → collects items.
4) Collected items pass through converters → normalized model.
5) Model fed into streamstore → streamed into Kopia wrapper.
6) Kopia wrapper commits snapshot into a repository (S3 or FS) via storage adapter.
7) Operations record metadata and metrics; exit.

##### Architectural Patterns & Principles
- Modular layering: CLI → orchestrator → domain services → engine
- Adapter pattern for external dependencies (Graph API, Kopia, S3/FS)
- Factory/DI to allow testing with mocks (internal/*/mock packages)
- Separation of concerns: converters, models, storage kept distinct
- Extensibility points: new M365 services or storage backends plug into existing abstractions

##### Relevant Technologies & Frameworks
- Go modules with well‑structured pkg/ and internal/ layout
- Kopia backup engine via Go wrapper
- GitHub Actions for CI, release, website linting
- Docusaurus for documentation site
- Docker for self‑contained builds and tooling (build/, docker/)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Layer
    subgraph "CLI Frontend"
        direction TB
        CLI_Entry["corso.go"]:::cli
        CLI_CLI["cli.go"]:::cli
        CLI_Backup["backup.go"]:::cli
        CLI_Export["export.go"]:::cli
        CLI_Restore["restore.go"]:::cli
        CLI_Debug["debug.go"]:::cli
    end

    %% Controller / Orchestrator
    subgraph "Controller / Orchestrator" 
        direction TB
        Factory["factory.go"]:::core
        Controller["controller.go"]:::core
        Op_Backup["backup.go"]:::core
        Op_Export["export.go"]:::core
        Op_Restore["restore.go"]:::core
        Op_Retention["retention_config.go"]:::core
    end

    %% M365 Service Adapters
    subgraph "M365 Service Adapters"
        direction TB
        GraphAPI["graph (API)"]:::external
        API_Access["api (access & client)"]:::external
        Service_Exchange["service/exchange"]:::external
        Service_OneDrive["service/onedrive"]:::external
        Service_SharePoint["service/sharepoint"]:::external
        Service_Groups["service/groups"]:::external
        Service_Teams["service/teamschats"]:::external
        Internal_Svc["internal/m365/service"]:::core
        Internal_Col["internal/m365/collection"]:::core
    end

    %% Converters & Model
    subgraph "Converters & Model"
        direction TB
        Conv_EML["eml.go"]:::core
        Conv_ICS["ics.go"]:::core
        Conv_VCF["vcf.go"]:::core
        Model["model.go"]:::core
    end

    %% StreamStore & Metrics
    subgraph "StreamStore & Metrics"
        direction TB
        StreamStore["streamstore.go"]:::core
        Collectables["collectables.go"]:::core
        Stats["stats.go"]:::core
        Metrics["metrics.go"]:::core
    end

    %% Kopia Engine Integration
    subgraph "Kopia Engine Integration"
        direction TB
        Kopia_Wrapper["wrapper.go"]:::core
        Kopia_Bases["backup_bases.go"]:::core
        Kopia_Merge["merge_collection.go"]:::core
        Kopia_Retention["opts.go"]:::core
        ModelStore["model_store.go"]:::core
        Store_Wrapper["wrapper (store)"]:::core
    end

    %% Storage Backends
    subgraph "Storage Backends"
        direction TB
        FS_Adapter["filesystem.go"]:::external
        S3_Adapter["s3.go"]:::external
        Repo_FS["filesystem.go (repo)"]:::external
        Repo_S3["s3.go (repo)"]:::external
    end

    %% Utilities & Common
    subgraph "Utilities & Common"
        direction TB
        Configs["configs.go"]:::util
        Retry["retry_handler.go"]:::util
        Limiter["limiter.go"]:::util
        Crash["crash.go"]:::util
        Logger["logger.go"]:::util
    end

    %% External Services
    GraphAPI_ext["Microsoft Graph API"]:::external
    S3_ext["S3 Storage"]:::external
    FS_ext["Filesystem"]:::external

    %% Connections
    CLI_Entry -->|invokes| CLI_CLI
    CLI_CLI -->|dispatches| CLI_Backup
    CLI_CLI -->|dispatches| CLI_Export
    CLI_CLI -->|dispatches| CLI_Restore
    CLI_CLI -->|dispatches| CLI_Debug

    CLI_Backup -->|uses| Factory
    CLI_Export -->|uses| Factory
    CLI_Restore -->|uses| Factory
    CLI_Debug -->|uses| Factory

    Factory --> Controller
    Controller -->|creates| Op_Backup
    Controller -->|creates| Op_Export
    Controller -->|creates| Op_Restore
    Controller -->|reads| Op_Retention

    Op_Backup -->|calls HTTP/REST| GraphAPI
    Op_Backup -->|via DI| Internal_Svc
    Internal_Svc -->|implements| Service_Exchange
    Internal_Svc -->|implements| Service_OneDrive
    Internal_Svc -->|implements| Service_SharePoint
    Internal_Svc -->|implements| Service_Groups
    Internal_Svc -->|implements| Service_Teams
    Internal_Col -->|collects| Internal_Svc

    Service_Exchange -->|REST| GraphAPI
    Service_OneDrive -->|REST| GraphAPI
    Service_SharePoint -->|REST| GraphAPI
    Service_Groups -->|REST| GraphAPI
    Service_Teams -->|REST| GraphAPI

    Op_Backup -->|outputs| Conv_EML
    Op_Backup -->|outputs| Conv_ICS
    Op_Backup -->|outputs| Conv_VCF
    Conv_EML --> Model
    Conv_ICS --> Model
    Conv_VCF --> Model

    Model -->|feeds| StreamStore
    StreamStore --> Collectables
    Collectables -->|streams| Kopia_Wrapper

    Kopia_Wrapper --> Kopia_Bases
    Kopia_Bases --> Kopia_Merge
    Kopia_Merge --> Kopia_Retention
    Kopia_Retention -->|commits| ModelStore
    ModelStore --> Store_Wrapper

    Store_Wrapper -->|writes| FS_Adapter
    Store_Wrapper -->|writes| S3_Adapter

    FS_Adapter -->|I/O| FS_ext
    S3_Adapter -->|API| S3_ext

    Op_Backup -->|records| Stats
    Stats --> Metrics

    %% Utilities usage
    Controller --> Configs
    Controller --> Retry
    Controller --> Limiter
    Controller --> Crash
    Controller --> Logger

    %% Click Events
    click CLI_Entry "https://github.com/alcionai/corso/blob/main/src/corso.go"
    click CLI_CLI "https://github.com/alcionai/corso/blob/main/src/cli/cli.go"
    click CLI_Backup "https://github.com/alcionai/corso/blob/main/src/cli/backup/backup.go"
    click CLI_Export "https://github.com/alcionai/corso/blob/main/src/cli/export/export.go"
    click CLI_Restore "https://github.com/alcionai/corso/blob/main/src/cli/restore/restore.go"
    click CLI_Debug "https://github.com/alcionai/corso/blob/main/src/cli/debug/debug.go"
    click Factory "https://github.com/alcionai/corso/blob/main/src/cmd/factory/factory.go"
    click Controller "https://github.com/alcionai/corso/blob/main/src/internal/controller.go"
    click Op_Backup "https://github.com/alcionai/corso/blob/main/src/internal/operations/backup.go"
    click Op_Export "https://github.com/alcionai/corso/blob/main/src/internal/operations/export.go"
    click Op_Restore "https://github.com/alcionai/corso/blob/main/src/internal/operations/restore.go"
    click Op_Retention "https://github.com/alcionai/corso/blob/main/src/internal/operations/retention_config.go"
    click GraphAPI "https://github.com/alcionai/corso/tree/main/src/pkg/services/m365/api/graph"
    click API_Access "https://github.com/alcionai/corso/tree/main/src/pkg/services/m365/api"
    click Service_Exchange "https://github.com/alcionai/corso/tree/main/src/pkg/services/m365/service/exchange"
    click Service_OneDrive "https://github.com/alcionai/corso/tree/main/src/pkg/services/m365/service/onedrive"
    click Service_SharePoint "https://github.com/alcionai/corso/tree/main/src/pkg/services/m365/service/sharepoint"
    click Service_Groups "https://github.com/alcionai/corso/tree/main/src/pkg/services/m365/service/groups"
    click Service_Teams "https://github.com/alcionai/corso/tree/main/src/pkg/services/m365/service/teamschats"
    click Internal_Svc "https://github.com/alcionai/corso/tree/main/src/internal/m365/service"
    click Internal_Col "https://github.com/alcionai/corso/tree/main/src/internal/m365/collection"
    click Conv_EML "https://github.com/alcionai/corso/blob/main/src/internal/converters/eml/eml.go"
    click Conv_ICS "https://github.com/alcionai/corso/blob/main/src/internal/converters/ics/ics.go"
    click Conv_VCF "https://github.com/alcionai/corso/blob/main/src/internal/converters/vcf/vcf.go"
    click Model "https://github.com/alcionai/corso/blob/main/src/internal/model/model.go"
    click Kopia_Wrapper "https://github.com/alcionai/corso/blob/main/src/internal/kopia/wrapper.go"
    click Kopia_Bases "https://github.com/alcionai/corso/blob/main/src/internal/kopia/backup_bases.go"
    click Kopia_Merge "https://github.com/alcionai/corso/blob/main/src/internal/kopia/merge_collection.go"
    click Kopia_Retention "https://github.com/alcionai/corso/blob/main/src/internal/kopia/retention/opts.go"
    click ModelStore "https://github.com/alcionai/corso/blob/main/src/pkg/store/model_store.go"
    click Store_Wrapper "https://github.com/alcionai/corso/tree/main/src/pkg/store/wrapper"
    click FS_Adapter "https://github.com/alcionai/corso/blob/main/src/pkg/storage/filesystem.go"
    click S3_Adapter "https://github.com/alcionai/corso/blob/main/src/pkg/storage/s3.go"
    click Repo_FS "https://github.com/alcionai/corso/blob/main/src/cli/repo/filesystem.go"
    click Repo_S3 "https://github.com/alcionai/corso/blob/main/src/cli/repo/s3.go"
    click StreamStore "https://github.com/alcionai/corso/blob/main/src/internal/streamstore/streamstore.go"
    click Collectables "https://github.com/alcionai/corso/blob/main/src/internal/streamstore/collectables.go"
    click Stats "https://github.com/alcionai/corso/blob/main/src/internal/stats/stats.go"
    click Metrics "https://github.com/alcionai/corso/blob/main/src/pkg/metrics/metrics.go"
    click Configs "https://github.com/alcionai/corso/blob/main/src/internal/common/configs.go"
    click Retry "https://github.com/alcionai/corso/blob/main/src/internal/common/retry_handler.go"
    click Limiter "https://github.com/alcionai/corso/blob/main/src/internal/common/limiters/limiter.go"
    click Crash "https://github.com/alcionai/corso/blob/main/src/internal/common/crash/crash.go"
    click Logger "https://github.com/alcionai/corso/blob/main/src/pkg/logger/logger.go"

    %% Styles
    classDef cli fill:#cce5ff,stroke:#66b0ff;
    classDef core fill:#d4edda,stroke:#7cc87c;
    classDef external fill:#fff3cd,stroke:#ffdf7e;
    classDef util fill:#e2e3e5,stroke:#b9bbbe;
```
