# cybercentrecanada/assemblyline-base architecture

> Architecture diagram of the GitHub repository cybercentrecanada/assemblyline-base, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cybercentrecanada/assemblyline-base
- Repository: https://github.com/cybercentrecanada/assemblyline-base

## Overview

##### Main Components

a) Data Storage Layer:
- CacheStore (/assemblyline/cachestore)
- DataStore (/assemblyline/datastore)
- FileStore (/assemblyline/filestore)

b) Object Document Mapper (ODM) Layer:
- Models (/assemblyline/odm/models)
- Messages (/assemblyline/odm/messages)

c) Remote Data Types Layer:
- Queues
- Caches
- Counters
- Sets
- Locks

d) Common Utilities Layer:
- Classification
- Security
- Identification
- Archiving
- Metrics

##### Component Relationships

**┌─────────────────────────────────────────────────────────┐**
- Application Layer                       │
- ┬─────────────────┬──────────────────┘

**┌───────────────────▼─────┐ ┌────────▼────────┐**
- Storage Layer       │ │   ODM Layer     │
- ┌─────┐ ┌─────┐ ┌────┐│ │ ┌────────────┐  │
- Cache│ │Data │ │File││ │ │   Models   │  │
- Store│ │Store│ │Store││ │ └────────────┘  │
- ┘ └─────┘ └────┘│ │ ┌────────────┐  │
- ┘ │ │  Messages  │  │
- ┘  │
- ┘
**┌──────────────────────────────────────────┐**
- Remote Data Types              │
- ┌─────┐ ┌─────┐ ┌─────┐ ┌────┐ ┌────┐  │
- Queue│ │Cache│ │Count│ │Sets│ │Lock│  │
- ┘ └─────┘ └─────┘ └────┘ └────┘  │
- ┘
**┌──────────────────────────────────────────┐**
- Common Utilities               │
- Security, Classification, Identification │
- ┘

##### Important Details

a) Storage Layer:
- Supported backends (Azure, S3, FTP, HTTP, Local)
- Classification boundaries
- Caching mechanisms

b) ODM Layer:
- Models (File, Result, Submission)
- Message types for system communication
- Validation flows

c) Remote Data Types:
- Queue priority systems
- Distributed lock mechanisms
- Counter and cache relationships

##### Implementation Notes

- Version information (4.x)
- Notes about Python 3.11 requirement
- Containerization boundaries

Remember to maintain a balance between detail and clarity, focusing on the core components while providing enough information to understand the system's architecture.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Application Layer"
        App[Assemblyline 4.x]:::app
    end

    subgraph "Storage Layer"
        direction TB
        CacheStore["CacheStore"]:::storage
        DataStore["DataStore"]:::storage
        FileStore["FileStore"]:::storage

        subgraph "FileStore Transport"
            direction TB
            Azure["Azure Transport"]:::transport
            S3["S3 Transport"]:::transport
            FTP["FTP Transport"]:::transport
            HTTP["HTTP Transport"]:::transport
            Local["Local Transport"]:::transport
        end

        FileStore --> Azure
        FileStore --> S3
        FileStore --> FTP
        FileStore --> HTTP
        FileStore --> Local
    end

    subgraph "ODM Layer"
        direction TB
        Models["Models"]:::odm
        Messages["Messages"]:::odm
    end

    subgraph "Remote Data Types"
        direction TB
        Queues["Queues"]:::remote
        Caches["Caches"]:::remote
        Counters["Counters"]:::remote
        Sets["Sets"]:::remote
        Locks["Locks"]:::remote
    end

    subgraph "Common Utilities"
        direction TB
        Classification["Classification"]:::util
        Security["Security"]:::util
        Identification["Identification"]:::util
        Archiving["Archiving"]:::util
        Metrics["Metrics"]:::util
    end

    App --> CacheStore
    App --> DataStore
    App --> FileStore
    App --> Models
    App --> Messages

    Models --> DataStore
    Messages --> DataStore
    CacheStore --> Caches
    DataStore --> Sets
    DataStore --> Locks
    FileStore --> Counters

    %% Click events for component mapping
    click CacheStore "https://github.com/CybercentreCanada/assemblyline-base/tree/master/assemblyline/cachestore/"
    click DataStore "https://github.com/CybercentreCanada/assemblyline-base/tree/master/assemblyline/datastore/"
    click FileStore "https://github.com/CybercentreCanada/assemblyline-base/tree/master/assemblyline/filestore/"
    click Models "https://github.com/CybercentreCanada/assemblyline-base/tree/master/assemblyline/odm/models/"
    click Messages "https://github.com/CybercentreCanada/assemblyline-base/tree/master/assemblyline/odm/messages/"
    click Queues "https://github.com/CybercentreCanada/assemblyline-base/tree/master/assemblyline/remote/datatypes/queues/"
    click Caches "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/remote/datatypes/cache.py"
    click Counters "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/remote/datatypes/counters.py"
    click Sets "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/remote/datatypes/set.py"
    click Locks "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/remote/datatypes/lock.py"
    click Classification "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/common/classification.py"
    click Security "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/common/security.py"
    click Identification "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/common/identify.py"
    click Archiving "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/common/archiving.py"
    click Metrics "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/common/metrics.py"
    click Azure "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/filestore/transport/azure.py"
    click S3 "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/filestore/transport/s3.py"
    click FTP "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/filestore/transport/ftp.py"
    click HTTP "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/filestore/transport/http.py"
    click Local "https://github.com/CybercentreCanada/assemblyline-base/blob/master/assemblyline/filestore/transport/local.py"

    %% Styling
    classDef app fill:#f9f,stroke:#333,stroke-width:2px
    classDef storage fill:#adf,stroke:#333,stroke-width:1px
    classDef odm fill:#ada,stroke:#333,stroke-width:1px
    classDef remote fill:#fda,stroke:#333,stroke-width:1px
    classDef util fill:#eee,stroke:#333,stroke-width:1px
    classDef transport fill:#ddf,stroke:#333,stroke-width:1px
```
