# openchemistry/distiller architecture

> Architecture diagram of the GitHub repository openchemistry/distiller, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openchemistry/distiller
- Repository: https://github.com/openchemistry/distiller

## Overview

##### Main System Components

a) Client Layer:
- React Frontend (under /frontend/distiller)
- Web Interface for monitoring and control
- Real-time data visualization components

b) API Layer:
- FastAPI Backend (under /backend/app)
- REST endpoints for various services
- Authentication and authorization

c) Processing Layer:
- Faust Workers (under /backend/faust)
- Multiple specialized workers (job, notebook, scan, etc.)
- Template processing

d) Monitoring Layer:
- Watch CLI (under /cli/watch)
- Custodian service (under /cli/custodian)
- Real-time file monitoring

e) External Services:
- NERSC HPC
- Jupyter Hub
- Kafka Message Broker
- Database (SQL, based on alembic migrations)

##### Component Relationships

**Frontend ←→ FastAPI Backend ←→ Kafka ←→ Faust Workers ←→ NERSC HPC**
- ↑                ↑

- ↓                ↓
- Database    File Monitoring System

##### Specific Implementation Details

a) Frontend Architecture:
- React components (/frontend/distiller/src/components)
- Feature modules (/frontend/distiller/src/features)
- State management (Redux, based on store.ts)

b) Backend Services:
- API endpoints (/backend/app/api/api_v1/endpoints)
- Authentication flow
- Database models (/backend/app/models)
- Schema definitions (/backend/app/schemas)

c) Worker Architecture:
- Job processing pipeline
- Notebook generation
- Scan processing
- File monitoring system

##### Integration Points

Integration with:
- NERSC Superfacility API
- Jupyter Hub
- Stempy processing library
- Microscopy equipment

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client["Client Layer"]
        Frontend["React Frontend"]:::frontend
        Components["UI Components"]:::frontend
        Features["Feature Modules"]:::frontend
        State["State Management"]:::frontend
        Frontend --> Components
        Frontend --> Features
        Frontend --> State
    end

    subgraph API["API Layer"]
        FastAPI["FastAPI Backend"]:::api
        Auth["Authentication"]:::api
        Endpoints["API Endpoints"]:::api
        FastAPI --> Auth
        FastAPI --> Endpoints
    end

    subgraph Processing["Processing Layer"]
        Workers["Faust Workers"]:::worker
        JobWorker["Job Worker"]:::worker
        NotebookWorker["Notebook Worker"]:::worker
        ScanWorker["Scan Worker"]:::worker
        Templates["Templates"]:::worker
        Workers --> JobWorker
        Workers --> NotebookWorker
        Workers --> ScanWorker
        Workers --> Templates
    end

    subgraph Monitoring["Monitoring Layer"]
        Watch["Watch CLI"]:::monitor
        Custodian["Custodian Service"]:::monitor
        FileMonitor["File Monitoring"]:::monitor
        Watch --> FileMonitor
        Custodian --> FileMonitor
    end

    subgraph External["External Services"]
        NERSC["NERSC HPC"]:::external
        JupyterHub["Jupyter Hub"]:::external
        Kafka["Kafka Message Broker"]:::messaging
        DB[(Database)]:::storage
    end

    Frontend <--> FastAPI
    FastAPI <--> DB
    FastAPI <--> Kafka
    Kafka <--> Workers
    Workers <--> NERSC
    FileMonitor --> Kafka
    FastAPI <--> JupyterHub

    %% Click events for component mapping
    click Frontend "https://github.com/openchemistry/distiller/tree/main/frontend/distiller"
    click Components "https://github.com/openchemistry/distiller/tree/main/frontend/distiller/src/components"
    click Features "https://github.com/openchemistry/distiller/tree/main/frontend/distiller/src/features"
    click State "https://github.com/openchemistry/distiller/blob/main/frontend/distiller/src/app/store.ts"
    click FastAPI "https://github.com/openchemistry/distiller/tree/main/backend/app"
    click Endpoints "https://github.com/openchemistry/distiller/tree/main/backend/app/app/api/api_v1/endpoints"
    click Auth "https://github.com/openchemistry/distiller/blob/main/backend/app/app/api/api_v1/endpoints/auth.py"
    click Workers "https://github.com/openchemistry/distiller/tree/main/backend/faust"
    click JobWorker "https://github.com/openchemistry/distiller/blob/main/backend/faust/job_worker.py"
    click NotebookWorker "https://github.com/openchemistry/distiller/blob/main/backend/faust/notebook_worker.py"
    click ScanWorker "https://github.com/openchemistry/distiller/blob/main/backend/faust/scan_worker.py"
    click Templates "https://github.com/openchemistry/distiller/tree/main/backend/faust/templates"
    click Watch "https://github.com/openchemistry/distiller/blob/main/cli/watch/distiller/watch.py"
    click Custodian "https://github.com/openchemistry/distiller/blob/main/cli/custodian/distiller/custodian.py"
    click DB "https://github.com/openchemistry/distiller/tree/main/backend/app/alembic"

    %% Styling
    classDef frontend fill:#3498db,stroke:#2980b9,color:white
    classDef api fill:#2ecc71,stroke:#27ae60,color:white
    classDef worker fill:#f1c40f,stroke:#f39c12,color:black
    classDef monitor fill:#e74c3c,stroke:#c0392b,color:white
    classDef external fill:#9b59b6,stroke:#8e44ad,color:white
    classDef messaging fill:#e67e22,stroke:#d35400,color:white
    classDef storage fill:#95a5a6,stroke:#7f8c8d,color:white

    %% Legend
    subgraph Legend
        Frontend_L["Frontend"]:::frontend
        API_L["API Services"]:::api
        Worker_L["Workers"]:::worker
        Monitor_L["Monitoring"]:::monitor
        External_L["External Services"]:::external
        Message_L["Message Broker"]:::messaging
        Storage_L["Storage"]:::storage
    end
```
