# openfreeenergy/alchemiscale architecture

> Architecture diagram of the GitHub repository openfreeenergy/alchemiscale, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openfreeenergy/alchemiscale
- Repository: https://github.com/openfreeenergy/alchemiscale

## Overview

##### Project Type and Purpose
- Type: A modular, full-stack scientific execution system (library + CLI + services)
- Purpose: Orchestrate and scale free-energy calculations across HPC, cloud, and volunteer platforms

##### File Structure (Top-Level Components)
- alchemiscale/cli.py + cli_utils.py
– Command-line interface entry point
- alchemiscale/base
– Core API definitions and client abstractions
- alchemiscale/compute
– Compute API, client, manager, service, settings → Schedules and runs jobs
- alchemiscale/interface
– External‐facing API and client (e.g., REST or gRPC)
- alchemiscale/storage
- alchemiscale/strategist
– Strategy automation service reads state and generates new experiments
- alchemiscale/security
– Authentication, authorization models
- alchemiscale/models.py
– Shared data models & schemas
- Settings & migrations
– Configuration plus versioned DB migrations
- Docker definitions
– alchemiscale-compute & alchemiscale-server containers

##### README & Docs
- Confirms high-throughput free energy workflows
- References HPC, cloud, Folding@Home integration
- Indicates RESTful API layers and CLI usage

##### Main Components & Interactions
a. Components
1. User & CLI
2. Interface Service (REST/gRPC)
3. Compute Service & Manager
4. Strategist Service
5. Storage Layer
– Object Store (sim files, results)
– State Store (job metadata)
6. Security Module (AuthN/Z)
7. External Execution Backends (HPC schedulers, cloud VMs, FAH)
8. Database (for state, migrations)
b. Relationships / Data Flow
- User invokes CLI → CLI uses Interface Client → Interface Service
- Interface Service authenticates user → stores job request in State Store
- Compute Manager polls State Store → dispatches tasks to external backends
- External backend runs simulations → uploads results to Object Store
- Compute Service updates State Store on completion
- Strategist Service subscribes to State Store changes → generates next tasks
- All components respect security policies via security.auth
c. Architectural Patterns
- Microservices style: distinct Interface, Compute, Strategist services
- Client/Server: base.client and clients in each module
- Event-driven or polling (Compute Manager / Strategist poll state store)
- Separation of concerns: storage vs compute vs orchestration
d. Key Technologies & Libraries
- Python 3, pyproject.toml with dependencies
- Flask/FastAPI or gRPC (in interface.api)
- Async job execution patterns
- Object store adapters (alchemiscale.storage.objectstore)
- State store adapters (e.g., Redis, RDBMS via statestore)
- Docker for containerization

##### Detailed Elements to Capture
- Layered view: Presentation (CLI), API, Orchestration, Storage, External Execution
- Shared modules (models.py, security) perched centrally to indicate reuse
- Configuration & migrations as a side module connected to State Store
- Docker-compose deployment grouping of compute & server

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        CLI["CLI"]:::client
        CLI_UTIL["CLI Utils"]:::client
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        Interface["Interface Service (REST/gRPC)"]:::service
        BaseAPI["Base API"]:::client
        BaseClient["Base Client"]:::client
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        ComputeAPI["Compute Service/API"]:::service
        ComputeClient["Compute Client"]:::client
        ComputeManager["Compute Manager"]:::service
        ComputeService["Compute Worker Service"]:::service
        Strategist["Strategist Service"]:::service
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        ObjectStore["Object Store Adapter"]:::storage
        StateStore["State Store Adapter"]:::storage
        SubgraphPersist["Subgraph Persistence"]:::storage
        StorageModels["Storage Models & Utils"]:::storage
    end

    %% Supporting Modules
    subgraph "Supporting Modules"
        direction TB
        SecurityAuth["Security Auth"]:::core
        SecurityModels["Security Models"]:::core
        SharedModels["Shared Data Models"]:::core
        Config["Configuration & Migrations"]:::core
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        HPC["HPC Scheduler"]:::external
        CloudVM["Cloud VMs"]:::external
        FAH["Folding@Home"]:::external
    end

    %% Docker Deployment
    subgraph "Docker Deployment"
        direction TB
        DockerCompute["Compute Container"]:::infra
        DockerServer["Server Container"]:::infra
    end

    %% Connections
    CLI -->|"invokes"| Interface
    CLI_UTIL -->|"supports"| CLI
    Interface -->|"uses Base API"| BaseAPI
    Interface -->|"uses Base Client"| BaseClient
    Interface -->|"authenticates via"| SecurityAuth
    Interface -->|"writes job to"| StateStore
    ComputeManager -->|"polls"| StateStore
    ComputeManager -->|"dispatches to"| ComputeService
    ComputeService -->|"runs task on"| HPC
    ComputeService -->|"runs task on"| CloudVM
    ComputeService -->|"runs task on"| FAH
    ComputeService -->|"uploads results to"| ObjectStore
    ComputeService -->|"updates"| StateStore
    Strategist -->|"subscribes to"| StateStore
    Strategist -->|"generates tasks"| ComputeManager
    ObjectStore -->|"stores data for"| StorageModels
    StateStore -->|"metadata for"| StorageModels
    SubgraphPersist -->|"used by"| StorageModels
    BaseAPI -->|"shared with"| ComputeAPI
    BaseClient -->|"shared with"| ComputeClient
    Interface -->|"containerized in"| DockerServer
    ComputeService -->|"containerized in"| DockerCompute
    Config -->|"migrations for"| StateStore
    SecurityModels -->|"used by"| SecurityAuth
    SharedModels -->|"used by all layers"| Interface

    %% Click Events
    click CLI "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/cli.py"
    click CLI_UTIL "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/cli_utils.py"
    click BaseAPI "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/base/api.py"
    click BaseClient "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/base/client.py"
    click ComputeAPI "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/compute/api.py"
    click ComputeClient "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/compute/client.py"
    click ComputeManager "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/compute/manager.py"
    click ComputeService "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/compute/service.py"
    click ComputeSettings "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/compute/settings.py"
    click Interface "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/interface/api.py"
    click InterfaceClient "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/interface/client.py"
    click ObjectStore "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/storage/objectstore.py"
    click StateStore "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/storage/statestore.py"
    click SubgraphPersist "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/storage/subgraph.py"
    click StorageModels "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/storage/models.py"
    click StorageCypher "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/storage/cypher.py"
    click Strategist "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/strategist/service.py"
    click StratSettings "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/strategist/settings.py"
    click SecurityAuth "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/security/auth.py"
    click SecurityModels "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/security/models.py"
    click SharedModels "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/models.py"
    click Config "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/settings.py"
    click Migration "https://github.com/openfreeenergy/alchemiscale/blob/main/alchemiscale/migrations/v03_to_v04.py"
    click DockerCompute "https://github.com/openfreeenergy/alchemiscale/tree/main/docker/alchemiscale-compute/Dockerfile"
    click DockerServer "https://github.com/openfreeenergy/alchemiscale/tree/main/docker/alchemiscale-server/Dockerfile"
    click DockerCompose "https://github.com/openfreeenergy/alchemiscale/blob/main/docker/alchemiscale-server/docker-compose.yml"

    %% Styles
    classDef service fill:#D0E8FF,stroke:#4DA6FF,color:#004080;
    classDef storage fill:#E0F8E0,stroke:#70C070,color:#006000;
    classDef client fill:#FFF4C2,stroke:#FFD440,color:#806000;
    classDef external fill:#F0F0F0,stroke:#B0B0B0,color:#505050;
    classDef core fill:#F5F5F5,stroke:#CCCCCC,color:#333333;
    classDef infra fill:#FFE0E0,stroke:#FF9090,color:#800000;
    class CLI_UTIL,BaseClient,ComputeClient,InterfaceClient client
    class ComputeSettings,StratSettings core
    class Migration core
```
