# neuroinformatics-unit/datashuttle architecture

> Architecture diagram of the GitHub repository neuroinformatics-unit/datashuttle, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neuroinformatics-unit/datashuttle
- Repository: https://github.com/neuroinformatics-unit/datashuttle

## Overview

##### Main Components
- Core Engine: At the heart of the project you have the datashuttle package which contains:
- The primary business logic (datashuttle_class.py and datashuttle_functions.py).
- Configuration management modules (all files in datashuttle/configs).
- Utilities and helper functions (in datashuttle/utils covering data_transfer, ssh, rclone, logging, validation, etc.).
- Text User Interface (TUI) Layer: This is a separate module (under datashuttle/tui) that provides:
- The entry point for the user interface (tui_launcher.py, app.py).
- UI components such as screens, tabs, custom widgets, CSS styling, and utility decorators/validators for interface logic.
- External Services / Integrations: Elements that facilitate operations such as SSH file transfer and rclone (found within datashuttle/utils/ssh.py and datashuttle/utils/rclone.py).
- Testing & Continuous Integration: The tests directory (with both unit and integration tests) ensures that business logic, configurations, and TUI functions are validated. GitHub workflows under .github/workflows define the CI/CD processes.
- Documentation and DevOps: The docs folder contains build scripts and documentation source files while configuration files (like pyproject.toml, tox.ini, .pre-commit-config.yaml) guide development standards and deployment.

##### Relationships and Interactions
- The TUI Layer serves as the primary user interaction surface. Its screens and widgets (found under datashuttle/tui/screens, tabs, and custom_widgets.py) interact with the Core Engine to perform tasks such as creating folders, transferring data, or validating configurations.
- The Core Engine uses the configuration management modules to load canonical configs, folder structures, and tags. It then relies on the utilities for actual processing tasks like data transfer or SSH interactions.
- The utility modules are responsible for lower-level tasks, error handling and logging, which are called upon by both the Core Engine and the TUI components.
- Testing components simulate both UI and backend flows to ensure that from user input (via the TUI) to folder creation and data transfer, every component functions as expected.
- The GitHub workflows (and associated CI configuration files) tie the development cycles together by automating code testing, documentation build, and deployment processes.

##### Architectural Patterns and Design Principles
- Separation of Concerns: The project separates the presentation (TUI) from the core business logic and configuration/utility layers.
- Modular Design: Notice how distinct functionalities (configuration, UI, core functions, utilities) are broken into separate modules/directories, facilitating easier testing and future extension.
- Reusability and Extensibility: By isolating configuration parsing, data transfer functions, and TUI components, the design allows for each part to be modified or extended independently.
- Clear Testing Infrastructure: The tests directory is split into unit, integration, and TUI-specific tests, which is a best practice ensuring each layer is verified.

##### Technology, Frameworks, and External Dependencies
- Python is the core language, with project management handled via pyproject.toml, tox.ini, etc.
- The TUI components likely use a text-based interface framework – the structure of the tui directory (screens, tabs, custom widgets) indicates a custom-built or extended framework for terminal-based UIs.
- External libraries for SSH and file transfers (e.g., through rclone) are integrated into the utils.
- The CI/CD pipelines in the .github folder ensure standardized testing and documentation deployment.

## Diagram (Mermaid)

```mermaid
graph TD
    User["User"]:::user

    subgraph "TUI Interface"
        TUILauncher["TUI Launcher (tui_launcher.py)"]:::tui
        TUIApp["TUI App (app.py)"]:::tui
        TUIScreen["UI Components (screens, tabs, widgets)"]:::tui
        TUILauncher -->|"leads to"| TUIApp
        TUIApp -->|"renders"| TUIScreen
    end

    subgraph "Core Engine"
        CoreClass["Business Logic: datashuttle_class.py"]:::core
        CoreFunctions["Business Logic: datashuttle_functions.py"]:::core
        CoreConfigs["Configuration Management (configs/)"]:::core
        CoreClass -->|"passes control to"| CoreFunctions
        CoreFunctions -->|"loads configs from"| CoreConfigs
    end

    subgraph "Utilities"
        Utils["Utilities (utils/)"]:::utils
    end

    subgraph "External Services"
        SSH["SSH Service (ssh.py)"]:::external
        Rclone["Rclone Service (rclone.py)"]:::external
    end

    subgraph "Testing & CI/CD"
        Tests["Tests (tests/)"]:::cicd
        Workflows["Workflows (.github/workflows/)"]:::cicd
        Tests -->|"runs via"| Workflows
    end

    subgraph "Documentation & DevOps"
        Docs["Documentation (docs/)"]:::docs
        DevConfigs["DevOps Configs (pyproject.toml, tox.ini, .pre-commit-config.yaml)"]:::docs
        Docs -->|"supports"| DevConfigs
    end

    %% Main Flow Connections
    User -->|"initiates"| TUILauncher
    TUIScreen -->|"sends commands to"| CoreClass
    CoreConfigs -->|"triggers processes in"| Utils
    Utils -->|"integrates with"| SSH
    Utils -->|"integrates with"| Rclone
    Workflows -->|"validates changes in"| CoreClass
    Workflows -->|"tests interface of"| TUILauncher
    DevConfigs -->|"configures"| Workflows

    %% Click Events
    click CoreClass "https://github.com/neuroinformatics-unit/datashuttle/blob/main/datashuttle/datashuttle_class.py"
    click CoreFunctions "https://github.com/neuroinformatics-unit/datashuttle/blob/main/datashuttle/datashuttle_functions.py"
    click CoreConfigs "https://github.com/neuroinformatics-unit/datashuttle/tree/main/datashuttle/configs/"
    click TUILauncher "https://github.com/neuroinformatics-unit/datashuttle/blob/main/datashuttle/tui/tui_launcher.py"
    click TUIApp "https://github.com/neuroinformatics-unit/datashuttle/blob/main/datashuttle/tui/app.py"
    click Utils "https://github.com/neuroinformatics-unit/datashuttle/tree/main/datashuttle/utils/"
    click SSH "https://github.com/neuroinformatics-unit/datashuttle/blob/main/datashuttle/utils/ssh.py"
    click Rclone "https://github.com/neuroinformatics-unit/datashuttle/blob/main/datashuttle/utils/rclone.py"
    click Tests "https://github.com/neuroinformatics-unit/datashuttle/tree/main/tests/"
    click Workflows "https://github.com/neuroinformatics-unit/datashuttle/tree/main/.github/workflows/"
    click Docs "https://github.com/neuroinformatics-unit/datashuttle/tree/main/docs/"
    click DevConfigs "https://github.com/neuroinformatics-unit/datashuttle/blob/main/pyproject.toml"

    %% Style Definitions
    classDef tui fill:#a8e6cf,stroke:#333,stroke-width:2px;
    classDef core fill:#81d4fa,stroke:#333,stroke-width:2px;
    classDef utils fill:#ffd54f,stroke:#333,stroke-width:2px;
    classDef external fill:#ce93d8,stroke:#333,stroke-width:2px;
    classDef cicd fill:#ef9a9a,stroke:#333,stroke-width:2px;
    classDef docs fill:#c5e1a5,stroke:#333,stroke-width:2px;
    classDef user fill:#fff59d,stroke:#333,stroke-width:2px;
```
