# idiap/gridtk architecture

> Architecture diagram of the GitHub repository idiap/gridtk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/idiap/gridtk
- Repository: https://github.com/idiap/gridtk

## Overview

##### Project type and purpose
- GridTK is a Python-based command-line application (CLI tool) that wraps and extends Slurm’s `sbatch` interface.
- Its goal is to simplify job submission, monitoring, dependencies, logging, and resubmission for Slurm clusters.

##### File structure
- Top-level:
- `.github/`, CI/CD workflows, code coverage, deployment.
- `doc/`, Sphinx docs.
- `src/gridtk/`, main application package.
- `tests/`, unit tests.
- `pyproject.toml`, package metadata.
- Inside `src/gridtk/`:
- `__main__.py`, CLI entrypoint.
- `cli.py`, command definitions (`submit`, `list`, `report`, `stop`, `delete`, `resubmit`, etc.).
- `manager.py`, core orchestration and business logic.
- `models.py`, data structures (job metadata, state transitions).
- `tools.py`, low-level helpers (filesystem operations, subprocess wrappers for `sbatch`, environment-variable handling).

##### Key insights from README
- Commands correspond to subcommands in `cli.py`.
- GridTK maintains “local job IDs,” wraps Slurm calls, tracks logs under a `logs/` directory.
- Supports environment-variable overrides for defaults, temporary-script generation, dependencies, arrays, repeat jobs.
- External integration point: Slurm cluster via system calls (`sbatch`, `squeue`, `scontrol`, `sacct`).

##### Main components
A. CLI Frontend
- The user interacts via the `gridtk` command (installed via pipx).
- Implements parsing of subcommands & options.
B. Core Service Layer (Manager)
- Receives parsed commands, applies business rules (e.g., dependency resolution, repeat logic).
C. Data Layer (Models & Metadata Store)
- Defines Job objects, maintains a local store (e.g., JSON or lightweight file) mapping local job IDs → Slurm IDs → state.
D. Tools Layer
- Subprocess wrapper for invoking Slurm commands.
- File/IO helpers for creating temporary scripts, writing log files, reading environment variables.
E. External Services
- Slurm scheduler (executes batch jobs).
F. Local File System
- `logs/` directory for job stdout/stderr.
- Temporary script storage (`/tmp/...`).
- Local metadata file(s) under working directory.

##### Relationships and data flows
- User Input → CLI Frontend
- CLI Frontend → Manager
- Manager ↔ Models (read/write job metadata)
- Manager ↔ Tools
- Tools → Slurm (via subprocess calls)
- Tools → File System (logs, temp scripts)
- Manager → Output back to CLI → Console
- Manager → Slurm → returns Slurm job ID, state updates
- Manager reads Slurm via `squeue`/`sacct` for status reports

##### Architectural patterns & principles
- Layered architecture: clear separation between CLI, service logic, data model, and low-level tools.
- Single responsibility: each module handles one concern (parsing, orchestration, data, IO).
- Dependency injection (via environment variables) for default options.

##### Technologies & frameworks
- Python packaging (`pyproject.toml`), pipx installation
- Standard library for subprocess and file IO
- Sphinx for documentation
- GitHub Actions for CI/CD

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "CLI Frontend"
        A1["__main__.py"]:::cli
        A2["cli.py"]:::cli
    end

    subgraph "Core Service Layer"
        B["Manager (Business Logic)"]:::service
    end

    subgraph "Data Layer"
        C["Models (Job Store)"]:::data
        E["Local Metadata File"]:::storage
    end

    subgraph "Tools Layer"
        D["Tools (IO & Subprocess)"]:::tools
    end

    subgraph "External & File System"
        G["Slurm Scheduler"]:::external
        F["Logs Directory"]:::storage
        H["File System (Temp Scripts)"]:::storage
        Env["Env Variables Override"]:::external
        OUT["Console Output"]:::external
    end

    A1 -->|"entrypoint"| A2
    A2 -->|"parse args"| B
    B -->|"read/write metadata"| C
    C -->|"persist state"| E
    B -->|"invoke tools"| D
    D -->|"submit sbatch / poll status"| G
    G -->|"job ID & status"| D
    D -->|"create/delete scripts"| H
    D -->|"write logs"| F
    D -->|"return output/status"| B
    B -->|"return result"| A2
    A2 -->|"print results"| OUT
    Env -->|"override defaults"| B
    Env -->|"env var read"| D

    click A1 "https://github.com/idiap/gridtk/blob/main/src/gridtk/__main__.py"
    click A2 "https://github.com/idiap/gridtk/blob/main/src/gridtk/cli.py"
    click B "https://github.com/idiap/gridtk/blob/main/src/gridtk/manager.py"
    click C "https://github.com/idiap/gridtk/blob/main/src/gridtk/models.py"
    click D "https://github.com/idiap/gridtk/blob/main/src/gridtk/tools.py"

    classDef cli fill:#D7E3FC,stroke:#4A69BD,color:#000
    classDef service fill:#CFE2F3,stroke:#2E86C1,color:#000
    classDef data fill:#E2EFDA,stroke:#3C763D,color:#000
    classDef tools fill:#FCE5CD,stroke:#E69138,color:#000
    classDef storage fill:#F4CCCC,stroke:#C12D2D,color:#000
    classDef external fill:#EAD1DC,stroke:#A64D79,color:#000
```
