# javfg/pis architecture

> Architecture diagram of the GitHub repository javfg/pis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/javfg/pis
- Repository: https://github.com/javfg/pis

## Overview

##### Main Components

a) Core Pipeline Components:
- Configuration Management System
- Task Registry
- Step Runner
- Storage System
- Manifest Manager
- Validation System

b) Task Execution Flow:
- Pre-tasks Phase
- Main Tasks Phase
- Validation Phase
- Upload Phase

**┌─ Input Layer ─────┐  ┌─ Processing Layer ──┐  ┌─ Storage Layer ──┐**
- CLI            │  │ - Step Runner       │  │ - Google Storage │
- Config Parser  │  │ - Task Registry     │  │ - Local Storage  │
- ENV Variables  │  │ - Task Executor     │  └─────────────────┘
- ┘  │ - Validators        │
- ┘

##### Component Details

a) Core Components (Rectangles):
- Configuration System (src/pis/config/)
- Task Registry (src/pis/task/task_registry.py)
- Step Runner (src/pis/step/)
- Storage Handlers (src/pis/storage/)

b) Task Types (Rounded Rectangles):
- Pre-tasks
- Main Tasks
- Validators

- Configuration Flow
- Task Execution Flow
- Data Movement
- Validation Results

##### Specific Implementation Details

Key architectural elements:

**┌─ Task System ──────────────┐**
- ┌─ Base Task ───────────┐ │
- run()              │ │
- validate()         │ │
- upload()           │ │
- ┘ │
- ┌─ Concrete Tasks ──────┐ │
- Download           │ │
- ElasticSearch      │ │
- Explode            │ │
- ┘ │
- ┘

##### Important Relationships

- Configuration flow from CLI/ENV/YAML to Task Registry
- Task Registry to Step Runner relationship
- Step Runner to Storage System interaction
- Manifest generation and management flow
- Validation system integration points

##### Additional Elements

Supporting components:
- Logging System (src/pis/util/logger.py)
- Error Handling (src/pis/util/errors.py)
- File System Utilities (src/pis/util/fs.py)

##### Execution Flow Representation

Four-phase execution process:
**[Init] → [Staging] → [Validation] → [Upload]**
- ↓          ↓            ↓           ↓
**[Pretasks] [Tasks] [Validators] [Storage]**

##### Notes for Implementation

- Keep the parallel execution nature visible
- Manifest file generation and management
- Integration points with external services
- Error handling and validation flows
- Configuration inheritance and override patterns

Remember to maintain a balance between detail and clarity, focusing on the pipeline nature of the system while showing the parallel execution capabilities and the four-phase approach to task execution.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef input fill:#a8d1f0,stroke:#333
    classDef processing fill:#90EE90,stroke:#333
    classDef storage fill:#FFB347,stroke:#333
    classDef validation fill:#FFE5B4,stroke:#333
    classDef error fill:#FFB6C1,stroke:#333
    classDef task fill:#E6E6FA,stroke:#333

    %% Input Layer
    subgraph Input
        CLI["CLI Interface"]
        ConfigParser["Config Parser"]
        EnvVars["Environment Variables"]
        ConfigSystem["Configuration Management"]
    end
    click CLI "https://github.com/javfg/pis/blob/main/src/pis/config/cli.py"
    click ConfigParser "https://github.com/javfg/pis/blob/main/src/pis/config/config.py"
    click EnvVars "https://github.com/javfg/pis/blob/main/src/pis/config/env.py"
    click ConfigSystem "https://github.com/javfg/pis/tree/main/src/pis/config/"

    %% Processing Layer
    subgraph Processing
        TaskRegistry["Task Registry"]
        StepRunner["Step Runner"]
        ManifestMgr["Manifest Manager"]
    end
    click TaskRegistry "https://github.com/javfg/pis/blob/main/src/pis/task/task_registry.py"
    click StepRunner "https://github.com/javfg/pis/blob/main/src/pis/step/step.py"
    click ManifestMgr "https://github.com/javfg/pis/tree/main/src/pis/manifest/"

    %% Task System
    subgraph Tasks
        BaseTask["Base Task"]
        DownloadTask["Download Task"]
        ESTask["ElasticSearch Task"]
        ExplodeTask["Explode Task"]
    end
    click BaseTask "https://github.com/javfg/pis/blob/main/src/pis/task/task.py"
    click DownloadTask "https://github.com/javfg/pis/blob/main/src/pis/tasks/download.py"
    click ESTask "https://github.com/javfg/pis/blob/main/src/pis/tasks/elasticsearch.py"
    click ExplodeTask "https://github.com/javfg/pis/blob/main/src/pis/tasks/explode.py"

    %% Storage Layer
    subgraph Storage
        StorageSystem["Storage System"]
        GoogleStorage["Google Storage"]
        NoopStorage["No-op Storage"]
        RemoteStorage["Remote Storage"]
    end
    click StorageSystem "https://github.com/javfg/pis/tree/main/src/pis/storage/"
    click GoogleStorage "https://github.com/javfg/pis/blob/main/src/pis/storage/google.py"
    click NoopStorage "https://github.com/javfg/pis/blob/main/src/pis/storage/noop.py"
    click RemoteStorage "https://github.com/javfg/pis/blob/main/src/pis/storage/remote_storage.py"

    %% Validation Layer
    subgraph Validation
        ValidationSystem["Validation System"]
        ESValidator["ElasticSearch Validator"]
        FileValidator["File Validator"]
    end
    click ValidationSystem "https://github.com/javfg/pis/tree/main/src/pis/validators/"
    click ESValidator "https://github.com/javfg/pis/blob/main/src/pis/validators/elasticsearch.py"
    click FileValidator "https://github.com/javfg/pis/blob/main/src/pis/validators/file.py"

    %% Utilities
    subgraph Utils
        Logger["Logging System"]
        ErrorHandler["Error Handler"]
        FSUtils["File System Utilities"]
    end
    click Logger "https://github.com/javfg/pis/blob/main/src/pis/util/logger.py"
    click ErrorHandler "https://github.com/javfg/pis/blob/main/src/pis/util/errors.py"
    click FSUtils "https://github.com/javfg/pis/blob/main/src/pis/util/fs.py"

    %% Relationships
    CLI --> ConfigSystem
    ConfigParser --> ConfigSystem
    EnvVars --> ConfigSystem
    ConfigSystem --> TaskRegistry
    TaskRegistry --> StepRunner
    StepRunner --> Tasks
    Tasks --> StorageSystem
    Tasks --> ValidationSystem
    ManifestMgr --> StorageSystem
    StorageSystem --> GoogleStorage
    StorageSystem --> NoopStorage
    StorageSystem --> RemoteStorage
    ValidationSystem --> ESValidator
    ValidationSystem --> FileValidator
    BaseTask --> DownloadTask
    BaseTask --> ESTask
    BaseTask --> ExplodeTask

    %% Apply styles
    class CLI,ConfigParser,EnvVars,ConfigSystem input
    class TaskRegistry,StepRunner,ManifestMgr processing
    class StorageSystem,GoogleStorage,NoopStorage,RemoteStorage storage
    class ValidationSystem,ESValidator,FileValidator validation
    class ErrorHandler error
    class BaseTask,DownloadTask,ESTask,ExplodeTask task
```
