# sogis/gretl architecture

> Architecture diagram of the GitHub repository sogis/gretl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sogis/gretl
- Repository: https://github.com/sogis/gretl

## Overview

##### Project Type and Purpose
- GRETL is an open-source, multi-project Gradle plugin and CLI runtime for performing SQL-centric (geo) data ETL workflows. It provides a set of prebuilt “tasks” (Av2ch, CsvImport, Gpkg2Shp, S3Upload, etc.) and underlying “steps” (Csv2ExcelStep, SqlExecutorStep, S3DownloadStep, etc.) that users can configure and run inside their Gradle build scripts or via a Dockerized runtime image.

##### High-Level File Structure
- Root folder: CI/CD definitions (.github workflows, Jenkinsfile), top-level Gradle config (build.gradle, settings.gradle, versioning.gradle), documentation (docs/), OpenShift manifests.
- Subproject “gretl”: core Java plugin code (src/main/java), unit tests (src/test/java), extensive integration tests (src/integrationTest), Gradle plugin descriptor (META-INF/gradle-plugins), build.gradle.
- Subproject “runtimeImage”: Dockerfile(s), entrypoint script (start-gretl.sh), Gradle init script to stage native extensions, build.gradle to assemble a containerized GRETL runtime.

##### Key Components
a. Developer/User Environment
- Gradle CLI or IDE invokes the GRETL plugin (gretl plugin ID).
- In Docker mode, user runs the GRETL image, passing job definitions.
b. GRETL Plugin Layer (Java)
- API & Plugin Registration (Connector, Endpoint, TransferSet, GretlPlugin)
- Logging Abstraction (GretlLogger, adapters)
- Task Definitions (one class per ETL task, e.g. CsvImport, GpkgExport)
- Step Implementations (one per processing step, e.g. Csv2ExcelStep, SqlExecutorStep)
- Utilities (DbConnector, SqlReader, Grooming, SimiSvcClient)
c. External Resource Endpoints
- Databases: PostgreSQL/PostGIS, SQLite, DuckDB (via JDBC connectors)
- Filesystems: Local disk, S3 buckets (via AWS SDK), HTTP/FTP endpoints
- Formats: Shapefiles (.shp/.dbf/.prj), GeoPackage (.gpkg), XSLT transformations, compressed archives (gzip)
d. CI/CD & Packaging
- GitHub Actions and Jenkins run unit & integration tests (spinning up ephemeral DB or using init_postgresql.sql).
- “runtimeImage” builds a Docker image that bundles the plugin and scripts.
- OpenShift manifests for deploying GRETL runtime in Kubernetes.

##### Relationships and Data Flow
- User (or CI) → invokes Gradle with “apply plugin: 'ch.so.agi.gretl'” → GretlPlugin registers custom tasks in Gradle’s task graph.
- Task Execution → for each configured Task:
1. Instantiate appropriate Step classes.
2. Step obtains a Connector (JDBC, S3, HTTP) via Endpoint.
3. Step reads/writes data (SQL scripts, file I/O, S3 API, XSL transform).
4. Logging layer emits structured logs.
- Integration Tests: run pipeline inside Gradle, verify correctness against test resources (SQL init scripts, sample CSV/GPkg files).
- Docker Runtime: user-supplied job definitions loaded by start-gretl.sh → plugin executes tasks inside container → interacts with outside endpoints via network.

##### Architectural Patterns and Principles
- Plugin Architecture Pattern: GRETL extends Gradle at runtime, injecting tasks via a Plugin class.
- Layered Modules: clear separation between API, plugin loader, task orchestration, step implementations, and connectivity utilities.
- Extension Points: new tasks (Java classes) and steps can be added without modifying core.
- Infrastructure as Code: CI workflows, Dockerfiles, OpenShift manifests codify build/test/deploy pipelines.

##### Essential Elements to Capture
- All major Java packages/modules: api, plugin, steps, tasks, util, logging.
- External dependencies and their roles (PostGIS, DuckDB extensions, AWS S3, XSLT).
- CI workflows and Docker/OpenShift deployment path.
- Precise data flow: from build script parameters → task instantiation → step execution → resource interaction → results.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User/Developer Environment
    subgraph "User/Developer Environment"
        GradleCLI["Gradle CLI"]:::user
        DockerRuntime["Docker Runtime"]:::user
    end

    %% GRETL Plugin Layer
    subgraph "gretl (Java Plugin)"
        subgraph "API & Plugin Registration"
            Connector["Connector"]:::plugin
            Endpoint["Endpoint"]:::plugin
            TransferSet["TransferSet"]:::plugin
            GretlPlugin["GretlPlugin"]:::plugin
        end
        subgraph "Logging Abstraction"
            GretlLogger["GretlLogger"]:::plugin
            LogFactory["LogFactory"]:::plugin
        end
        subgraph "Task Definitions"
            Tasks["Tasks"]:::plugin
        end
        subgraph "Step Implementations"
            Steps["Steps"]:::plugin
        end
        subgraph "Utilities"
            Utils["Utilities"]:::plugin
        end
    end

    %% External Systems
    subgraph "External Resources"
        DB["Databases<br/>PostgreSQL, SQLite, DuckDB"]:::external
        FS["Filesystems<br/>Local, S3, HTTP/FTP"]:::external
        Formats["Formats<br/>Shapefile, GeoPackage, XSLT, gzip"]:::external
    end

    %% CI/CD & Packaging
    subgraph "CI/CD & Packaging"
        GH["GitHub Actions"]:::ci
        Jenkins["Jenkins Pipeline"]:::ci
        RootGradle["Root Gradle Config"]:::ci
        PluginGradle["Plugin build.gradle"]:::ci
        RuntimeGradle["runtimeImage build.gradle"]:::ci
        DockerFile["Dockerfiles & OpenShift"]:::ci
    end

    %% Data Flow and Relationships
    GradleCLI -->|"apply plugin"| GretlPlugin
    DockerRuntime -->|"start-gretl.sh" | GretlPlugin
    GretlPlugin -->|"registers"| Tasks
    Tasks -->|"instantiates"| Steps
    Steps -->|"uses"| Connector
    Connector -->|JDBC| DB
    Steps -->|AWS SDK| FS
    Steps -->|formats| Formats
    Steps -->|"logs to"| LogFactory
    Steps -->|"logs to"| GretlLogger

    GH -->|"runs tests"| Tasks
    GH -->|"builds image"| DockerFile
    Jenkins -->|"runs pipeline"| Tasks
    Jenkins -->|"deploys image"| DockerFile
    RootGradle -->|configures| Tasks
    PluginGradle -->|builds| GretlPlugin
    RuntimeGradle -->|packages| DockerFile

    %% Click Events
    click GradleCLI "https://github.com/sogis/gretl/blob/main/gretl/src/main/resources/META-INF/gradle-plugins/ch.so.agi.gretl.properties"
    click DockerRuntime "https://github.com/sogis/gretl/blob/main/runtimeImage/start-gretl.sh"
    click Connector "https://github.com/sogis/gretl/blob/main/gretl/src/main/java/ch/so/agi/gretl/api/Connector.java"
    click Endpoint "https://github.com/sogis/gretl/blob/main/gretl/src/main/java/ch/so/agi/gretl/api/Endpoint.java"
    click TransferSet "https://github.com/sogis/gretl/blob/main/gretl/src/main/java/ch/so/agi/gretl/api/TransferSet.java"
    click GretlPlugin "https://github.com/sogis/gretl/blob/main/gretl/src/main/java/ch/so/agi/gretl/gradle/GretlPlugin.java"
    click GretlLogger "https://github.com/sogis/gretl/blob/main/gretl/src/main/java/ch/so/agi/gretl/logging/GretlLogger.java"
    click Tasks "https://github.com/sogis/gretl/tree/main/gretl/src/main/java/ch/so/agi/gretl/tasks/"
    click Steps "https://github.com/sogis/gretl/tree/main/gretl/src/main/java/ch/so/agi/gretl/steps/"
    click Utils "https://github.com/sogis/gretl/blob/main/gretl/src/main/java/ch/so/agi/gretl/util/DbConnector.java"
    click GH "https://github.com/sogis/gretl/blob/main/.github/workflows/gretl.yml"
    click Jenkins "https://github.com/sogis/gretl/tree/main/Jenkinsfile"
    click RootGradle "https://github.com/sogis/gretl/blob/main/build.gradle"
    click PluginGradle "https://github.com/sogis/gretl/blob/main/gretl/build.gradle"
    click RuntimeGradle "https://github.com/sogis/gretl/blob/main/runtimeImage/build.gradle"
    click DockerFile "https://github.com/sogis/gretl/tree/main/runtimeImage/gretl/Dockerfile"

    %% Styles
    classDef plugin fill:#cce5ff,stroke:#004085,color:#004085
    classDef external fill:#d4edda,stroke:#155724,color:#155724
    classDef ci fill:#fff3cd,stroke:#856404,color:#856404
    classDef user fill:#e2e3e5,stroke:#6c757d,color:#6c757d
```
