How sogis/gretl works
Project Type and Purpose
• GRETL is an open-source, multi-project Gradle plugin and CLI runtime for performing SQL-centric (geo) data ETL workflows. It provides a set of prebuilt “tasks” (Av2ch, CsvImport, Gpkg2Shp, S3Upload, etc.) and underlying “steps” (Csv2ExcelStep, SqlExecutorStep, S3DownloadStep, etc.) that users can configure and run inside their Gradle build scripts or via a Dockerized runtime image.
High-Level File Structure
• Root folder: CI/CD definitions (.github workflows, Jenkinsfile), top-level Gradle config (build.gradle, settings.gradle, versioning.gradle), documentation (docs/), OpenShift manifests.
• Subproject “gretl”: core Java plugin code (src/main/java), unit tests (src/test/java), extensive integration tests (src/integrationTest), Gradle plugin descriptor (META-INF/gradle-plugins), build.gradle.
• Subproject “runtimeImage”: Dockerfile(s), entrypoint script (start-gretl.sh), Gradle init script to stage native extensions, build.gradle to assemble a containerized GRETL runtime.
Key Components
a. Developer/User Environment
• Gradle CLI or IDE invokes the GRETL plugin (gretl plugin ID).
• In Docker mode, user runs the GRETL image, passing job definitions.
b. GRETL Plugin Layer (Java)
• API & Plugin Registration (Connector, Endpoint, TransferSet, GretlPlugin)
• Logging Abstraction (GretlLogger, adapters)
• Task Definitions (one class per ETL task, e.g. CsvImport, GpkgExport)
• Step Implementations (one per processing step, e.g. Csv2ExcelStep, SqlExecutorStep)
• Utilities (DbConnector, SqlReader, Grooming, SimiSvcClient)
c. External Resource Endpoints
• Databases: PostgreSQL/PostGIS, SQLite, DuckDB (via JDBC connectors)
• Filesystems: Local disk, S3 buckets (via AWS SDK), HTTP/FTP endpoints
• Formats: Shapefiles (.shp/.dbf/.prj), GeoPackage (.gpkg), XSLT transformations, compressed archives (gzip)
d. CI/CD & Packaging
• GitHub Actions and Jenkins run unit & integration tests (spinning up ephemeral DB or using init_postgresql.sql).
• “runtimeImage” builds a Docker image that bundles the plugin and scripts.
• OpenShift manifests for deploying GRETL runtime in Kubernetes.
Relationships and Data Flow
• User (or CI) → invokes Gradle with “apply plugin: 'ch.so.agi.gretl'” → GretlPlugin registers custom tasks in Gradle’s task graph.
• Task Execution → for each configured Task:
1. Instantiate appropriate Step classes.
2. Step obtains a Connector (JDBC, S3, HTTP) via Endpoint.
3. Step reads/writes data (SQL scripts, file I/O, S3 API, XSL transform).
4. Logging layer emits structured logs.
• Integration Tests: run pipeline inside Gradle, verify correctness against test resources (SQL init scripts, sample CSV/GPkg files).
• Docker Runtime: user-supplied job definitions loaded by start-gretl.sh → plugin executes tasks inside container → interacts with outside endpoints via network.
Architectural Patterns and Principles
• Plugin Architecture Pattern: GRETL extends Gradle at runtime, injecting tasks via a Plugin class.
• Layered Modules: clear separation between API, plugin loader, task orchestration, step implementations, and connectivity utilities.
• Extension Points: new tasks (Java classes) and steps can be added without modifying core.
• Infrastructure as Code: CI workflows, Dockerfiles, OpenShift manifests codify build/test/deploy pipelines.
Essential Elements to Capture
• All major Java packages/modules: api, plugin, steps, tasks, util, logging.
• External dependencies and their roles (PostGIS, DuckDB extensions, AWS S3, XSLT).
• CI workflows and Docker/OpenShift deployment path.
• Precise data flow: from build script parameters → task instantiation → step execution → resource interaction → results.