# edanalytics/earthmover architecture

> Architecture diagram of the GitHub repository edanalytics/earthmover, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/edanalytics/earthmover
- Repository: https://github.com/edanalytics/earthmover

## Overview

##### Project Type and Purpose
- earthmover is an open-source Python CLI/library that “moves” tabular data (flat files, database queries, FTP files) through a configurable pipeline—defined in YAML—into text‐based outputs (JSONL, XML).
- It’s not a web app or microservice; it’s a standalone data‐transformation tool with a core engine, a plugin-style node/operation framework, and a CLI entry point.

##### File Structure
- Top level contains packaging/configuration (setup.py, requirements.txt, mkdocs.yml) and documentation (docs/).
- The earthmover/ package contains:
– __main__.py – CLI entry
– yaml_parser.py – turns user YAML into an internal spec
– graph.py – builds the execution graph (DAG) of nodes
– nodes/ – source, transformation, destination node classes
– operations/ – elemental operations (row, column, dataframe, groupby)
– runs_file.py – persists run metadata
– error_handler.py & util.py – logging and error management
– include/starter_project – boilerplate for new users
- example_projects/ – sample configs & sources to illustrate use cases
- earthmover/tests/ – integration tests that exercise the full flow

##### README for Additional Insights
- Confirms the YAML→pipeline→text output flow
- Points to docs/design.md for deeper architectural discussion (use that file to enrich labels, if desired)

##### Main Components & Their Relationships
a. CLI / Entry Point
- earthmover.__main__.py parses CLI arguments, locates the YAML config, invokes the engine.
b. Configuration Parser
- yaml_parser.py reads/parses YAML into in-memory structures (nodes, operations, run parameters).
c. Graph Builder
- graph.py uses the parsed config to instantiate Node objects and connect them into a directed graph (DAG) representing data flow.
d. Node & Operation Layer
- Nodes (source, transformation, destination) wrap individual responsibilities.
- Each node delegates to Operation classes for actual data manipulation (e.g., dataframe join, groupby).
e. Execution Engine
- Traverses the DAG, invoking each node in topological order, passing data frames or streams from sources through chained operations to outputs.
f. I/O / External Systems
- File system (CSV, JSONL, XML templates)
- Optional FTP fetch or database query nodes (configured via operations or custom nodes)
g. Utilities & Error Handling
- error_handler.py centralizes how failures are captured, logged, and surfaced.
- runs_file.py records metadata about each pipeline execution.

##### Architectural Patterns & Design Principles
- DAG Pattern – the pipeline is explicitly modeled as a directed acyclic graph for safe, repeatable execution.
- Separation of Concerns – parsing, graph construction, execution, and I/O are each handled in dedicated modules.

##### Diagramming Conventions
– CLI → Parser → Graph Builder → Executor → I/O
- Labels:

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Engine Components
    subgraph "Core Engine" 
        direction TB
        CLI["CLI / Entry Point"]:::core
        Parser["Configuration Parser"]:::core
        GraphBuilder["Graph Builder"]:::core
        Executor["Execution Engine"]:::core
        RunsFile["Run Metadata Persistence"]:::core
        ErrorHandler["Error Handling"]:::core
        Util["Utility Functions"]:::core
    end

    %% Plugin Extension Points
    subgraph "Plugin Extension Points" 
        direction TB
        subgraph "Node Framework" 
            direction TB
            NodeBase["Base Node Class"]:::plugin
            SourceNode["Source Nodes"]:::plugin
            TransformNode["Transformation Nodes"]:::plugin
            DestNode["Destination Nodes"]:::plugin
        end
        subgraph "Operation Framework"
            direction TB
            OpBase["Base Operation Class"]:::plugin
            RowOp["Row Operations"]:::plugin
            ColOp["Column Operations"]:::plugin
            DFOp["DataFrame Operations"]:::plugin
            GBOp["GroupBy Operations"]:::plugin
        end
    end

    %% External I/O Systems
    subgraph "External Systems"
        direction TB
        FileSystem["File System (CSV/JSONL/XML)"]:::external
        DBFTP["DB / FTP"]:::external
        ExampleProjects["Example Projects"]:::external
        StarterProject["Starter Project Boilerplate"]:::external
        Tests["Integration Tests"]:::external
    end

    %% Data Flow
    CLI -->|"CLI args & YAML path"| Parser
    Parser -->|"YAML spec"| GraphBuilder
    GraphBuilder -->|"DAG of Nodes"| Executor
    Executor -->|"calls"| NodeBase
    Executor -->|"delegates to"| OpBase
    Executor -->|"logs & errors"| ErrorHandler
    Executor -->|"persists run"| RunsFile
    Executor -->|"util functions"| Util
    Executor -->|"writes output"| FileSystem
    SourceNode -->|feeds data| GraphBuilder
    TransformNode -->|applies transforms| GraphBuilder
    DestNode -->|collects results| GraphBuilder
    RowOp -->|operation| Executor
    ColOp -->|operation| Executor
    DFOp -->|operation| Executor
    GBOp -->|operation| Executor
    DBFTP -->|source/destination| SourceNode
    ExampleProjects -->|samples| CLI
    StarterProject -->|boilerplate| Parser
    Tests -->|validate flow| Executor

    %% Click Events
    click CLI "https://github.com/edanalytics/earthmover/blob/main/earthmover/__main__.py"
    click Parser "https://github.com/edanalytics/earthmover/blob/main/earthmover/yaml_parser.py"
    click GraphBuilder "https://github.com/edanalytics/earthmover/blob/main/earthmover/graph.py"
    click Executor "https://github.com/edanalytics/earthmover/blob/main/earthmover/earthmover.py"
    click RunsFile "https://github.com/edanalytics/earthmover/blob/main/earthmover/runs_file.py"
    click ErrorHandler "https://github.com/edanalytics/earthmover/blob/main/earthmover/error_handler.py"
    click Util "https://github.com/edanalytics/earthmover/blob/main/earthmover/util.py"
    click NodeBase "https://github.com/edanalytics/earthmover/blob/main/earthmover/nodes/node.py"
    click SourceNode "https://github.com/edanalytics/earthmover/blob/main/earthmover/nodes/source.py"
    click TransformNode "https://github.com/edanalytics/earthmover/blob/main/earthmover/nodes/transformation.py"
    click DestNode "https://github.com/edanalytics/earthmover/blob/main/earthmover/nodes/destination.py"
    click OpBase "https://github.com/edanalytics/earthmover/blob/main/earthmover/operations/operation.py"
    click RowOp "https://github.com/edanalytics/earthmover/blob/main/earthmover/operations/row.py"
    click ColOp "https://github.com/edanalytics/earthmover/blob/main/earthmover/operations/column.py"
    click DFOp "https://github.com/edanalytics/earthmover/blob/main/earthmover/operations/dataframe.py"
    click GBOp "https://github.com/edanalytics/earthmover/blob/main/earthmover/operations/groupby.py"
    click StarterProject "https://github.com/edanalytics/earthmover/tree/main/earthmover/include/starter_project/"
    click ExampleProjects "https://github.com/edanalytics/earthmover/tree/main/example_projects/"
    click Tests "https://github.com/edanalytics/earthmover/tree/main/earthmover/tests/"

    %% Styles
    classDef core fill:#AEDFF7,stroke:#025E8E,color:#000
    classDef plugin fill:#E7F9C4,stroke:#4E8A4F,color:#000
    classDef external fill:#FDE2E1,stroke:#D32F2F,color:#000
```
