How edanalytics/earthmover works
Project Type and Purpose
• earthmover is an open-source Python CLI/library that “moves” tabular data (flat files, database queries, FTP files) through a configurable pipeline—defined in YAML—into text‐based outputs (JSONL, XML).
• It’s not a web app or microservice; it’s a standalone data‐transformation tool with a core engine, a plugin-style node/operation framework, and a CLI entry point.
File Structure
• Top level contains packaging/configuration (setup.py, requirements.txt, mkdocs.yml) and documentation (docs/).
• The earthmover/ package contains:
– __main__.py – CLI entry
– yaml_parser.py – turns user YAML into an internal spec
– graph.py – builds the execution graph (DAG) of nodes
– nodes/ – source, transformation, destination node classes
– operations/ – elemental operations (row, column, dataframe, groupby)
– runs_file.py – persists run metadata
– error_handler.py & util.py – logging and error management
– include/starter_project – boilerplate for new users
• example_projects/ – sample configs & sources to illustrate use cases
• earthmover/tests/ – integration tests that exercise the full flow
README for Additional Insights
• Confirms the YAML→pipeline→text output flow
• Points to docs/design.md for deeper architectural discussion (use that file to enrich labels, if desired)
Main Components & Their Relationships
a. CLI / Entry Point
• earthmover.__main__.py parses CLI arguments, locates the YAML config, invokes the engine.
b. Configuration Parser
• yaml_parser.py reads/parses YAML into in-memory structures (nodes, operations, run parameters).
c. Graph Builder
• graph.py uses the parsed config to instantiate Node objects and connect them into a directed graph (DAG) representing data flow.
d. Node & Operation Layer
• Nodes (source, transformation, destination) wrap individual responsibilities.
• Each node delegates to Operation classes for actual data manipulation (e.g., dataframe join, groupby).
e. Execution Engine
• Traverses the DAG, invoking each node in topological order, passing data frames or streams from sources through chained operations to outputs.
f. I/O / External Systems
• File system (CSV, JSONL, XML templates)
• Optional FTP fetch or database query nodes (configured via operations or custom nodes)
g. Utilities & Error Handling
• error_handler.py centralizes how failures are captured, logged, and surfaced.
• runs_file.py records metadata about each pipeline execution.
Architectural Patterns & Design Principles
• DAG Pattern – the pipeline is explicitly modeled as a directed acyclic graph for safe, repeatable execution.
• Separation of Concerns – parsing, graph construction, execution, and I/O are each handled in dedicated modules.
Diagramming Conventions
– CLI → Parser → Graph Builder → Executor → I/O
• Labels: