# growgraph/ethexco architecture

> Architecture diagram of the GitHub repository growgraph/ethexco, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/growgraph/ethexco
- Repository: https://github.com/growgraph/ethexco

## Overview

##### Project type and purpose
- This repository, “EthExCo,” is a Python-based data-processing and model-training pipeline (open-source tool) that ingests raw ethics-related data, breaks it into “frames,” augments and labels those frames, assembles a dataset, and finally trains a model (including a “modal” variant).
- It is not a front-end/back-end web app, but rather a command-line-driven ETL + ML pipeline library with reusable components under `ethexco/` and orchestration scripts in `run/`.

##### File structure
- Top-level config and metadata:
- `.gitignore`, `.pylintrc`, `mypy.ini`, `pyproject.toml`
- Pre-commit and virtual-env setup files
- Core library: `ethexco/`
- `frame.py` (frame-creation logic)
- `util.py` (shared utilities)
- Orchestration scripts: `run/`
- `chunk_raw_data.py`: splits raw inputs
- `generate_frames.py`: applies frame logic
- `frames_add_text.py`: annotates frames with text
- `generate_dataset.py`: bundles frames into dataset
- `train.py` & `train_modal.py`: kick off model training
- `testing/`: unit tests for pipelines and frames
- Lock file: `uv.lock` suggests use of the “uv” CLI for environment management.

##### README for additional insights
- Workflow steps (chunk → frames → dataset → train)
- Dev setup uses “uv” to create and sync virtual environments
- No external services or web-APIs, purely file-based I/O and local compute

##### Elements
- Clear, self-explanatory labels for each pipeline stage and module
- Blue rectangles for Python modules (`ethexco/`)
- Green rounded rectangles for orchestration scripts (`run/`)
- Orange parallelograms for environment/tooling (venv, uv)

##### Detailed capture of architecture
- Do not oversimplify: represent each script as its own node
- Interactions between scripts and library functions

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Stores
    RawData["Raw Data Store"]:::data
    ChunkedData["Chunked Data"]:::data
    FramesData["Frames"]:::data
    AnnotatedFrames["Annotated Frames"]:::data
    DatasetData["Dataset"]:::data
    ModelOutput["Model Output"]:::data
    ModalModelOutput["Modal Model Output"]:::data

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        ChunkRaw["chunk_raw_data.py"]:::script
        GenFrames["generate_frames.py"]:::script
        AddText["frames_add_text.py"]:::script
        GenDataset["generate_dataset.py"]:::script
        Train["train.py"]:::script
        TrainModal["train_modal.py"]:::script
    end

    %% Core Library
    subgraph "Core Library"
        Init["__init__.py"]:::library
        FrameModule["frame.py"]:::library
        UtilModule["util.py"]:::library
    end

    %% Test Harness
    subgraph "Test Harness"
        TestFrames["test_frames.py"]:::test
        TestDatasets["test_datasets.py"]:::test
        TestModal["test_modal.py"]:::test
    end

    %% Environment & Tooling
    subgraph "Environment & Tooling"
        Pyproject["pyproject.toml"]:::env
        UVLock["uv.lock"]:::env
        Pylintrc[".pylintrc"]:::env
        MypyIni[".mypy.ini"]:::env
        PreCommit[".pre-commit-config.yaml"]:::env
    end

    %% Flow of Data
    RawData --> ChunkRaw --> ChunkedData --> GenFrames --> FramesData --> AddText --> AnnotatedFrames --> GenDataset --> DatasetData
    DatasetData --> Train --> ModelOutput
    DatasetData --> TrainModal --> ModalModelOutput

    %% Scripts call library functions
    ChunkRaw --> FrameModule
    ChunkRaw --> UtilModule
    GenFrames --> FrameModule
    GenFrames --> UtilModule
    AddText --> UtilModule
    GenDataset --> UtilModule
    Train --> UtilModule
    TrainModal --> UtilModule

    %% Test interactions
    TestFrames --> FrameModule
    TestDatasets --> GenDataset
    TestModal --> TrainModal

    %% Click Events
    click FrameModule "https://github.com/growgraph/ethexco/blob/main/ethexco/frame.py"
    click UtilModule "https://github.com/growgraph/ethexco/blob/main/ethexco/util.py"
    click Init "https://github.com/growgraph/ethexco/blob/main/ethexco/__init__.py"
    click ChunkRaw "https://github.com/growgraph/ethexco/blob/main/run/chunk_raw_data.py"
    click GenFrames "https://github.com/growgraph/ethexco/blob/main/run/generate_frames.py"
    click AddText "https://github.com/growgraph/ethexco/blob/main/run/frames_add_text.py"
    click GenDataset "https://github.com/growgraph/ethexco/blob/main/run/generate_dataset.py"
    click Train "https://github.com/growgraph/ethexco/blob/main/run/train.py"
    click TrainModal "https://github.com/growgraph/ethexco/blob/main/run/train_modal.py"
    click TestFrames "https://github.com/growgraph/ethexco/blob/main/run/testing/test_frames.py"
    click TestDatasets "https://github.com/growgraph/ethexco/blob/main/run/testing/test_datasets.py"
    click TestModal "https://github.com/growgraph/ethexco/blob/main/run/testing/test_modal.py"
    click Pyproject "https://github.com/growgraph/ethexco/blob/main/pyproject.toml"
    click UVLock "https://github.com/growgraph/ethexco/blob/main/uv.lock"
    click Pylintrc "https://github.com/growgraph/ethexco/blob/main/.pylintrc"
    click MypyIni "https://github.com/growgraph/ethexco/blob/main/.mypy.ini"
    click PreCommit "https://github.com/growgraph/ethexco/blob/main/.pre-commit-config.yaml"

    %% Styles
    classDef library fill:#D5E8D4,stroke:#4B8BBE,color:#000
    classDef script fill:#D5F5E1,stroke:#27AE60,color:#000
    classDef data fill:#FCF3CF,stroke:#F1C40F,color:#000
    classDef test fill:#E8DAEF,stroke:#8E44AD,color:#000
    classDef env fill:#FDEDEC,stroke:#E74C3C,color:#000
```
