# smwko/data-ingestion-framework architecture

> Architecture diagram of the GitHub repository smwko/data-ingestion-framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/smwko/data-ingestion-framework
- Repository: https://github.com/smwko/data-ingestion-framework

## Overview

##### Project Type & Purpose
- This project is a Python-based data processing pipeline. Its primary goal is to ingest, transform, and process data (likely event data in JSON format) while using a modular structure.
- The structure suggests an ETL-like workflow with clearly separated components for configuration, data ingestion, transformations, orchestration (engine), and utility functions such as logging.

##### Main Components
a. Data Source:
- “data” directory as the source of input events (e.g., JSON files under data/input/events/person).
b. Configuration Layer:
- The “src/config” folder with metadata files (both JSON and Python) indicates the use of external configurations. This component will be responsible for system settings, metadata definitions, etc.
c. Ingestion Layer:
- In “src/ingestion”, you have modules for reading (reader.py) and writing (writer.py) data. This layer is responsible for interfacing with the raw data files.
d. Transformation Layer:
- In “src/transformations”, the add_fields.py and validator.py modules suggest that data is processed in stages—fields are computed/added and then validated.
e. Engine / Orchestration:
- The “src/engine/dataflow_engine.py” indicates the presence of a central engine that drives the data flow through the ingestion and transformation steps.
f. Utility Layer:
- “src/utils/logger.py” plays a cross-cutting role, providing logging capabilities across the application.
g. Entry Points and Setup:
- main.py acts as the primary executable that orchestrates the workflow.
- setup.py and .gitignore indicate the project’s packaging and version control processes.

##### Relationships and Interactions
- Data flows from the “data/input” directory into the ingestion layer where reader.py picks up the incoming JSON events.
- The data is then passed from the ingestion module to the transformation layer where add_fields.py enriches the data and validator.py ensures the integrity/validity of the data.
- The engine (dataflow_engine.py) coordinates the entire sequence: it calls the ingestion functions, applies the necessary transformations, and then uses the writer (in ingestion) to output or persist the results.
- The configuration layer (src/config) supplies critical metadata and parameters used throughout the process.
- Utility functions such as logging help track the workflow’s progression and help with debugging or monitoring.

##### Architectural Patterns and Design Principles
- Modular Separation: Each major function (ingestion, transformation, engine, configuration, utilities) is encapsulated in its own directory/module.
- Pipeline / ETL Architecture: The system follows a pipeline where data is extracted, transformed, and then forwarded for further processing or storage.
- Dependency Injection / Loose Coupling: Components use configuration files (metadata.json/metadata.py) and utility modules, supporting easier maintenance and testing.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Entry & Setup"
        MAIN("main.py"):::entrypoint
        SETUP("setup.py"):::entrypoint
    end

    subgraph "Dataflow Engine"
        ENGINE("Dataflow Engine (dataflow_engine.py)"):::engine
    end

    subgraph "Configuration"
        CONFIG("Configuration (metadata.json, metadata.py)"):::config
    end

    subgraph "Ingestion"
        READER("Reader (reader.py)"):::ingestion
        WRITER("Writer (writer.py)"):::ingestion
    end

    subgraph "Transformation"
        ADDFIELDS("Add Fields (add_fields.py)"):::transformation
        VALIDATOR("Validator (validator.py)"):::transformation
    end

    DATA("Data Source: data/input/events/person"):::source
    LOGGER("Logger (logger.py)"):::utility

    %% Flow Connections
    MAIN -->|"invokes"| ENGINE
    ENGINE -->|"loads"| CONFIG
    ENGINE -->|"triggers"| READER
    ENGINE -->|"triggers"| WRITER
    ENGINE -->|"orchestrates"| ADDFIELDS
    ENGINE -->|"orchestrates"| VALIDATOR

    DATA -->|"provides_data"| READER
    READER -->|"passes_data"| ADDFIELDS
    ADDFIELDS -->|"enriches"| VALIDATOR
    VALIDATOR -->|"outputs_validated_data"| WRITER

    %% Logger usage (dotted lines)
    ENGINE ---|"logs"| LOGGER
    READER ---|"logs"| LOGGER
    WRITER ---|"logs"| LOGGER
    ADDFIELDS ---|"logs"| LOGGER
    VALIDATOR ---|"logs"| LOGGER
    CONFIG ---|"logs"| LOGGER
    MAIN ---|"logs"| LOGGER

    %% Click Event Mappings
    click DATA "https://github.com/smwko/data-ingestion-framework/tree/main/data/input/events/person"
    click CONFIG "https://github.com/smwko/data-ingestion-framework/tree/main/src/config"
    click READER "https://github.com/smwko/data-ingestion-framework/blob/main/src/ingestion/reader.py"
    click WRITER "https://github.com/smwko/data-ingestion-framework/blob/main/src/ingestion/writer.py"
    click ADDFIELDS "https://github.com/smwko/data-ingestion-framework/blob/main/src/transformations/add_fields.py"
    click VALIDATOR "https://github.com/smwko/data-ingestion-framework/blob/main/src/transformations/validator.py"
    click ENGINE "https://github.com/smwko/data-ingestion-framework/blob/main/src/engine/dataflow_engine.py"
    click LOGGER "https://github.com/smwko/data-ingestion-framework/blob/main/src/utils/logger.py"
    click MAIN "https://github.com/smwko/data-ingestion-framework/blob/main/main.py"
    click SETUP "https://github.com/smwko/data-ingestion-framework/blob/main/setup.py"

    %% Styling
    classDef entrypoint fill:#ddf,stroke:#333,stroke-width:2px;
    classDef engine fill:#ffb,stroke:#333,stroke-width:2px;
    classDef config fill:#bbf,stroke:#333,stroke-width:2px;
    classDef ingestion fill:#dfe,stroke:#333,stroke-width:2px;
    classDef transformation fill:#fdd,stroke:#333,stroke-width:2px;
    classDef source fill:#f9f,stroke:#333,stroke-width:2px;
    classDef utility fill:#dfd,stroke:#333,stroke-width:2px;
```
