# piotrlaczkowski/keras-data-processor architecture

> Architecture diagram of the GitHub repository piotrlaczkowski/keras-data-processor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/piotrlaczkowski/keras-data-processor
- Repository: https://github.com/piotrlaczkowski/keras-data-processor

## Overview

##### Project Type and Purpose
- KDP is a Python-based data preprocessing library built on top of TensorFlow and Keras. Its purpose is to efficiently transform raw data into processed features ready for machine learning models. The README confirms that the tool handles tasks such as feature normalization, distribution-aware encoding, tabular attention, feature selection, and even specialized processing via a feature-wise mixture of experts.
- It is designed to be production-ready and can be integrated directly as a preprocessing layer in ML models.

##### File Structure
- The main code resides in the “kdp” directory, which contains the core modules:
– Base files (e.g., auto_config.py, dynamic_pipeline.py, features.py, pipeline.py, processor.py, stats.py, model_advisor.py, moe.py) that likely orchestrate the preprocessing.
– A “layers” subdirectory with specific implementations (e.g., distribution aware, gated layers, transformer, text preprocessing) that serve as building blocks for feature processing.
– “layers_factory.py” which suggests use of the factory pattern to instantiate the correct processing layer based on feature type.
- The existence of detailed documentation (docs folder), comprehensive testing (test and tests directories), and CI/CD workflows (GitHub workflows, pre-commit hooks, etc.) shows that the project emphasizes quality, extensibility, and maintainability.

##### README for Additional Insights
- The README explains that KDP builds a preprocessing pipeline as a Keras model that can be used either independently or as a layer.
- It highlights key features like “efficient single-pass processing”, “distribution-aware encoding”, “tabular attention”, and a “feature-wise mixture of experts.” These features align with different layers implemented under “kdp/layers.”

a. Main Components:
– Client/External User: Consumer code that imports KDP and configures a preprocessing model.
– Preprocessing Model Interface: The high-level API (e.g., PreprocessingModel) that users interact with.
– Dynamic Pipeline: Module (in kdp/dynamic_pipeline.py and kdp/pipeline.py) that assembles various processing steps.
– Feature Processing Layers: Represented by the different specialized layers (e.g., date encoding, transformer block, tabular attention) grouped under “kdp/layers.”
– Layers Factory: (kdp/layers_factory.py) that instantiates and wires the appropriate layers based on feature specifications.
– Testing and CI/CD Modules: Represent external business rules and code quality frameworks (although not part of the runtime, they are crucial for ensuring the architecture remains robust).

b. Relationships and Interactions:
– Clearly denote integrations with TensorFlow/Keras as the underlying framework that hosts the final model.
– Optionally, show side components such as testing modules and developer scripts connecting to CI/CD workflows indicating continuous validation and documentation generation.

c. Architectural Patterns and Design Principles:
– Highlight the use of the Factory Pattern in “layers_factory.py” to manage layer instantiation.
– Indicate the Pipeline Pattern that is implemented by modules like dynamic_pipeline.py and pipeline.py, ensuring single-pass processing.
– Show the integration with Keras’s Model architecture – the preprocessing steps are built into a model that can be plugged directly into any larger ML pipeline.

d. Technologies, Frameworks, and Libraries:
– Python is used as the core language.
– TensorFlow and Keras serve as the backbone for building and training models.
– Documentation (mkdocs, Sphinx-like tooling) and CI/CD integration via GitHub Actions are supporting technologies.
– Utility scripts (for doc generation and analysis) indicate an emphasis on maintainability and ease of developer adoption.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Client Layer"
        CU("Client/External User"):::client
    end

    subgraph "Core Processing"
        PM("Preprocessing Model Interface"):::core
        DP("Dynamic Pipeline (dynamic_pipeline.py)"):::core
        PL("Pipeline (pipeline.py)"):::core
        LF("Layers Factory"):::core
        FP("Feature Processing Layers"):::core
        TF("TensorFlow/Keras Model"):::external
    end

    subgraph "Developer Tools"
        SU("Supporting Utilities"):::utility
        TC("Testing & CI/CD Modules"):::utility
    end

    CU -->|"APIcall"| PM
    PM -->|"assemble"| DP
    DP -->|"processChain"| PL
    PL -->|"invoke"| LF
    LF -->|"instantiate"| FP
    FP -->|"integrate"| TF

    PM -->|"supportTools"| SU
    SU -->|"validate"| TC

    %% Click Events
    click CU "https://github.com/piotrlaczkowski/keras-data-processor/blob/main/examples/custom_preprocessing_example.py"
    click PM "https://github.com/piotrlaczkowski/keras-data-processor/blob/main/kdp/processor.py"
    click DP "https://github.com/piotrlaczkowski/keras-data-processor/blob/main/kdp/dynamic_pipeline.py"
    click PL "https://github.com/piotrlaczkowski/keras-data-processor/blob/main/kdp/pipeline.py"
    click LF "https://github.com/piotrlaczkowski/keras-data-processor/blob/main/kdp/layers_factory.py"
    click FP "https://github.com/piotrlaczkowski/keras-data-processor/tree/main/kdp/layers/"
    click SU "https://github.com/piotrlaczkowski/keras-data-processor/blob/main/scripts/generate_model_diagrams.py"
    click TC "https://github.com/piotrlaczkowski/keras-data-processor/tree/main/test/"

    %% Styles
    classDef client fill:#F4D03F,stroke:#333,stroke-width:2px;
    classDef core fill:#A9DFBF,stroke:#333,stroke-width:2px;
    classDef external fill:#85C1E9,stroke:#333,stroke-width:2px;
    classDef utility fill:#F5B7B1,stroke:#333,stroke-width:2px;
```
