# jotace5/h24_cloud_project architecture

> Architecture diagram of the GitHub repository jotace5/h24_cloud_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jotace5/h24_cloud_project
- Repository: https://github.com/jotace5/h24_cloud_project

## Overview

##### Main Components
- Backend Application (h24_ai_app): This contains the core backend code (main_file.py), various processors (such as ml_tabular_ext, ml_cat_prediction, ml_page_classifier, etc.), configuration/credential files, and sample input data. It likely acts as the orchestrator that calls the different ML models for data extraction and analysis.
- Data Engineering Pipeline (h24_data_engenieering): This component comprises notebooks, reports, and layer-wise analyses (L1, L2, L3) that process and prepare raw data. These processes feed into model training and quality validation.
- Machine Learning Models (h24_ml_model): This houses several ML model directories (entity_extractor, form_classifier, page_classifier, tabular_extractor), each with its own training datasets, configuration files, credentials, and output analysis. They encapsulate the specific algorithms used for document detection, classification, and data extraction.

##### Relationships and Interactions
- Data Flow:
- The system receives raw document data (e.g., PDFs from the input_data folder in h24_ai_app).
- The backend application (h24_ai_app) processes these files through its processors which then call the corresponding ML modules.
- The ML models located in h24_ml_model perform specific tasks (entity extraction, page classification, tabular extraction, form classification) and return processed data.
- The data engineering component (h24_data_engenieering) deals with the post-processing, analysis, and preparation of training as well as validation datasets helping to refine the models over time.
- Dependencies:
- The backend relies on credentials and configuration files stored in its “keys” directory to securely access the ML models and possibly external services.
- The ML model components might exchange data (or metadata) with the data engineering process to maintain updated training information.

##### Architectural Patterns & Design Principles
- Modular Design: The clear separation into three main modules (backend, ML models, and data engineering) suggests a modular architecture where each layer is independently developed and maintained.
- Pipeline Processing: The system follows a pipeline approach – document ingestion, processing through different ML models, and subsequent analysis – which is typical in data engineering and machine learning workflows.
- Clear Separation of Concerns: Different directories handle distinct aspects such as model training (h24_ml_model), data preparation and validation (h24_data_engenieering), and application logic (h24_ai_app).

##### Relevant Technologies & Frameworks
- Python is the core programming language.
- FastAPI is indicated as a dependency (as seen in the README badges) and is likely used to expose or serve the backend application.
- Data processing libraries include Pandas, NumPy, Matplotlib, Seaborn, and Scikit-learn.
- Jupyter notebooks are used within the data engineering component for interactive analysis and experimentation.
- Google Cloud is referenced and may be used for model hosting or additional cloud services.

##### Specific Components
- h24_ai_app:
- Main entry point (main_file.py)
- Processors (ml_cat_prediction, ml_evol_data_ext, ml_key_value_pair_ext, ml_page_classifier, ml_tabular_ext)
- Input and temporary output directories (input_data, temp)
- Configuration/credentials management (keys folder)
- h24_ml_model:
- Each ML subcomponent as separate modules:
▪ Entity Extractor
▪ Form Classifier
▪ Page Classifier
▪ Tabular Extractor
- Training and testing data pipelines within those modules.
- h24_data_engenieering:
- Data Layers (Layer 1, Layer 2, Layer 3)
- Notebooks and metadata files that drive the extraction and training process.
- Quality checking components such as “Qlty Datasets”

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Backend Application (h24_ai_app)"
        A1["Input Data"]
        A2["Keys/Config"]
        A3["Main Entry Point (main_file.py)"]
        A4["Processors"]
        A5["Temporary Output"]
    end

    subgraph "Machine Learning Models (h24_ml_model)"
        B1["Entity Extractor"]
        B2["Form Classifier"]
        B3["Page Classifier"]
        B4["Tabular Extractor"]
    end

    subgraph "Data Engineering Pipeline (h24_data_engenieering)"
        C1["Layer 1 Analysis & Notebooks"]
        C2["Layer 2 Analysis"]
        C3["Layer 3 Analysis & Quality Datasets"]
    end

    %% Data Flow from Backend to ML Models
    A1 -->|"feeds"| A3
    A2 -->|"provides"| A3
    A3 -->|"orchestrates"| A4
    A4 -->|"calls"| B1
    A4 -->|"calls"| B2
    A4 -->|"calls"| B3
    A4 -->|"calls"| B4
    A4 -->|"storesOutputIn"| A5

    %% Data Flow from ML Models to Data Engineering
    B1 -->|"outputs"| C1
    B2 -->|"outputs"| C2
    B3 -->|"outputs"| C1
    B4 -->|"outputs"| C3

    %% Feedback loop from Data Engineering to ML Models
    C2 -->|"feedback"| B1

    %% Styles
    classDef backend fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    classDef processor fill:#ffeeba,stroke:#856404,stroke-width:2px;
    classDef ml fill:#ccffcc,stroke:#009900,stroke-width:2px;
    classDef data fill:#cce5ff,stroke:#004085,stroke-width:2px;

    class A1,A2,A3,A5 backend;
    class A4 processor;
    class B1,B2,B3,B4 ml;
    class C1,C2,C3 data;

    %% Click Events for Backend Application
    click A3 "https://github.com/jotace5/h24_cloud_project/blob/main/h24-ai-app/main_file.py"
    click A4 "https://github.com/jotace5/h24_cloud_project/tree/main/h24-ai-app/processors"
    click A1 "https://github.com/jotace5/h24_cloud_project/tree/main/h24-ai-app/input_data"
    click A5 "https://github.com/jotace5/h24_cloud_project/tree/main/h24-ai-app/temp"
    click A2 "https://github.com/jotace5/h24_cloud_project/tree/main/h24-ai-app/keys"

    %% Click Events for Machine Learning Models
    click B1 "https://github.com/jotace5/h24_cloud_project/tree/main/h24_ml_model/ml_model - entity_extractor"
    click B2 "https://github.com/jotace5/h24_cloud_project/tree/main/h24_ml_model/ml_model - form_classifier"
    click B3 "https://github.com/jotace5/h24_cloud_project/tree/main/h24_ml_model/ml_model - page_classifier"
    click B4 "https://github.com/jotace5/h24_cloud_project/tree/main/h24_ml_model/ml_model - tabular_extractor"

    %% Click Events for Data Engineering Pipeline
    click C1 "https://github.com/jotace5/h24_cloud_project/tree/main/h24_data_engenieering/h24_layer1_analysis"
    click C2 "https://github.com/jotace5/h24_cloud_project/tree/main/h24_data_engenieering/h24_layer2_analysis"
    click C3 "https://github.com/jotace5/h24_cloud_project/tree/main/h24_data_engenieering/h24_layer3_analysis"
```
