How jotace5/h24_cloud_project works
Main Components
• Backend Application (h24_ai_app): This contains the core backend code (main_file.py), various processors (such as ml_tabular_ext, ml_cat_prediction, ml_page_classifier, etc.), configuration/credential files, and sample input data. It likely acts as the orchestrator that calls the different ML models for data extraction and analysis.
• Data Engineering Pipeline (h24_data_engenieering): This component comprises notebooks, reports, and layer-wise analyses (L1, L2, L3) that process and prepare raw data. These processes feed into model training and quality validation.
• Machine Learning Models (h24_ml_model): This houses several ML model directories (entity_extractor, form_classifier, page_classifier, tabular_extractor), each with its own training datasets, configuration files, credentials, and output analysis. They encapsulate the specific algorithms used for document detection, classification, and data extraction.
Relationships and Interactions
• Data Flow:
• The system receives raw document data (e.g., PDFs from the input_data folder in h24_ai_app).
• The backend application (h24_ai_app) processes these files through its processors which then call the corresponding ML modules.
• The ML models located in h24_ml_model perform specific tasks (entity extraction, page classification, tabular extraction, form classification) and return processed data.
• The data engineering component (h24_data_engenieering) deals with the post-processing, analysis, and preparation of training as well as validation datasets helping to refine the models over time.
• Dependencies:
• The backend relies on credentials and configuration files stored in its “keys” directory to securely access the ML models and possibly external services.
• The ML model components might exchange data (or metadata) with the data engineering process to maintain updated training information.
Architectural Patterns & Design Principles
• Modular Design: The clear separation into three main modules (backend, ML models, and data engineering) suggests a modular architecture where each layer is independently developed and maintained.
• Pipeline Processing: The system follows a pipeline approach – document ingestion, processing through different ML models, and subsequent analysis – which is typical in data engineering and machine learning workflows.
• Clear Separation of Concerns: Different directories handle distinct aspects such as model training (h24_ml_model), data preparation and validation (h24_data_engenieering), and application logic (h24_ai_app).
Relevant Technologies & Frameworks
• Python is the core programming language.
• FastAPI is indicated as a dependency (as seen in the README badges) and is likely used to expose or serve the backend application.
• Data processing libraries include Pandas, NumPy, Matplotlib, Seaborn, and Scikit-learn.
• Jupyter notebooks are used within the data engineering component for interactive analysis and experimentation.
• Google Cloud is referenced and may be used for model hosting or additional cloud services.
Specific Components
• h24_ai_app:
• Main entry point (main_file.py)
• Processors (ml_cat_prediction, ml_evol_data_ext, ml_key_value_pair_ext, ml_page_classifier, ml_tabular_ext)
• Input and temporary output directories (input_data, temp)
• Configuration/credentials management (keys folder)
• h24_ml_model:
• Each ML subcomponent as separate modules:
▪ Entity Extractor
▪ Form Classifier
▪ Page Classifier
▪ Tabular Extractor
• Training and testing data pipelines within those modules.
• h24_data_engenieering:
• Data Layers (Layer 1, Layer 2, Layer 3)
• Notebooks and metadata files that drive the extraction and training process.
• Quality checking components such as “Qlty Datasets”