# stochastic-sisyphus/adv_data_processing_pipeline architecture

> Architecture diagram of the GitHub repository stochastic-sisyphus/adv_data_processing_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stochastic-sisyphus/adv_data_processing_pipeline
- Repository: https://github.com/stochastic-sisyphus/adv_data_processing_pipeline

## Overview

##### Main Components

a) Data Input Layer
- Multiple data source inputs (CSV, Excel, JSON, Parquet, SQL, APIs, S3)
- A data loading orchestrator (from loading.py)

b) Core Processing Pipeline
- Parallel processing capabilities via Dask
- Main processing stages:
- Data Cleaning (cleaning.py)
- Preprocessing (data_preprocessing.py)
- Transformation (transformation.py, data_transformation.py)
- Feature Engineering (feature_engineering.py)
- Dimensionality Reduction (dimensionality_reduction.py)

c) Specialized Processing Units
- Text Analytics Module (text_analytics.py)
- Entity Recognition System (entity_recognition.py)
- Topic Modeling Component (topic_modeling.py)
- Imbalanced Data Handler (imbalanced_data.py)

d) Model Integration Layer
- Model Evaluation Component (model_evaluation.py)
- Optimization Module (optimization.py)
- Feature Selection System (feature_selection.py)

e) Monitoring and Support Systems
- Monitoring System (monitoring.py)
- Logging System (logging_setup.py)
- Error Handling (error_handling.py)
- Metrics Collection (metrics.py)

**[Data Sources] → [Input Layer] → [Core Pipeline] → [Output Layer]**
- ↕             ↕                    ↕
- [Support Systems] [Processing Units] [Monitoring]

##### Specific Implementation Details

**[Data Loading] → [Cleaning] → [Preprocessing] → [Transformation] → [Feature Engineering]**
- ↕             ↕            ↕                   ↕                    ↕
**[Validation] ← [Error Handling] → [Logging] ← [Monitoring] ← [Metrics Collection]**

b) Distributed processing capability:
**[Dask Scheduler]**
- ↓
**[Worker 1] [Worker 2] [Worker N]**

c) A separate section for ML Integration:
**[Feature Selection] → [Model Evaluation] → [Optimization]**
- ↓                   ↓                  ↓
**[Dimensionality Reduction] [Metrics] [Hyperparameter Tuning]**

##### Important Elements to Highlight

- Configuration Management: Show how config.py interfaces with all components
- Batch Processing: Illustrate the batch_processing.py workflow
- Pipeline Flexibility: Show optional processing paths
- Caching Mechanism: Include cache storage and retrieval paths
- GPU Support: Indicate components with GPU acceleration
- API Integration: Show the api_server.py interface

##### Technical Details

- Memory management flows
- Error propagation paths
- Validation checkpoints
- Data persistence points
- Monitoring hooks

This system design should effectively communicate the sophisticated nature of the pipeline while making it easy to understand the flow of data and control through the system.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef dataSource fill:#ADD8E6,stroke:#333
    classDef processing fill:#90EE90,stroke:#333
    classDef monitoring fill:#FFD700,stroke:#333
    classDef error fill:#FFB6C1,stroke:#333
    classDef model fill:#DDA0DD,stroke:#333
    classDef config fill:#F0F8FF,stroke:#333

    %% Data Sources
    subgraph DataSources["Data Sources"]
        CSV["CSV Files"]:::dataSource
        Excel["Excel Files"]:::dataSource
        JSON["JSON Files"]:::dataSource
        Parquet["Parquet Files"]:::dataSource
        SQL["SQL DBs"]:::dataSource
        APIs["APIs"]:::dataSource
        S3["S3 Storage"]:::dataSource
    end

    %% Input Layer
    subgraph InputLayer["Data Input Layer"]
        LoadOrch["Data Loading Orchestrator"]:::processing
        DataVal["Data Validation"]:::processing
    end

    %% Core Pipeline
    subgraph CorePipeline["Core Processing Pipeline"]
        Pipeline["Main Pipeline"]:::processing
        Cleaning["Data Cleaning"]:::processing
        Preproc["Data Preprocessing"]:::processing
        Transform["Data Transformation"]:::processing
        FeatEng["Feature Engineering"]:::processing
        DimRed["Dimensionality Reduction"]:::processing
    end

    %% Specialized Processing
    subgraph SpecialProc["Specialized Processing Units"]
        TextAn["Text Analytics"]:::processing
        EntityRec["Entity Recognition"]:::processing
        TopicMod["Topic Modeling"]:::processing
        ImbalData["Imbalanced Data Handler"]:::processing
    end

    %% Model Integration
    subgraph ModelInt["Model Integration Layer"]
        ModelEval["Model Evaluation"]:::model
        Optim["Optimization"]:::model
        FeatSel["Feature Selection"]:::model
    end

    %% Support Systems
    subgraph Support["Monitoring and Support Systems"]
        Monitor["Monitoring System"]:::monitoring
        Logging["Logging System"]:::monitoring
        ErrorHandle["Error Handling"]:::error
        Metrics["Metrics Collection"]:::monitoring
    end

    %% Configuration
    Config["Configuration Management"]:::config
    BatchProc["Batch Processing"]:::processing
    APIServer["API Server"]:::processing

    %% Connections
    DataSources --> LoadOrch
    LoadOrch --> DataVal
    DataVal --> Pipeline
    Pipeline --> |"Main Flow"| Cleaning
    Cleaning --> Preproc
    Preproc --> Transform
    Transform --> FeatEng
    FeatEng --> DimRed
    
    DimRed --> ModelInt
    SpecialProc -.-> CorePipeline
    
    Config --> CorePipeline
    Config --> Support
    Config --> ModelInt
    
    CorePipeline -.-> Monitor
    CorePipeline -.-> Logging
    CorePipeline -.-> ErrorHandle
    CorePipeline -.-> Metrics
    
    BatchProc --> Pipeline
    APIServer --> Pipeline

    %% Click events
    click LoadOrch "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/loading.py"
    click Pipeline "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/pipeline.py"
    click Cleaning "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/cleaning.py"
    click Preproc "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/data_preprocessing.py"
    click Transform "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/data_transformation.py"
    click FeatEng "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/feature_engineering.py"
    click DimRed "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/dimensionality_reduction.py"
    click TextAn "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/text_analytics.py"
    click EntityRec "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/entity_recognition.py"
    click TopicMod "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/topic_modeling.py"
    click ImbalData "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/imbalanced_data.py"
    click ModelEval "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/model_evaluation.py"
    click Optim "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/optimization.py"
    click FeatSel "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/feature_selection.py"
    click Monitor "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/monitoring.py"
    click Logging "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/logging_setup.py"
    click ErrorHandle "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/error_handling.py"
    click Metrics "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/metrics.py"
    click Config "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/config.py"
    click BatchProc "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/batch_processing.py"
    click APIServer "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing_pipeline/api_server.py"
    click DataVal "https://github.com/stochastic-sisyphus/adv_data_processing_pipeline/blob/main/adv_data_processing/data_validation.py"
```
