# aziddddd/mlops-on-vertex-ai-pipelines architecture

> Architecture diagram of the GitHub repository aziddddd/mlops-on-vertex-ai-pipelines, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aziddddd/mlops-on-vertex-ai-pipelines
- Repository: https://github.com/aziddddd/mlops-on-vertex-ai-pipelines

## Overview

##### Project Type and Purpose
- This repository is a Vertex AI Pipeline Base Template for MLOps. It is structured to allow iterative development, testing, and monitoring of machine learning pipelines.
- The project supports multiple use cases (evident from directories like adult_income, usecase_classification, usecase_forecast, and usecase_recsys) that all follow a similar architectural approach.
- The README reinforces that the focus is on the complete ML lifecycle – from training and prediction to monitoring and maintenance – rather than merely model building.

##### Analyzing the File Structure
- Top-level directories and files:
- CI/CD configuration: .gitlab-ci.yml, ci-cd directory (with YAML files for different use cases) – handles pipeline automation and deployment integrations.
- Documentation: docs directory (with images illustrating pipeline flows) and README.md providing an overview.
- Test artifacts: .coveragerc, test_requirements.txt, ml_components/tests – ensuring quality assurance.
- Each use case (adult_income, usecase_classification, usecase_forecast, usecase_recsys) encapsulates:
- Notebooks (grand_pipeline*.ipynb) for interactive pipeline execution.
- Main scripts (main_train.py, main_pred.py, or main.py) that execute the pipelines.
- An ml_components subdirectory containing distinct Python modules (alert, datacheck, dataimport, datapreproc, modelling, outbound, prediction, pipelinehelper) which clearly separate pipeline functionalities.

##### Key Components and Their Roles
- Data Components:
- Dataimport, datapreproc, and datacheck: Handle data ingestion, cleaning, preprocessing, and quality control.
- Modelling and Prediction:
- Modelling: For building and training models.
- Prediction: For running inference on new data.
- Support and Orchestration:
- Pipelinehelper: Likely orchestrates the full pipeline flow.
- Alert & Outbound: Manage notifications and result dissemination.
- Configuration:
- config.py: Centralizes settings (like RUNNER mode switching between 'dev' and 'prod').
- CI/CD Integration:
- ci-cd directory contents and YAML files for each use case indicate automated deployment/testing integrated with Vertex AI.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Pipeline Configuration & Control
    subgraph "Pipeline Configuration & Control"
        CONFIG["Configuration (config.py)"]:::config
        CICD["CI/CD Scripts (.gitlab-ci.yml, ci-cd/)"]:::config
    end

    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        DATA_IMPORT["Data Import (dataimport.py)"]:::data
        DATA_PREPROC["Data Preprocessing (datapreproc.py)"]:::data
        DATA_CHECK["Data Quality Check (datacheck.py)"]:::data
    end

    %% Model Training Module
    subgraph "Model Training Module"
        TRAIN_MODEL["Model Training (modelling.py)"]:::model
        TRAIN_NOTEBOOK["Training Notebook (grand_pipeline_train.ipynb)"]:::model
        TRAIN_SCRIPT["Training Script (main_train.py)"]:::model
    end

    %% Prediction Module
    subgraph "Prediction Module"
        PRED_INFERENCE["Inference (prediction.py)"]:::pred
        PRED_NOTEBOOK["Prediction Notebook (grand_pipeline_pred.ipynb)"]:::pred
        PRED_SCRIPT["Prediction Script (main_pred.py)"]:::pred
    end

    %% Supporting Modules
    subgraph "Supporting Modules"
        ALERT["Alerting (alert.py)"]:::support
        OUTBOUND["Notifications/Outbound (outbound.py)"]:::support
        PIPELINE_HELPER["Pipeline Helper (pipelinehelper.py)"]:::support
    end

    %% Testing/Quality Assurance
    subgraph "Testing/Quality Assurance"
        QA["Unit Tests (ml_components/tests/)"]:::test
    end

    %% Documentation & External Resources
    subgraph "Documentation & External Resources"
        DOCS["Documentation & Diagrams (docs/)"]:::docs
    end

    %% External Services & Monitoring
    VERTEX["Vertex AI Monitoring"]:::external

    %% Connections - Main Pipeline Flow
    CONFIG -->|"init"| DATA_IMPORT
    DATA_IMPORT -->|"process"| DATA_PREPROC
    DATA_PREPROC -->|"prepare"| TRAIN_MODEL
    TRAIN_MODEL -->|"model"| PRED_INFERENCE

    %% Internal training module links (not mandatory to show alternative execution paths)
    TRAIN_MODEL ---|"orchestrates"| TRAIN_NOTEBOOK
    TRAIN_MODEL ---|"orchestrates"| TRAIN_SCRIPT
    PRED_INFERENCE ---|"orchestrates"| PRED_NOTEBOOK
    PRED_INFERENCE ---|"orchestrates"| PRED_SCRIPT

    %% CI/CD triggers training and prediction
    CICD -->|"triggers"| TRAIN_SCRIPT
    CICD -->|"triggers"| PRED_SCRIPT

    %% Supporting modules interactions
    PIPELINE_HELPER -->|"coordinates"| TRAIN_MODEL
    PIPELINE_HELPER -->|"coordinates"| PRED_INFERENCE
    TRAIN_MODEL -->|"alerts"| ALERT
    PRED_INFERENCE -->|"alerts"| ALERT
    ALERT -->|"notifies"| OUTBOUND

    %% External monitoring integration
    CICD -->|"monitors"| VERTEX
    VERTEX -->|"feedback"| ALERT

    %% Testing and Documentation feedback
    QA -->|"tests"| CONFIG
    QA -->|"tests"| TRAIN_MODEL
    QA -->|"tests"| PRED_INFERENCE
    DOCS ---|"references"| CONFIG
    DOCS ---|"references"| TRAIN_MODEL
    DOCS ---|"references"| PRED_INFERENCE

    %% Class definitions for color coding
    classDef config fill:#ffdddd,stroke:#ff0000,stroke-width:2px;
    classDef data fill:#ddffdd,stroke:#00aa00,stroke-width:2px;
    classDef model fill:#ddddff,stroke:#0000ff,stroke-width:2px;
    classDef pred fill:#ffffdd,stroke:#cccc00,stroke-width:2px;
    classDef support fill:#ddffff,stroke:#00cccc,stroke-width:2px;
    classDef test fill:#ffddff,stroke:#ff00ff,stroke-width:2px;
    classDef docs fill:#e0e0e0,stroke:#999999,stroke-width:2px;
    classDef external fill:#ffe6cc,stroke:#cc6600,stroke-width:2px;

    %% Click Events for Pipeline Configuration & Control
    click CONFIG "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/config.py"
    click CICD "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/ci-cd/adult_income.yaml"

    %% Click Events for Data Ingestion Layer
    click DATA_IMPORT "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/ml_components/dataimport.py"
    click DATA_PREPROC "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/ml_components/datapreproc.py"
    click DATA_CHECK "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/ml_components/datacheck.py"

    %% Click Events for Model Training Module
    click TRAIN_MODEL "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/ml_components/modelling.py"
    click TRAIN_NOTEBOOK "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/grand_pipeline_train.ipynb"
    click TRAIN_SCRIPT "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/main_train.py"

    %% Click Events for Prediction Module
    click PRED_INFERENCE "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/ml_components/prediction.py"
    click PRED_NOTEBOOK "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/grand_pipeline_pred.ipynb"
    click PRED_SCRIPT "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/main_pred.py"

    %% Click Events for Supporting Modules
    click ALERT "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/ml_components/alert.py"
    click OUTBOUND "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/ml_components/outbound.py"
    click PIPELINE_HELPER "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/blob/main/adult_income/ml_components/pipelinehelper.py"

    %% Click Events for Testing/Quality Assurance
    click QA "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/tree/main/adult_income/ml_components/tests/"

    %% Click Events for Documentation & External Resources
    click DOCS "https://github.com/aziddddd/mlops-on-vertex-ai-pipelines/tree/main/docs/"
```
