# baranidharan27/fine_tune_pipeline architecture

> Architecture diagram of the GitHub repository baranidharan27/fine_tune_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/baranidharan27/fine_tune_pipeline
- Repository: https://github.com/baranidharan27/fine_tune_pipeline

## Overview

##### Project type and purpose
- This is a Python-based data‐pipeline library/tool. Its purpose is to ingest raw data, apply a series of transformations (tokenization, cleaning, formatting, splitting), and persist the results.
- The key functionality resides in the `data_pipeline` package; `main.py` is simply an entry point.

##### File structure
- Top‐level:
- config.yaml – pipeline configuration
- setup.py, requirements.txt – packaging and dependencies
- `data_pipeline/`:
- loader.py – data ingestion
- preprocessor.py – basic data normalization
- tokenizer_handler.py – tokenization logic
- cleaner.py – data cleaning (e.g., remove noise)
- formatter.py – format or serialize intermediate data
- splitter.py – train/test split or segmentation
- saver.py – data persistence (files, databases)
- pipeline.py – orchestrates the above steps

##### README
- (Empty in this case; we infer architecture purely from file names and typical pipeline patterns.)

##### Omitting the user/CLI
- Instead, start the visualization at the Loader stage, assuming an automated or embedded invocation

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Global entities
    DS["Data Source"]:::source
    Pipeline["Pipeline Orchestrator"]:::orchestrator
    Config["Configuration Manager"]:::config

    %% Pipeline stages
    Loader["Loader"]:::ingestion
    Preprocessor["Preprocessor"]:::transform
    Tokenizer["Tokenizer Handler"]:::transform
    Cleaner["Cleaner"]:::transform
    Formatter["Formatter"]:::transform
    Splitter["Splitter"]:::transform
    Saver["Saver"]:::persistence

    %% Data flow
    DS --> Loader --> Preprocessor --> Tokenizer --> Cleaner --> Formatter --> Splitter --> Saver

    %% Orchestration
    Pipeline --> Loader
    Pipeline --> Preprocessor
    Pipeline --> Tokenizer
    Pipeline --> Cleaner
    Pipeline --> Formatter
    Pipeline --> Splitter
    Pipeline --> Saver

    %% Configuration
    Config -.-> Loader
    Config -.-> Preprocessor
    Config -.-> Tokenizer
    Config -.-> Cleaner
    Config -.-> Formatter
    Config -.-> Splitter
    Config -.-> Saver

    %% Click Events
    click Loader "https://github.com/baranidharan27/fine_tune_pipeline/blob/main/data_pipeline/loader.py"
    click Preprocessor "https://github.com/baranidharan27/fine_tune_pipeline/blob/main/data_pipeline/preprocessor.py"
    click Tokenizer "https://github.com/baranidharan27/fine_tune_pipeline/blob/main/data_pipeline/tokenizer_handler.py"
    click Cleaner "https://github.com/baranidharan27/fine_tune_pipeline/blob/main/data_pipeline/cleaner.py"
    click Formatter "https://github.com/baranidharan27/fine_tune_pipeline/blob/main/data_pipeline/formatter.py"
    click Splitter "https://github.com/baranidharan27/fine_tune_pipeline/blob/main/data_pipeline/splitter.py"
    click Saver "https://github.com/baranidharan27/fine_tune_pipeline/blob/main/data_pipeline/saver.py"
    click Pipeline "https://github.com/baranidharan27/fine_tune_pipeline/blob/main/data_pipeline/pipeline.py"
    click Config "https://github.com/baranidharan27/fine_tune_pipeline/blob/main/config.yaml"

    %% Styles
    classDef source fill:#87CEEB,stroke:#333,stroke-width:1px
    classDef ingestion fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef transform fill:#90EE90,stroke:#333,stroke-width:1px
    classDef persistence fill:#FFD700,stroke:#333,stroke-width:1px
    classDef orchestrator fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef config fill:#F0F0F0,stroke:#333,stroke-width:1px
```
