# fayouzz/multi-document-summarization-with-centroid-based-pretraining architecture

> Architecture diagram of the GitHub repository fayouzz/multi-document-summarization-with-centroid-based-pretraining, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fayouzz/multi-document-summarization-with-centroid-based-pretraining
- Repository: https://github.com/fayouzz/multi-document-summarization-with-centroid-based-pretraining

## Overview

##### Main Components

A. Data Flow Layer:
- Raw Data Storage (/data/raw)
- Processed Data Storage (/data/processed)
- Centroids Storage (/data/centroids)

B. Processing Pipeline:
- Preprocessing Module
- Centroid Generation Module
- Pretraining Module
- Fine-tuning Module
- Evaluation Module

C. Model Management:
- Pretrained Models Storage
- Fine-tuned Models Storage

D. Experiment & Results Management:
- Experiment Configuration
- Results Storage
- Logging System

**Level 1: Data Input Layer**
**Level 2: Processing Pipeline**
**Level 3: Model Layer**
**Level 4: Evaluation & Results Layer**

##### Component Relationships

- Data flow from raw → processed → centroids
- Processing pipeline: preprocessing → centroid generation → pretraining → fine-tuning → evaluation
- Model evolution: pretrained models → fine-tuned models
- Results collection: evaluation → results storage

##### Technical Details to Highlight

- The centroid-based pretraining approach as the core innovation
- The modular nature of the processing pipeline
- The experiment tracking and evaluation framework

##### Implementation Notes

- src/ contains the core processing modules
- models/ stores the model artifacts
- results/ captures all outputs
- experiments/ manages configurations and logs

##### Key Relationships to Emphasize

Clear connections between:
- Preprocessing → Centroid Generation
- Centroid Generation → Pretraining
- Pretraining → Fine-tuning
- Fine-tuning → Evaluation

##### Additional Elements

Include:
- Version control integration points
- Data validation checkpoints
- Model checkpoint storage

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Input Layer
    subgraph DataLayer[Data Layer]
        RawData[(Raw Data)]
        ProcessedData[(Processed Data)]
        Centroids[(Centroids)]
    end

    %% Processing Pipeline Layer
    subgraph ProcessingLayer[Processing Pipeline]
        Preprocess[Preprocessing]
        CentroidGen[Centroid Generation]
        Pretrain[Pretraining]
        FineTune[Fine-tuning]
        Evaluate[Evaluation]
    end

    %% Model Layer
    subgraph ModelLayer[Model Layer]
        PretrainedModels[(Pretrained Models)]
        FineTunedModels[(Fine-tuned Models)]
    end

    %% Results Layer
    subgraph ResultsLayer[Results & Documentation]
        Config[Config Management]
        Logging[Logging System]
        Results[(Results Storage)]
        Plots[Results Visualization]
        Docs[Documentation]
    end

    %% Data Flow Connections
    RawData --> Preprocess
    Preprocess --> ProcessedData
    ProcessedData --> CentroidGen
    CentroidGen --> Centroids
    Centroids --> Pretrain
    
    %% Processing Pipeline Flow
    Pretrain --> PretrainedModels
    PretrainedModels --> FineTune
    FineTune --> FineTunedModels
    FineTunedModels --> Evaluate
    
    %% Results Flow
    Evaluate --> Results
    Results --> Plots
    Config --> ProcessingLayer
    ProcessingLayer --> Logging
    
    %% Styling
    classDef storage fill:#3498db,stroke:#2980b9,color:white
    classDef processing fill:#2ecc71,stroke:#27ae60,color:white
    classDef model fill:#e67e22,stroke:#d35400,color:white
    classDef results fill:#9b59b6,stroke:#8e44ad,color:white
    
    class RawData,ProcessedData,Centroids,Results storage
    class Preprocess,CentroidGen,Pretrain,FineTune,Evaluate processing
    class PretrainedModels,FineTunedModels model
    class Config,Logging,Plots,Docs results

    %% Click Events
    click RawData "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/tree/main/data/raw"
    click ProcessedData "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/tree/main/data/processed"
    click Centroids "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/tree/main/data/centroids"
    click Preprocess "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/blob/main/src/preprocessing.py"
    click CentroidGen "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/blob/main/src/centroid_generation.py"
    click Pretrain "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/blob/main/src/pretraining.py"
    click FineTune "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/blob/main/src/fine_tuning.py"
    click Evaluate "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/blob/main/src/evaluation.py"
    click PretrainedModels "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/tree/main/models/pretrained"
    click FineTunedModels "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/tree/main/models/fine_tuned"
    click Config "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/blob/main/experiments/config.json"
    click Results "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/tree/main/results"
    click Logging "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/blob/main/experiments/logs.txt"
    click Plots "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/tree/main/results/plots"
    click Docs "https://github.com/Fayouzz/Multi-Document-Summarization-with-Centroid-Based-Pretraining/tree/main/docs"
```
