# sunnysavita10/genai_bootcamp architecture

> Architecture diagram of the GitHub repository sunnysavita10/genai_bootcamp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sunnysavita10/genai_bootcamp
- Repository: https://github.com/sunnysavita10/genai_bootcamp

## Overview

##### Main Components
- Data Sources and Inputs:
- The “data” folder contains raw datasets (e.g., training, validation, test data in different languages, PDFs, etc.).
- The “gotbooks” folder includes texts (likely used as a corpus for embedding experiments).
- The “GoogleNews-vectors-negative300.bin.gz” file represents pre-trained word embeddings.
- Preprocessing and Text Handling:
- Notebooks like “text_preprocessing.ipynb”, “text_splitter/text_splitter.ipynb”, and “encoding.ipynb” indicate dedicated steps to preprocess and split text data.
- Embedding and Transformation Layers:
- Notebooks “word_embedding.ipynb”, “sentence_embedding.ipynb”, and “transformer.ipynb” handle the generation of vector representations using different techniques (word embeddings, sentence embeddings, transformers).
- The outputs are further stored and organized in the “results” directory (which contains results for average word embeddings using a specific method).
- Advanced NLP/Integration Modules:
- The “RAG” directory (with an introduction notebook) suggests a module for applying retrieval augmented generation.
- The “langchain” folder contains a tutorial notebook, implying experiments with language chain integration or similar high-level functionalities.
- The “vectordata” folder indicates work with vector databases, integrating vector storage or retrieval mechanisms.
- External Libraries and Dependencies:
- The README and the “requirements.txt” file clarify that the environment is built using conda and pip-installed libraries.
- Pre-trained models (e.g., Google News vectors) and frameworks (likely PyTorch via “pytorch.ipynb”, and transformer-related libraries) are key technologies in play.

##### Relationships and Data Flows
- The preprocessed text then flows into the embedding layers (word, sentence, and transformer-based experiments), which also may leverage external resources (GoogleNews vectors).
- The outputs from these embedding experiments are stored in the “results” directory for further analysis.
- Parallel experiments include modules for RAG and vector database handling – showing integration points where processed outputs might feed into retrieval applications or language chain systems.
- Dependencies like PyTorch and specialized libraries support various processing modules.

##### Architectural Patterns and Design Principles
- A layered pipeline architecture is apparent: Data Ingestion → Preprocessing → Feature Extraction (Embedding/Transformation) → Advanced Processing (RAG/Vector Search).
- Modular design is evident given the separation into distinct notebooks and directories for each stage. This helps maintain separation of concerns and scalability in adding new experimental modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        DS1["\"Raw Data: data\""]:::data
        DS2["\"Raw Data: gotbooks\""]:::data
        DS3["\"Pre-trained Embeddings\""]:::data
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        PP1["\"Preprocessing Notebook: text_preprocessing.ipynb\""]:::preprocess
        PP2["\"Encoding Notebook: encoding.ipynb\""]:::preprocess
        PP3["\"Text Splitter Notebook: text_splitter/text_splitter.ipynb\""]:::preprocess
    end

    %% Embedding & Transformation Layer
    subgraph "Embedding & Transformation Layer"
        EM1["\"Word Embeddings: word_embedding.ipynb\""]:::embedding
        EM2["\"Sentence Embeddings: sentence_embedding.ipynb\""]:::embedding
        EM3["\"Transformer: transformer.ipynb\""]:::embedding
        EM4["\"Results Storage\""]:::embedding
    end

    %% Advanced NLP Modules
    subgraph "Advanced NLP Modules"
        AN1["\"RAG: introduction_of_rag.ipynb\""]:::advanced
        AN2["\"Langchain: langchain_tutorial.ipynb\""]:::advanced
    end

    %% Vector Database Experiments
    subgraph "Vector Database Experiments"
        VD1["\"Vector DB: vector_db.ipynb\""]:::vector
    end

    %% External Dependencies
    subgraph "External Dependencies"
        EXT1["\"PyTorch & Transformer Libraries\""]:::external
    end

    %% Data Flow Connections
    DS1 -->|"feeds"| PP1
    DS2 -->|"feeds"| PP1
    DS3 -->|"supplies"| PP1

    PP1 -->|"processes"| PP2
    PP1 -->|"processes"| PP3

    PP1 -->|"inputs"| EM1
    PP2 -->|"inputs"| EM2
    PP3 -->|"inputs"| EM3

    EM1 -->|"generates"| EM4
    EM2 -->|"generates"| EM4
    EM3 -->|"generates"| EM4

    EM4 -->|"used_by"| AN1
    EM4 -->|"used_by"| AN2
    EM4 -->|"used_by"| VD1

    EXT1 -->|"supports"| EM3
    DS3 -->|"integrates"| EM1

    %% Styles
    classDef data fill:#f4cccc,stroke:#333,stroke-width:2px;
    classDef preprocess fill:#c9daf8,stroke:#333,stroke-width:2px;
    classDef embedding fill:#d9ead3,stroke:#333,stroke-width:2px;
    classDef advanced fill:#fff2cc,stroke:#333,stroke-width:2px;
    classDef vector fill:#ead1dc,stroke:#333,stroke-width:2px;
    classDef external fill:#fce5cd,stroke:#333,stroke-width:2px;

    %% Click Events
    click DS1 "https://github.com/sunnysavita10/genai_bootcamp/tree/main/data"
    click DS2 "https://github.com/sunnysavita10/genai_bootcamp/tree/main/gotbooks"
    click DS3 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/GoogleNews-vectors-negative300.bin.gz"
    click PP1 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/text_preprocessing.ipynb"
    click PP2 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/encoding.ipynb"
    click PP3 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/text_splitter/text_splitter.ipynb"
    click EM1 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/word_embedding.ipynb"
    click EM2 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/sentence_embedding.ipynb"
    click EM3 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/transformer.ipynb"
    click EM4 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/results/..."
    click AN1 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/RAG/introduction_of_rag.ipynb"
    click AN2 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/langchain/langchain_tutorial.ipynb"
    click VD1 "https://github.com/sunnysavita10/genai_bootcamp/blob/main/vectordata/vector_db.ipynb"
```
