# morrissimons/free_building_vectorstore_from_scratch architecture

> Architecture diagram of the GitHub repository morrissimons/free_building_vectorstore_from_scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/morrissimons/free_building_vectorstore_from_scratch
- Repository: https://github.com/morrissimons/free_building_vectorstore_from_scratch

## Overview

##### Project type and purpose
- This is an experimental NLP/data-science pipeline implemented via Jupyter notebooks. Its goal is to build a vector store and search engine from scratch by training Word2Vec and Doc2Vec embeddings on raw text data, storing those vectors, and implementing query/similarity functions.

##### File structure
- Top-level:
- README.md – project overview and methodology.
- .gitignore – standard ignore rules.
- assets/ – images used in documentation.
- data/ – raw text data (Royal_data.txt).
- exp1_word2vec.ipynb through exp5.ipynb – sequential notebooks for:
– Training custom Word2Vec (and bigram counts)
– Building Doc2Vec via averaged Word2Vec embeddings
– Implementing a custom vector store
– Query/similarity functions
– Final experiments and evaluation

##### README for additional insights
- The README outlines the plan: build vocab, train embeddings, build a vector store, implement query/similarity, then evaluate.
- Emphasis on the differences between skip-gram and bigram statistical counts.
- No external services or deployed API layers—this is a self-contained pipeline.

a. Main components:
1. Data Ingestion & Preprocessing
– Raw text (data/Royal_data.txt)
– Tokenization, cleaning, chunking
2. Embedding Training
– Custom bigram counts (statistical)
– Word2Vec (skip-gram neural model)
– Doc2Vec (averaging word vectors)
3. Vector Store
– Data structure/index for storing document vectors
– Persistence layer (e.g., in-memory, on-disk file)
4. Query/Similarity Engine
– Takes user query → preprocess → embed → nearest-neighbor search
5. Evaluation Module
– Metrics to compare custom embeddings vs. baselines
– Plots/reports (notebooks)

b. Relationships & data flow:
- Raw Data → Preprocessing → Embedding Training
- Embedding Training → Vector Store
- User Query → Preprocessing → Embedding Model → Vector Store → Results
- Evaluation Module reads from Vector Store & embeddings to compute metrics

c. Architectural patterns & design principles:
– Linear data pipeline with clear separation of stages (ETL → train → index → query → evaluate).
– Reproducibility via notebooks.
– Modular design: each notebook corresponds to one component.
– Custom vs. baseline models shown side-by-side for comparison.

d. Technologies, frameworks, libraries:
– Python & Jupyter Notebooks
– Gensim or custom numpy code for Word2Vec
– Standard file I/O for storing index (e.g., pickle/JSON)
– Matplotlib/Seaborn for evaluation plots

##### Final advice
- Essential components and their interactions.
- Don’t over-modularize beyond the five core stages—focus on clarity.
- Custom code segments (e.g., bigram vs. skip-gram) to highlight experimental branches.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Stage
    subgraph "Data Stage"
        RawData["Raw Text File\ndata/Royal_data.txt"]:::dataClass
    end

    %% Preprocessing
    Preprocess["Data Ingestion & Preprocessing\ntokenization, cleaning"]:::dataClass

    %% Embedding Training
    subgraph "Embedding Training"
        W2V["Word2Vec & Bigram Counts"]:::modelClass
        Doc2Vec["Doc2Vec Builder"]:::modelClass
    end

    %% Vector Store
    subgraph "Vector Store"
        VS1["Initial Custom Index\nexp3.ipynb"]:::storageClass
        VS2["Final Custom Vector Store\nexp4_final_custom_vector_store.ipynb"]:::storageClass
    end

    %% Query & Evaluation
    QueryEngine["Query/Similarity Engine\nexp3.ipynb"]:::queryClass
    Evaluation["Evaluation Module\nexp5.ipynb"]:::evalClass

    %% Documentation
    subgraph "Documentation"
        README["README.md"]:::dataClass
        Ignore[".gitignore"]:::dataClass
        Assets["assets/"]:::dataClass
    end

    %% Flow
    RawData -->|"raw text"| Preprocess
    Preprocess -->|"cleaned tokens"| W2V
    W2V -->|"word embeddings"| Doc2Vec
    W2V -->|"tuning"| W2V
    Doc2Vec -->|"document embeddings"| VS1
    VS1 -->|"refined index"| VS2
    VS2 -->|"indexed vectors"| QueryEngine
    QueryEngine -->|"search"| VS2
    W2V -->|"embeddings"| Evaluation
    Doc2Vec -->|"embeddings"| Evaluation
    VS2 -->|"vectors"| Evaluation

    %% Click Events
    click RawData "https://github.com/morrissimons/free_building_vectorstore_from_scratch/blob/main/data/Royal_data.txt"
    click W2V "https://github.com/morrissimons/free_building_vectorstore_from_scratch/blob/main/exp1_word2vec.ipynb"
    click Doc2Vec "https://github.com/morrissimons/free_building_vectorstore_from_scratch/blob/main/exp2_doc2vec.ipynb"
    click VS1 "https://github.com/morrissimons/free_building_vectorstore_from_scratch/blob/main/exp3.ipynb"
    click VS2 "https://github.com/morrissimons/free_building_vectorstore_from_scratch/blob/main/exp4_final_custom_vector_store.ipynb"
    click QueryEngine "https://github.com/morrissimons/free_building_vectorstore_from_scratch/blob/main/exp3.ipynb"
    click Evaluation "https://github.com/morrissimons/free_building_vectorstore_from_scratch/blob/main/exp5.ipynb"
    click README "https://github.com/morrissimons/free_building_vectorstore_from_scratch/blob/main/README.md"
    click Ignore "https://github.com/morrissimons/free_building_vectorstore_from_scratch/blob/main/.gitignore"
    click Assets "https://github.com/morrissimons/free_building_vectorstore_from_scratch/tree/main/assets/"

    %% Styles
    classDef dataClass fill:#ADD8E6,stroke:#000;
    classDef modelClass fill:#90EE90,stroke:#000;
    classDef storageClass fill:#FFA500,stroke:#000;
    classDef queryClass fill:#FFA500,stroke:#000;
    classDef evalClass fill:#DDA0DD,stroke:#000;
```
