How morrissimons/free_building_vectorstore_from_scratch works
Project type and purpose
• This is an experimental NLP/data-science pipeline implemented via Jupyter notebooks. Its goal is to build a vector store and search engine from scratch by training Word2Vec and Doc2Vec embeddings on raw text data, storing those vectors, and implementing query/similarity functions.
File structure
• Top-level:
• README.md – project overview and methodology.
• .gitignore – standard ignore rules.
• assets/ – images used in documentation.
• data/ – raw text data (Royal_data.txt).
• exp1_word2vec.ipynb through exp5.ipynb – sequential notebooks for:
– Training custom Word2Vec (and bigram counts)
– Building Doc2Vec via averaged Word2Vec embeddings
– Implementing a custom vector store
– Query/similarity functions
– Final experiments and evaluation
README for additional insights
• The README outlines the plan: build vocab, train embeddings, build a vector store, implement query/similarity, then evaluate.
• Emphasis on the differences between skip-gram and bigram statistical counts.
• No external services or deployed API layers—this is a self-contained pipeline.
a. Main components:
1. Data Ingestion & Preprocessing
– Raw text (data/Royal_data.txt)
– Tokenization, cleaning, chunking
2. Embedding Training
– Custom bigram counts (statistical)
– Word2Vec (skip-gram neural model)
– Doc2Vec (averaging word vectors)
3. Vector Store
– Data structure/index for storing document vectors
– Persistence layer (e.g., in-memory, on-disk file)
4. Query/Similarity Engine
– Takes user query → preprocess → embed → nearest-neighbor search
5. Evaluation Module
– Metrics to compare custom embeddings vs. baselines
– Plots/reports (notebooks)
b. Relationships & data flow:
• Raw Data → Preprocessing → Embedding Training
• Embedding Training → Vector Store
• User Query → Preprocessing → Embedding Model → Vector Store → Results
• Evaluation Module reads from Vector Store & embeddings to compute metrics
c. Architectural patterns & design principles:
– Linear data pipeline with clear separation of stages (ETL → train → index → query → evaluate).
– Reproducibility via notebooks.
– Modular design: each notebook corresponds to one component.
– Custom vs. baseline models shown side-by-side for comparison.
d. Technologies, frameworks, libraries:
– Python & Jupyter Notebooks
– Gensim or custom numpy code for Word2Vec
– Standard file I/O for storing index (e.g., pickle/JSON)
– Matplotlib/Seaborn for evaluation plots
Final advice
• Essential components and their interactions.
• Don’t over-modularize beyond the five core stages—focus on clarity.
• Custom code segments (e.g., bigram vs. skip-gram) to highlight experimental branches.