How sunnysavita10/genai_bootcamp works
Main Components
• Data Sources and Inputs:
• The “data” folder contains raw datasets (e.g., training, validation, test data in different languages, PDFs, etc.).
• The “gotbooks” folder includes texts (likely used as a corpus for embedding experiments).
• The “GoogleNews-vectors-negative300.bin.gz” file represents pre-trained word embeddings.
• Preprocessing and Text Handling:
• Notebooks like “text_preprocessing.ipynb”, “text_splitter/text_splitter.ipynb”, and “encoding.ipynb” indicate dedicated steps to preprocess and split text data.
• Embedding and Transformation Layers:
• Notebooks “word_embedding.ipynb”, “sentence_embedding.ipynb”, and “transformer.ipynb” handle the generation of vector representations using different techniques (word embeddings, sentence embeddings, transformers).
• The outputs are further stored and organized in the “results” directory (which contains results for average word embeddings using a specific method).
• Advanced NLP/Integration Modules:
• The “RAG” directory (with an introduction notebook) suggests a module for applying retrieval augmented generation.
• The “langchain” folder contains a tutorial notebook, implying experiments with language chain integration or similar high-level functionalities.
• The “vectordata” folder indicates work with vector databases, integrating vector storage or retrieval mechanisms.
• External Libraries and Dependencies:
• The README and the “requirements.txt” file clarify that the environment is built using conda and pip-installed libraries.
• Pre-trained models (e.g., Google News vectors) and frameworks (likely PyTorch via “pytorch.ipynb”, and transformer-related libraries) are key technologies in play.
Relationships and Data Flows
• The preprocessed text then flows into the embedding layers (word, sentence, and transformer-based experiments), which also may leverage external resources (GoogleNews vectors).
• The outputs from these embedding experiments are stored in the “results” directory for further analysis.
• Parallel experiments include modules for RAG and vector database handling – showing integration points where processed outputs might feed into retrieval applications or language chain systems.
• Dependencies like PyTorch and specialized libraries support various processing modules.
Architectural Patterns and Design Principles
• A layered pipeline architecture is apparent: Data Ingestion → Preprocessing → Feature Extraction (Embedding/Transformation) → Advanced Processing (RAG/Vector Search).
• Modular design is evident given the separation into distinct notebooks and directories for each stage. This helps maintain separation of concerns and scalability in adding new experimental modules.