How parth1609/ramayana works
Project type and purpose
• This is an NLP/data‐analysis pipeline, implemented as a series of Jupyter Notebooks and Python scripts, whose goal is to scrape, clean, enrich, and analyze the Ramayana text. The end result is a prediction pipeline that verifies statements (TRUE/FALSE/NONE) against context retrieved from the text using sentence embeddings and an LLM.
File structure
• Top level artifacts:
• scrape_data.ipynb – notebook to extract raw shlokas+translations
• modify_data.ipynb, version_1.ipynb, version_2.ipynb – data‐cleaning and pipeline evolution
• cleaned_Ramayana_Dataset.csv, ramayana_shloka_dataset.csv, initial.csv – intermediate & raw CSV data dumps
• Statements_predictions.csv – final output of statement verification
• requirements.txt – dependency list, including Transformers, Langchain, PCA, sentence-transformers, bitsandbytes, etc.
• No frontend/backend split; it’s a batch/interactive pipeline.
README for additional insights
• The pipeline uses three broad stages:
a) Data scraping & prep (scrape_data.ipynb)
b) Embedding generation + dimensionality reduction (all-MiniLM-L6-v2 + PCA)
c) Statement verification via LLM (google/flan-t5-large loaded in 8-bit through bitsandbytes, orchestrated by Langchain’s HuggingFacePipeline)
• External dependencies: ValmikiRamayan site, Hugging Face Hub, PyTorch/CUDA for GPU inference, NLTK for tokenization.
Main components & their interactions
a) Data Source (External Website)
→ scrape_data.ipynb
→ Raw CSVs (initial.csv, ramayana_shloka_dataset.csv)
→ Data cleaning notebooks (modify_data.ipynb, version_1.ipynb, version_2.ipynb)
→ Cleaned dataset (cleaned_Ramayana_Dataset.csv)
b) Embedding & Indexing
→ Python module/notebook uses sentence-transformers to produce dense vectors from cleaned shlokas
→ PCA via scikit-learn to reduce dimensionality
→ Builds an index for similarity search
c) Statement Verification Pipeline
→ Input: user statements + top-k similar contexts retrieved by embedding index
→ LLM inference: HuggingFace transformers pipeline, wrapped by LangchainPipeline with bitsandbytes for 8-bit quantization
→ Output: Statements_predictions.csv
d) Supporting Components
• requirements.txt for reproducible environment
• NLTK downloads for tokenizers
• GPU/CUDA subsystem for accelerating model inference
Architectural patterns & design principles
• Modular ETL pipeline: separate notebooks/modules for scraping, cleaning, embedding, inference
• Data flow as linear DAG: Source → ETL → Feature Extraction → Model Inference → Output
• Reuse of pre-trained models & quantization for efficiency
• Separation of concerns: embedding/indexing logic is distinct from LLM inference logic
• Clear, unique labels for every component (e.g., “Scrape Notebook”, “Cleaned CSV”, “Embedding + PCA Module”, “LLM Inference”)
• A title block that names the overall system (“Ramayana Text Analysis Pipeline”) and notes the technologies stack