# parth1609/ramayana architecture

> Architecture diagram of the GitHub repository parth1609/ramayana, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/parth1609/ramayana
- Repository: https://github.com/parth1609/ramayana

## Overview

##### Project type and purpose
- This is an NLP/data‐analysis pipeline, implemented as a series of Jupyter Notebooks and Python scripts, whose goal is to scrape, clean, enrich, and analyze the Ramayana text. The end result is a prediction pipeline that verifies statements (TRUE/FALSE/NONE) against context retrieved from the text using sentence embeddings and an LLM.

##### File structure
- Top level artifacts:
- scrape_data.ipynb – notebook to extract raw shlokas+translations
- modify_data.ipynb, version_1.ipynb, version_2.ipynb – data‐cleaning and pipeline evolution
- cleaned_Ramayana_Dataset.csv, ramayana_shloka_dataset.csv, initial.csv – intermediate & raw CSV data dumps
- Statements_predictions.csv – final output of statement verification
- requirements.txt – dependency list, including Transformers, Langchain, PCA, sentence-transformers, bitsandbytes, etc.
- No frontend/backend split; it’s a batch/interactive pipeline.

##### README for additional insights
- The pipeline uses three broad stages:
a) Data scraping & prep (scrape_data.ipynb)
b) Embedding generation + dimensionality reduction (all-MiniLM-L6-v2 + PCA)
c) Statement verification via LLM (google/flan-t5-large loaded in 8-bit through bitsandbytes, orchestrated by Langchain’s HuggingFacePipeline)
- External dependencies: ValmikiRamayan site, Hugging Face Hub, PyTorch/CUDA for GPU inference, NLTK for tokenization.

##### Main components & their interactions
a) Data Source (External Website)
→ scrape_data.ipynb
→ Raw CSVs (initial.csv, ramayana_shloka_dataset.csv)
→ Data cleaning notebooks (modify_data.ipynb, version_1.ipynb, version_2.ipynb)
→ Cleaned dataset (cleaned_Ramayana_Dataset.csv)
b) Embedding & Indexing
→ Python module/notebook uses sentence-transformers to produce dense vectors from cleaned shlokas
→ PCA via scikit-learn to reduce dimensionality
→ Builds an index for similarity search
c) Statement Verification Pipeline
→ Input: user statements + top-k similar contexts retrieved by embedding index
→ LLM inference: HuggingFace transformers pipeline, wrapped by LangchainPipeline with bitsandbytes for 8-bit quantization
→ Output: Statements_predictions.csv
d) Supporting Components
- requirements.txt for reproducible environment
- NLTK downloads for tokenizers
- GPU/CUDA subsystem for accelerating model inference

##### Architectural patterns & design principles
- Modular ETL pipeline: separate notebooks/modules for scraping, cleaning, embedding, inference
- Data flow as linear DAG: Source → ETL → Feature Extraction → Model Inference → Output
- Reuse of pre-trained models & quantization for efficiency
- Separation of concerns: embedding/indexing logic is distinct from LLM inference logic

- Clear, unique labels for every component (e.g., “Scrape Notebook”, “Cleaned CSV”, “Embedding + PCA Module”, “LLM Inference”)
- A title block that names the overall system (“Ramayana Text Analysis Pipeline”) and notes the technologies stack

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Title
    A[/"Ramayana Text Analysis Pipeline"/]:::title

    %% External Sources
    subgraph "External Dependencies" 
        direction TB
        Valmiki["Valmiki Ramayan Site"]:::external
        HFHub["Hugging Face Hub"]:::external
        GPU["GPU/CUDA Subsystem"]:::external
        NLTK["NLTK Tokenizer Downloads"]:::external
    end

    %% Data Ingestion / Cleaning
    subgraph "Data Ingestion & Cleaning" 
        direction TB
        Scrape["Scrape Notebook<br/>(scrape_data.ipynb)"]:::ingestion
        Raw1["Raw CSV<br/>(initial.csv)"]:::datastore
        Raw2["Structured CSV<br/>(ramayana_shloka_dataset.csv)"]:::datastore
        subgraph "Cleaning Notebooks" 
            direction TB
            Modify["modify_data.ipynb"]:::ingestion
            V1["version_1.ipynb"]:::ingestion
            V2["version_2.ipynb"]:::ingestion
        end
        Cleaned["Cleaned CSV<br/>(cleaned_Ramayana_Dataset.csv)"]:::datastore
    end

    %% Feature Extraction / Indexing
    subgraph "Feature Extraction & Indexing" 
        direction TB
        Embeddings["Embedding Module<br/>(Sentence-Transformers)"]:::extraction
        PCA["Dimensionality Reduction<br/>(scikit-learn PCA)"]:::extraction
        Index["Similarity Index"]:::datastore
    end

    %% Model Inference
    subgraph "Statement Verification Inference" 
        direction TB
        Inference["LLM Inference<br/>(Flan-T5 + bitsandbytes + Langchain)"]:::inference
        Output["Predictions CSV<br/>(Statements_predictions.csv)"]:::datastore
    end

    %% Supporting Files
    subgraph "Project Files" 
        direction TB
        Req["requirements.txt"]:::external
        Readme["README.md"]:::external
        License["LICENSE"]:::external
        GitIgnore[".gitignore"]:::external
    end

    %% Connections
    Valmiki -->|shloka text| Scrape
    Scrape -->|raw shlokas| Raw1
    Scrape -->|structured data| Raw2
    Raw1 --> Modify
    Raw2 --> Modify
    Modify -->|cleaned data| V1
    V1 --> V2
    V2 --> Cleaned
    Cleaned --> Embeddings
    Embeddings --> PCA
    PCA --> Index
    Index -->|top-k contexts| Inference
    Inference -->|statement + context| Output

    HFHub --> Inference
    GPU --> Inference
    NLTK --> Modify

    %% Click Events
    click Scrape "https://github.com/parth1609/ramayana/blob/main/scrape_data.ipynb"
    click Raw1 "https://github.com/parth1609/ramayana/blob/main/initial.csv"
    click Raw2 "https://github.com/parth1609/ramayana/blob/main/ramayana_shloka_dataset.csv"
    click Modify "https://github.com/parth1609/ramayana/blob/main/modify_data.ipynb"
    click V1 "https://github.com/parth1609/ramayana/blob/main/version_1.ipynb"
    click V2 "https://github.com/parth1609/ramayana/blob/main/version_2.ipynb"
    click Cleaned "https://github.com/parth1609/ramayana/blob/main/cleaned_Ramayana_Dataset.csv"
    click Output "https://github.com/parth1609/ramayana/blob/main/Statements_predictions.csv"
    click Req "https://github.com/parth1609/ramayana/blob/main/requirements.txt"
    click Readme "https://github.com/parth1609/ramayana/blob/main/README.md"
    click License "https://github.com/parth1609/ramayana/tree/main/LICENSE"
    click GitIgnore "https://github.com/parth1609/ramayana/blob/main/.gitignore"

    %% Styles
    classDef title fill:#ffffff,stroke:none,font-weight:bold,font-size:16px
    classDef ingestion fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef extraction fill:#d4edda,stroke:#333,stroke-width:1px
    classDef inference fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef external fill:#e2e3e5,stroke:#333,stroke-width:1px
    classDef datastore fill:#f8d7da,stroke:#333,stroke-width:1px
```
