# jatin-mehra119/rti-knowledge-pipeline architecture

> Architecture diagram of the GitHub repository jatin-mehra119/rti-knowledge-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jatin-mehra119/rti-knowledge-pipeline
- Repository: https://github.com/jatin-mehra119/rti-knowledge-pipeline

## Overview

##### Project type and purpose
- This is a data-processing/ETL pipeline written in Python. It scrapes RTI case-law links and content from the web, extracts text (including from PDFs), cleans and normalizes it, generates embeddings, and produces RAG-optimized vectors for downstream retrieval-augmented generation.

##### File structure
- Top-level scripts and directories:
- run_all_scrapers.py – the orchestrator that executes sub-tasks in sequence
- Scrapers/ – four independent scraper modules (Link_Scraper, Content_Scraper, pdf_extracter, pdf_Q&Aextracter)
- embedding_gen/ – script to generate embeddings from cleaned text
- Misc/ – miscellaneous cleaning utilities
- requirements.txt – dependency list
- Other directories referenced in README (Extracted_data, Cleaned_data, pdfs, links, logs, output_embeddings, output_embeddings_rag)

##### Key insights from README
- The workflow is strictly linear:
1. Link scraping → 2. Content scraping → 3. PDF text extraction → 4. PDF Q&A extraction → 5. Embedding generation → 6. RAG optimization
- Intermediary artifacts (CSVs, JSONL, .npy) are stored in feature-named directories
- All operations are logged to logs/

a. Main components
- Orchestrator (run_all_scrapers.py)
- Scraper services (LinkScraper, ContentScraper, PDFTextExtractor, PDFQ&AExtractor)
- Data stores:
– Raw: links/, Extracted_data/, pdfs/
– Cleaned: Cleaned_data/
– Embeddings: output_embeddings/, output_embeddings_rag/
- Embedding Generator (embedding_gen.py)
- Logs (logs/)
b. Relationships and data flows
1. Orchestrator invokes each scraper in order
2. LinkScraper writes URLs → links/
3. ContentScraper reads links/ → writes HTML/text → Extracted_data/
4. PDF extractors read pdfs/ → write raw text/FAQs → Extracted_data/
5. Misc cleaning scripts read Extracted_data/ → write Cleaned_data/
6. EmbeddingGen reads Cleaned_data/ → writes vectors (.npy) → output_embeddings/
7. RAG optimizer reads output_embeddings/ → writes metadata.json + combined embeddings → output_embeddings_rag/
8. At each step logs are appended to logs/
c. Architectural patterns and design principles
- Pipeline (ETL) pattern: modular, sequential tasks
- Single-responsibility scrapers and processing modules
- Separation of raw vs cleaned vs embedded artifacts
- Orchestration via a single entrypoint script
d. Key technologies and libraries
- Python 3.x, requests/BeautifulSoup (presumed) for web scraping
- PyPDF2/pdfminer for PDF text extraction
- numpy for array storage
- Sentence-transformers or OpenAI API for embedding generation
- JSON/CSV for intermediate data
- Standard logging module

##### Detail but avoid clutter
- Script as its own node

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Ingestion"
        Orchestrator["Orchestrator\n(run_all_scrapers.py)"]:::code
        LinkScraper["LinkScraper\n(Scrapers/Link_Scraper.py)"]:::code
        linksDir(("links/\n(CSV)")):::data
        ContentScraper["ContentScraper\n(Scrapers/Content_Scraper.py)"]:::code
        pdfsDir(("pdfs/")):::data
        PDFTextExtractor["PDF Text Extractor\n(Scrapers/pdf_extracter.py)"]:::code
        PDFQAExtractor["PDF Q&A Extractor\n(Scrapers/pdf_Q&Aextracter.py)"]:::code
        ExtractedData(("Extracted_data/\n(JSONL/TXT)")):::data
    end

    subgraph "Processing/Cleaning"
        MiscClean["Cleaning Utilities\n(Misc/clean_cases_data.py)"]:::code
        CleanedData(("Cleaned_data/\n(CSV)")):::data
    end

    subgraph "Embedding Generation"
        EmbeddingGen["Embedding Generator\n(embedding_gen/embedding_gen.py)"]:::code
        OutputEmbeddings(("output_embeddings/\n(NPY)")):::data
        RAGOptimizer["RAG Optimizer"]:::code
        OutputRag(("output_embeddings_rag/\n(NPY + JSON)")):::data
    end

    Logs(("logs/")):::logs

    Orchestrator -->|"start pipeline"| LinkScraper
    LinkScraper -->|"CSV"| linksDir
    linksDir -->|"read"| ContentScraper
    ContentScraper -->|"JSONL/TXT"| ExtractedData
    pdfsDir -->|"PDF"| PDFTextExtractor
    PDFTextExtractor -->|"text"| ExtractedData
    pdfsDir -->|"PDF"| PDFQAExtractor
    PDFQAExtractor -->|"Q&A JSON"| ExtractedData

    ExtractedData -->|"raw data"| MiscClean
    MiscClean -->|"CSV"| CleanedData

    CleanedData -->|"read"| EmbeddingGen
    EmbeddingGen -->|"NPY"| OutputEmbeddings

    OutputEmbeddings -->|"read"| RAGOptimizer
    RAGOptimizer -->|"NPY+JSON"| OutputRag

    Orchestrator -->|logs| Logs
    LinkScraper -->|logs| Logs
    ContentScraper -->|logs| Logs
    PDFTextExtractor -->|logs| Logs
    PDFQAExtractor -->|logs| Logs
    MiscClean -->|logs| Logs
    EmbeddingGen -->|logs| Logs
    RAGOptimizer -->|logs| Logs

    click Orchestrator "https://github.com/jatin-mehra119/rti-knowledge-pipeline/blob/main/run_all_scrapers.py"
    click LinkScraper "https://github.com/jatin-mehra119/rti-knowledge-pipeline/blob/main/Scrapers/Link_Scraper.py"
    click ContentScraper "https://github.com/jatin-mehra119/rti-knowledge-pipeline/blob/main/Scrapers/Content_Scraper.py"
    click PDFTextExtractor "https://github.com/jatin-mehra119/rti-knowledge-pipeline/blob/main/Scrapers/pdf_extracter.py"
    click PDFQAExtractor "https://github.com/jatin-mehra119/rti-knowledge-pipeline/blob/main/Scrapers/pdf_Q&Aextracter.py"
    click MiscClean "https://github.com/jatin-mehra119/rti-knowledge-pipeline/blob/main/Misc/clean_cases_data.py"
    click EmbeddingGen "https://github.com/jatin-mehra119/rti-knowledge-pipeline/blob/main/embedding_gen/embedding_gen.py"

    classDef code fill:#ADD8E6,stroke:#000;
    classDef data fill:#9f6,stroke:#000;
    classDef logs fill:#f96,stroke:#000;
```
