How jatin-mehra119/rti-knowledge-pipeline works
Project type and purpose
• This is a data-processing/ETL pipeline written in Python. It scrapes RTI case-law links and content from the web, extracts text (including from PDFs), cleans and normalizes it, generates embeddings, and produces RAG-optimized vectors for downstream retrieval-augmented generation.
File structure
• Top-level scripts and directories:
• run_all_scrapers.py – the orchestrator that executes sub-tasks in sequence
• Scrapers/ – four independent scraper modules (Link_Scraper, Content_Scraper, pdf_extracter, pdf_Q&Aextracter)
• embedding_gen/ – script to generate embeddings from cleaned text
• Misc/ – miscellaneous cleaning utilities
• requirements.txt – dependency list
• Other directories referenced in README (Extracted_data, Cleaned_data, pdfs, links, logs, output_embeddings, output_embeddings_rag)
Key insights from README
• The workflow is strictly linear:
1. Link scraping → 2. Content scraping → 3. PDF text extraction → 4. PDF Q&A extraction → 5. Embedding generation → 6. RAG optimization
• Intermediary artifacts (CSVs, JSONL, .npy) are stored in feature-named directories
• All operations are logged to logs/
a. Main components
• Orchestrator (run_all_scrapers.py)
• Scraper services (LinkScraper, ContentScraper, PDFTextExtractor, PDFQ&AExtractor)
• Data stores:
– Raw: links/, Extracted_data/, pdfs/
– Cleaned: Cleaned_data/
– Embeddings: output_embeddings/, output_embeddings_rag/
• Embedding Generator (embedding_gen.py)
• Logs (logs/)
b. Relationships and data flows
1. Orchestrator invokes each scraper in order
2. LinkScraper writes URLs → links/
3. ContentScraper reads links/ → writes HTML/text → Extracted_data/
4. PDF extractors read pdfs/ → write raw text/FAQs → Extracted_data/
5. Misc cleaning scripts read Extracted_data/ → write Cleaned_data/
6. EmbeddingGen reads Cleaned_data/ → writes vectors (.npy) → output_embeddings/
7. RAG optimizer reads output_embeddings/ → writes metadata.json + combined embeddings → output_embeddings_rag/
8. At each step logs are appended to logs/
c. Architectural patterns and design principles
• Pipeline (ETL) pattern: modular, sequential tasks
• Single-responsibility scrapers and processing modules
• Separation of raw vs cleaned vs embedded artifacts
• Orchestration via a single entrypoint script
d. Key technologies and libraries
• Python 3.x, requests/BeautifulSoup (presumed) for web scraping
• PyPDF2/pdfminer for PDF text extraction
• numpy for array storage
• Sentence-transformers or OpenAI API for embedding generation
• JSON/CSV for intermediate data
• Standard logging module
Detail but avoid clutter
• Script as its own node