How bigdatateam5/building-a-rag-pipeline-with-airflow works
Main Components
• Airflow Pipeline: Represent the Apache Airflow component—the Airflow/ directory includes its Dockerfile, DAGs (like the nvidia_quarterly_reports_pipeline.py), configuration files, logs, and supporting documentation. This service orchestrates the extraction, processing, and scheduling of data pipelines.
• Backend Service: Based on the Backend/ directory, this component (likely built with FastAPI) handles API requests and acts as a bridge between the UI and the underlying data processing logic.
• Frontend Interface: The Frontend/ directory contains a Dockerfile, app.py (a Streamlit or similar interactive application), and additional configuration (secret.toml), which provides the user interface for uploading PDF files and selecting various options such as parser, chunking strategy, and RAG method.
• Chunking Strategies Module: The Chunking_Strats/ directory includes different implementations for content chunking. They are essential for reducing input tokens and influencing how text is segmented for retrieval processes.
• Testing Modules: Files under testing_code/ illustrate various tests for components like Selenium-based web scraping, Airflow pipelines, and encryption tests. These tests help validate the functionality and integration of components.
• External Services & Data Sources: Although not directly represented as code, the README discusses integration with external technologies (such as Pinecone & ChromaDB for vector searches) and the use of NVIDIA’s quarterly PDFs downloaded into a specified folder (e.g., /data/raw_reports).
Relationships and Data Flow
• Data Ingestion: Begin with raw NVIDIA reports (PDFs) stored on disk (/data/raw_reports). Show how Airflow picks these up using its DAGs.
• Pipeline Processing: Airflow manages the workflow and passes processed data to the backend (for further retrieval and transformations), including interactions with chunking strategy modules (for fixed-size, semantic, or sliding window chunking).
• API & RAG Logic: The backend exposes API endpoints (using FastAPI) that incorporate the retrieval-augmented generation process. This part can optionally integrate with external search backends (Pinecone or ChromaDB) as outlined in the README.
• Docker & Deployment: Indicate that each major component (Airflow, Backend, Frontend) has its own Dockerfile and can be orchestrated via the docker-compose.yml file. This separation shows the containerized deployment and inter-service communication.
Architectural Patterns and Technologies
• Microservices / Containerized Deployment: Each major component (Airflow, Backend, Frontend) runs in its own Docker container. Illustrate these as separate nodes with defined service boundaries.
• Modular Pipeline Design: Highlight the use of DAGs in Airflow for scheduling, the decoupling of UI from API logic (frontend vs. backend), and the ability to plug in different parsing, chunking, or RAG approaches.
• Integration Points: Mark the external integrations with vector databases (Pinecone, ChromaDB) and potential external OCR services (Mistral OCR).
Final Checklist for the Principal Software Engineer
• Major component (Airflow orchestration, Backend API/FastAPI, Frontend UI/Streamlit, Chunking Strategies).