# bigdatateam5/building-a-rag-pipeline-with-airflow architecture

> Architecture diagram of the GitHub repository bigdatateam5/building-a-rag-pipeline-with-airflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bigdatateam5/building-a-rag-pipeline-with-airflow
- Repository: https://github.com/bigdatateam5/building-a-rag-pipeline-with-airflow

## Overview

##### Main Components
- Airflow Pipeline: Represent the Apache Airflow component—the Airflow/ directory includes its Dockerfile, DAGs (like the nvidia_quarterly_reports_pipeline.py), configuration files, logs, and supporting documentation. This service orchestrates the extraction, processing, and scheduling of data pipelines.
- Backend Service: Based on the Backend/ directory, this component (likely built with FastAPI) handles API requests and acts as a bridge between the UI and the underlying data processing logic.
- Frontend Interface: The Frontend/ directory contains a Dockerfile, app.py (a Streamlit or similar interactive application), and additional configuration (secret.toml), which provides the user interface for uploading PDF files and selecting various options such as parser, chunking strategy, and RAG method.
- Chunking Strategies Module: The Chunking_Strats/ directory includes different implementations for content chunking. They are essential for reducing input tokens and influencing how text is segmented for retrieval processes.
- Testing Modules: Files under testing_code/ illustrate various tests for components like Selenium-based web scraping, Airflow pipelines, and encryption tests. These tests help validate the functionality and integration of components.
- External Services & Data Sources: Although not directly represented as code, the README discusses integration with external technologies (such as Pinecone & ChromaDB for vector searches) and the use of NVIDIA’s quarterly PDFs downloaded into a specified folder (e.g., /data/raw_reports).

##### Relationships and Data Flow
- Data Ingestion: Begin with raw NVIDIA reports (PDFs) stored on disk (/data/raw_reports). Show how Airflow picks these up using its DAGs.
- Pipeline Processing: Airflow manages the workflow and passes processed data to the backend (for further retrieval and transformations), including interactions with chunking strategy modules (for fixed-size, semantic, or sliding window chunking).
- API & RAG Logic: The backend exposes API endpoints (using FastAPI) that incorporate the retrieval-augmented generation process. This part can optionally integrate with external search backends (Pinecone or ChromaDB) as outlined in the README.
- Docker & Deployment: Indicate that each major component (Airflow, Backend, Frontend) has its own Dockerfile and can be orchestrated via the docker-compose.yml file. This separation shows the containerized deployment and inter-service communication.

##### Architectural Patterns and Technologies
- Microservices / Containerized Deployment: Each major component (Airflow, Backend, Frontend) runs in its own Docker container. Illustrate these as separate nodes with defined service boundaries.
- Modular Pipeline Design: Highlight the use of DAGs in Airflow for scheduling, the decoupling of UI from API logic (frontend vs. backend), and the ability to plug in different parsing, chunking, or RAG approaches.
- Integration Points: Mark the external integrations with vector databases (Pinecone, ChromaDB) and potential external OCR services (Mistral OCR).

##### Final Checklist for the Principal Software Engineer
- Major component (Airflow orchestration, Backend API/FastAPI, Frontend UI/Streamlit, Chunking Strategies).

## Diagram (Mermaid)

```mermaid
flowchart TD
    DataSource["Raw PDF Reports (/data/raw_reports)"]:::data
    AirflowPipeline["Airflow Pipeline (Apache Airflow)"]:::orchestration
    ChunkingStrategies["Chunking Strategies Module"]:::module
    BackendService["Backend Service (FastAPI)"]:::backend
    FrontendUI["Frontend Interface (Streamlit UI)"]:::frontend
    ExternalServices["External Services (Pinecone,ChromaDB,OCR)"]:::external
    DockerCompose["Docker Compose & Deployment"]:::deployment
    TestingModules["Testing Modules"]:::testing

    %% Data flow connections
    DataSource -->|"ingests"| AirflowPipeline
    AirflowPipeline -->|"calls"| ChunkingStrategies
    AirflowPipeline -->|"sendsProcessedData"| BackendService
    BackendService -->|"queries"| ExternalServices
    FrontendUI -->|"uploadsParameters"| BackendService
    BackendService -->|"returnsResults"| FrontendUI

    %% Deployment connections
    DockerCompose --> AirflowPipeline
    DockerCompose --> BackendService
    DockerCompose --> FrontendUI

    %% Testing interconnections
    TestingModules --- AirflowPipeline
    TestingModules --- BackendService
    TestingModules --- FrontendUI

    %% Click Events
    click AirflowPipeline "https://github.com/bigdatateam5/building-a-rag-pipeline-with-airflow/tree/main/Airflow/"
    click BackendService "https://github.com/bigdatateam5/building-a-rag-pipeline-with-airflow/tree/main/Backend/"
    click FrontendUI "https://github.com/bigdatateam5/building-a-rag-pipeline-with-airflow/tree/main/Frontend/"
    click ChunkingStrategies "https://github.com/bigdatateam5/building-a-rag-pipeline-with-airflow/tree/main/Chunking_Strats/"
    click TestingModules "https://github.com/bigdatateam5/building-a-rag-pipeline-with-airflow/tree/main/testing_code/"

    %% Style definitions
    classDef data fill:#f9e79f,stroke:#d4ac0d,stroke-width:2px;
    classDef orchestration fill:#aed6f1,stroke:#2471a3,stroke-width:2px;
    classDef backend fill:#a9dfbf,stroke:#1e8449,stroke-width:2px;
    classDef frontend fill:#f5b7b1,stroke:#c0392b,stroke-width:2px;
    classDef module fill:#d7bde2,stroke:#8e44ad,stroke-width:2px;
    classDef external fill:#fad7a0,stroke:#d35400,stroke-width:2px;
    classDef deployment fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef testing fill:#fcf3cf,stroke:#f1c40f,stroke-width:2px;
```
