# astronomer/gen-ai-fine-tune-rag-use-case architecture

> Architecture diagram of the GitHub repository astronomer/gen-ai-fine-tune-rag-use-case, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/astronomer/gen-ai-fine-tune-rag-use-case
- Repository: https://github.com/astronomer/gen-ai-fine-tune-rag-use-case

## Overview

##### Project type and purpose
- This is a full-stack orchestration pipeline for Retrieval-Augmented Generation (RAG) plus GPT-3.5 fine-tuning using Apache Airflow.
- It provides a blueprint to ingest a knowledge base into a vector store (Weaviate), chunk and retrieve content, call OpenAI’s API for generation and fine-tuning, then surface results in a Streamlit UI.

##### File structure
- Top-level files:
- Dockerfile, docker-compose.override.yml, .devcontainer → container definitions for Airflow, Weaviate, Streamlit.
- .env_example, requirements.txt, packages.txt → environment and dependency specs.
- “dags/” directory → Airflow DAG definitions orchestrating each stage: ingestion, fine-tuning, evaluation, champion/challenger.
- “include/” directory → custom operators, task functions (ETL code, chunking, token counting, plotting), example data and knowledge-base guides.
- “api_scripts/” → ad-hoc Python scripts to kick off or clean up pipeline runs outside DAG context.
- “plugins/” → extra Airflow plugins (e.g., OpenAI operator enhancements).
- “tests/” → unit tests for DAG integrity.

##### README insights
- Orchestrates locally with Astro CLI in Docker.
- Uses Airflow Datasets for inter-DAG dependencies.
- Key external services: OpenAI API, Weaviate vector DB, Streamlit UI.

##### Detail
- Environment/configuration loading (.env) feeding secrets into Airflow & Streamlit.
- Flow from user action in Streamlit back into Airflow for champion vs challenger feedback loop.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Environment
    Env[".env_example"]:::config
    Env --> AirflowScheduler
    Env --> StreamlitApp

    %% Airflow Container
    subgraph "Container: Airflow" 
        direction TB
        AirflowScheduler["Scheduler"]:::compute
        AirflowWorker["Worker"]:::compute
        Webserver["Webserver"]:::compute
        MetadataDB[(Airflow Metadata DB)]:::db
    end

    %% DAG Orchestration
    subgraph "Airflow DAGs"
        direction TB
        subgraph "Ingestion DAGs"
            direction TB
            IngestKB["Ingest KB"]:::compute
            IngestTrain["Ingest Train Examples"]:::compute
            IngestVal["Ingest Validation Examples"]:::compute
        end
        subgraph "Fine-Tuning DAG"
            direction TB
            FineTune["Fine-Tune GPT"]:::compute
        end
        subgraph "Evaluation DAG"
            direction TB
            EvalCost["Eval & Cost Estimate"]:::compute
        end
        subgraph "Champion vs Challenger DAG"
            direction TB
            Champion["Champion vs Challenger"]:::compute
        end
    end

    %% Data Stores
    subgraph "Data Stores"
        direction TB
        Weaviate[(Weaviate Vector Store)]:::db
        LocalStore[(Local File Store)]:::db
    end

    %% External Services
    OpenAICloud["OpenAI API"]:::external

    %% UI
    StreamlitApp["Streamlit App"]:::ui

    %% Custom Operators & Tasks
    subgraph "Custom Operators & Task Functions"
        direction TB
        GPTFineTune["gpt_fine_tune Operator"]:::compute
        Extract["extract"]:::compute
        Split["split"]:::compute
        PrepIngest["prep_for_ingest"]:::compute
        TokenCount["token_counting"]:::compute
        Plotting["plotting"]:::compute
        ValHelper["examples_validation"]:::compute
    end

    %% Plugins & Scripts
    subgraph "Plugins & Scripts"
        direction TB
        OpenAIPlugin["openai_extra_link_plugin"]:::compute
        StartFT["start_fine_tuning_pipeline.py"]:::compute
        Cleanup["delete_all_queued_dataset_events.py"]:::compute
    end

    %% Tests & Specs
    subgraph "Tests & Specs"
        direction TB
        TestDAG1["test_dag_integrity_default.py"]:::compute
        TestDAG2["test_dag_example.py"]:::compute
        Requirements["requirements.txt"]:::config
        Packages["packages.txt"]:::config
    end

    %% Connections
    AirflowScheduler -->|triggers| IngestKB
    AirflowScheduler --> FineTune
    AirflowScheduler --> EvalCost
    AirflowScheduler --> Champion
    AirflowScheduler --> Webserver
    AirflowScheduler --> MetadataDB
    MetadataDB --> AirflowWorker

    AirflowWorker --> Extract
    AirflowWorker --> Split
    AirflowWorker --> PrepIngest
    AirflowWorker --> TokenCount
    AirflowWorker --> Plotting
    AirflowWorker --> ValHelper
    AirflowWorker --> GPTFineTune

    IngestKB -->|chunks & vectors| Weaviate
    IngestTrain -->|chunks & vectors| Weaviate
    IngestVal -->|chunks & vectors| Weaviate

    FineTune -->|pulls data| Weaviate
    FineTune -->|calls| OpenAICloud
    FineTune -->|stores model ID| MetadataDB

    EvalCost -->|calls| OpenAICloud
    EvalCost -->|produces plots| Plotting
    EvalCost -->|stores metrics| MetadataDB

    Champion -->|reads metrics| MetadataDB
    Champion -->|writes decision| MetadataDB
    Champion -->|signals| StreamlitApp

    StreamlitApp -->|requests generation| OpenAICloud
    StreamlitApp -->|fetches model| MetadataDB

    %% Custom Operators & Plugins Integration
    GPTFineTune --> FineTune
    OpenAIPlugin --> AirflowScheduler

    %% Scripts
    StartFT --> FineTune
    Cleanup --> AirflowScheduler

    %% Tests
    TestDAG1 -->|validates| IngestKB
    TestDAG2 -->|validates| FineTune

    %% Click Events
    click IngestKB "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/dags/ingest_knowledge_base.py"
    click IngestTrain "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/dags/ingest_train_examples.py"
    click IngestVal "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/dags/ingest_validation_examples.py"
    click FineTune "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/dags/fine_tune_gpt.py"
    click EvalCost "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/dags/eval_and_cost_estimate.py"
    click Champion "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/dags/champion_challenger.py"
    click GPTFineTune "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/include/custom_operators/gpt_fine_tune.py"
    click Extract "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/include/task_functions/extract.py"
    click Split "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/include/task_functions/split.py"
    click PrepIngest "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/include/task_functions/prep_for_ingest.py"
    click TokenCount "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/include/task_functions/token_counting.py"
    click Plotting "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/include/task_functions/plotting.py"
    click ValHelper "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/include/task_functions/examples_validation.py"
    click OpenAIPlugin "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/plugins/openai_extra_link_plugin.py"
    click StartFT "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/api_scripts/start_fine_tuning_pipeline.py"
    click Cleanup "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/api_scripts/delete_all_queued_dataset_events.py"
    click TestDAG1 "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/.astro/test_dag_integrity_default.py"
    click TestDAG2 "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/tests/dags/test_dag_example.py"
    click Requirements "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/requirements.txt"
    click Packages "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/packages.txt"
    click StreamlitApp "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/include/streamlit/app.py"
    click Weaviate "https://github.com/astronomer/gen-ai-fine-tune-rag-use-case/blob/main/include/weaviate/schema.json"

    %% Styles
    classDef compute fill:#a2d2ff,stroke:#026AA7
    classDef db fill:#ffafcc,stroke:#B0003A
    classDef external fill:#caffbf,stroke:#0B6623
    classDef ui fill:#fdffb6,stroke:#9C661F
    classDef config fill:#e0fbfc,stroke:#05668D
    classDef layer fill:#ffffff,stroke:#000000,stroke-width:2px
```
