How astronomer/gen-ai-fine-tune-rag-use-case works
Project type and purpose
• This is a full-stack orchestration pipeline for Retrieval-Augmented Generation (RAG) plus GPT-3.5 fine-tuning using Apache Airflow.
• It provides a blueprint to ingest a knowledge base into a vector store (Weaviate), chunk and retrieve content, call OpenAI’s API for generation and fine-tuning, then surface results in a Streamlit UI.
File structure
• Top-level files:
• Dockerfile, docker-compose.override.yml, .devcontainer → container definitions for Airflow, Weaviate, Streamlit.
• .env_example, requirements.txt, packages.txt → environment and dependency specs.
• “dags/” directory → Airflow DAG definitions orchestrating each stage: ingestion, fine-tuning, evaluation, champion/challenger.
• “include/” directory → custom operators, task functions (ETL code, chunking, token counting, plotting), example data and knowledge-base guides.
• “api_scripts/” → ad-hoc Python scripts to kick off or clean up pipeline runs outside DAG context.
• “plugins/” → extra Airflow plugins (e.g., OpenAI operator enhancements).
• “tests/” → unit tests for DAG integrity.
README insights
• Orchestrates locally with Astro CLI in Docker.
• Uses Airflow Datasets for inter-DAG dependencies.
• Key external services: OpenAI API, Weaviate vector DB, Streamlit UI.
Detail
• Environment/configuration loading (.env) feeding secrets into Airflow & Streamlit.
• Flow from user action in Streamlit back into Airflow for champion vs challenger feedback loop.