How dnau15/spbrag works
Main Components
a. Data & Storage
• Raw data directory (data/) and Milvus vector DB (configured via CLI args).
• Model artifacts directory (models/).
b. Data Pipeline
• Data loading (data_loading.py)
• Preprocessing (data_preprocessing.py)
• Dataset creation (dataset_creation.py, datasets.py)
c. Training
• BERT fine-tuning module (training/fine_tune_bert.py)
d. Retrieval & Generation
• Retrieval engine (retrieval/rag.py)
• LL M inference via Ollama/Huggingface
e. Evaluation
• Evaluator (evaluation/evaluator.py)
• Metrics calculation (metrics.py)
f. Visualization
• Plotting graphs (visualization/plot_graphs.py)
g. Orchestration & Configuration
• Scripts (run.sh, setup.sh)
• Environment (.env, micromamba, PYTHONPATH)
• Notebooks for prototyping (notebooks/)
Architectural Patterns & Principles
• Modular ML pipeline: clear separation of data processing, model training, retrieval, inference, evaluation.
• Config-driven: CLI args and .env allow swapping LLMs (Ollama vs. Huggingface) or DB URIs.
• Extensible evaluation: evaluation and metrics modules decoupled for new metrics.
• Reproducible environment: micromamba, setup scripts, PYTHONPATH.
Incorporate Key Technologies & Frameworks
• Python modules under src/rag_system/…
• Huggingface Transformers & Sentence-Transformers
• Milvus for vector store
• Ollama for Llama-3.2 inference
• BERT (google-bert/bert-base-uncased)
• Micromamba for env management