# robel-dev/ecowissen architecture

> Architecture diagram of the GitHub repository robel-dev/ecowissen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/robel-dev/ecowissen
- Repository: https://github.com/robel-dev/ecowissen

## Overview

##### Project type and purpose
- This is a full-stack Retrieval-Augmented Generation (RAG) system named “EcoWissen Freiburg” that answers sustainability and environmental questions about Freiburg by combining vector-based document retrieval with a large language model.
- It consists of three logical subsystems:
a. A Flask-based backend (RAG-backend/) exposing REST endpoints for document ingestion and query handling.
b. A Streamlit-based frontend (RAG-frontend/) providing a bilingual chat UI with carbon-footprint tracking and export functionality.
c. A fine-tuning pipeline (finetune/) that prepares data and trains a custom model on sustainability Q&A pairs.

##### File structure
- Top-level:
- RAG-backend/
– chatbot.py: Flask server main
– utils.py: text cleaning, formatting
– database/: Chroma vector-store files
– dataset/: raw PDFs & JSONs used to populate Chroma
– energy-measurement.py: carbon-footprint tracking
- RAG-frontend/
– streamlit-chat.py: UI entry point
– model.py: wrappers around the LLM API
– translations.py: bilingual label dictionary
- finetune/
– prepare_training_data.py & notebooks: JSON Q&A prep
– configs/: YAML for fine-tuning
– models/: output folder for saved fine-tuned LLM
- requirements.txt
- Patterns & configs:
- Separation of concerns (ingest vs query vs UI)
- Chroma vector database as retrieval layer
- LangChain or similar for document embedding & querying
- PyTorch/Transformers for model fine-tuning

##### README for additional insights
- Confirms the three main modules: Backend, Frontend, Fine-tuning Pipeline
- Technical stack: Flask, Streamlit, ChromaDB, LangChain, Meta-Llama-3, OpenLlama 3B
- Special features: bilingual UI, carbon footprint visualization, PDF export

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User and Frontend
    User["User (Web Browser)"]:::external
    subgraph "Frontend Layer"
        direction TB
        StreamlitUI["Streamlit Chat UI"]:::frontend
        ModelWrapper["LLM API Wrapper"]:::frontend
        Translations["Bilingual Translations"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Layer"
        direction TB
        FlaskAPI["Flask API (chatbot.py)"]:::backend
        Utils["Text Cleaning & Formatting (utils.py)"]:::backend
        Carbon["Carbon-Footprint Tracker (energy-measurement.py)"]:::backend
        subgraph "Vector Store & Datasets"
            direction TB
            ChromaDB["Chroma Vector Store (db)"]:::db
            PDF["Raw PDF Dataset"]:::storage
            JSONData["JSON Indicators Dataset"]:::storage
        end
    end

    %% External LLM Inference
    LLM["Meta-Llama-3-8B-Instruct Inference"]:::external

    %% Fine-Tuning Pipeline (Offline)
    subgraph "Fine-Tuning Pipeline (Offline)"
        direction TB
        PrepData["Data Preparation Script"]:::compute
        InputJSON["Fine-Tuning JSON Helper"]:::compute
        Interim["Interim Training Data"]:::storage
        Config["Fine-Tuning Config (config.yml)"]:::compute
        ModelsOut["Fine-Tuned Model Artifacts"]:::models
    end

    %% Runtime Flow
    User -->|"opens UI"| StreamlitUI
    StreamlitUI -->|"HTTP POST /query"| FlaskAPI
    FlaskAPI -->|"clean & format"| Utils
    FlaskAPI -->|"vector search"| ChromaDB
    FlaskAPI -->|"LLM prompt"| LLM
    LLM -->|"generated answer"| FlaskAPI
    FlaskAPI -->|"measure energy"| Carbon
    FlaskAPI -->|"response + footprint"| StreamlitUI

    %% Ingestion Flow
    PDF -->|"ingest via /ingest"| FlaskAPI
    JSONData -->|"ingest via /ingest"| FlaskAPI

    %% Fine-Tuning Flow
    PrepData -->|"uses"| InputJSON
    InputJSON -->|"outputs"| Interim
    Interim -->|"with"| Config
    Config -->|"trains model"| ModelsOut

    %% Click Events
    click StreamlitUI "https://github.com/robel-dev/ecowissen/blob/main/RAG-frontend/streamlit-chat.py"
    click ModelWrapper "https://github.com/robel-dev/ecowissen/blob/main/RAG-frontend/model.py"
    click Translations "https://github.com/robel-dev/ecowissen/blob/main/RAG-frontend/translations.py"
    click FlaskAPI "https://github.com/robel-dev/ecowissen/blob/main/RAG-backend/chatbot.py"
    click Utils "https://github.com/robel-dev/ecowissen/blob/main/RAG-backend/utils.py"
    click Carbon "https://github.com/robel-dev/ecowissen/blob/main/RAG-backend/energy-measurement.py"
    click PDF "https://github.com/robel-dev/ecowissen/blob/main/RAG-backend/dataset/pdf/frei-sus-2016.pdf"
    click JSONData "https://github.com/robel-dev/ecowissen/blob/main/RAG-backend/dataset/sus_goal_indicator_data/SGD_indicator.json"
    click ChromaDB "https://github.com/robel-dev/ecowissen/tree/main/RAG-backend/database/db"
    click PrepData "https://github.com/robel-dev/ecowissen/blob/main/finetune/prepare_training_data.py"
    click Config "https://github.com/robel-dev/ecowissen/blob/main/finetune/configs/config.yml"
    click InputJSON "https://github.com/robel-dev/ecowissen/blob/main/finetune/input_json.py"
    click Interim "https://github.com/robel-dev/ecowissen/blob/main/finetune/data/interim/output.json"
    click ModelsOut "https://github.com/robel-dev/ecowissen/tree/main/finetune/models/fine_tuned_models"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#2c7cd1
    classDef backend fill:#d1e7dd,stroke:#198754
    classDef db fill:#d4edda,stroke:#28a745
    classDef compute fill:#cfe2ff,stroke:#0d6efd
    classDef models fill:#fff3cd,stroke:#ffc107
    classDef storage fill:#e2e3e5,stroke:#6c757d
    classDef external fill:#e0cffc,stroke:#6f42c1
```
