# whybe-choi/dacon-hansoldeco-season3 architecture

> Architecture diagram of the GitHub repository whybe-choi/dacon-hansoldeco-season3, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/whybe-choi/dacon-hansoldeco-season3
- Repository: https://github.com/whybe-choi/dacon-hansoldeco-season3

## Overview

The dacon-hansoldeco-season3 project is an NLP pipeline developed for a construction accident prevention and response generation challenge. Its core functionality involves processing raw construction safety documents and test data, then applying query expansion and a Retrieval Augmented Generation (RAG) pipeline built around language models and re-ranking.

##### Main Components
- Data Sources and Data Layer:
- Raw datasets found in CSV files (e.g., train.csv, test.csv, test_preprocessed.csv) and JSONL files in the data/documents folder that include various safety guidelines.
- Additional reference documents like PDFs in the “건설안전지침” subdirectory present historical or traditional file sources.
- Preprocessing:
- Notebooks under the “notebooks” directory (e.g., data-preprocessing-csv.ipynb, data-preprocessing-pdf.ipynb) are used to clean, transform, and prepare data for subsequent stages.
- Query Expansion Module:
- The script src/expand_query.py is responsible for enhancing queries with additional context using an LLM (rtzr/ko-gemma-2-9b-it). This module takes preprocessed test data and outputs the expanded queries.
- Retrieval Augmented Generation (RAG) Pipeline:
- The main pipeline is executed via src/main.py. It orchestrates the following sub-components:
- Embedding Model (using nlpai-lab/KURE-v1) to convert documents and queries into vector space.
- A vector database-like module implemented in src/vectordb.py that likely helps to perform similarity match between the expanded queries and indexed document embeddings.
- The LLM component (rtzr/ko-gemma-2-9b-it) for answer generation along with prompt design (src/prompt.py).
- A re-ranking module leveraging a re-ranker model (dragonkue/bge-reranker-v2-m3-ko) to choose the best suggestions among top-k documents.
- Utility Components:
- Supporting scripts like src/arguments.py, src/utils.py, and src/llm.py for orchestration, parameter passing, logging, and LLM interactions.

##### Relationships and Data Flow
- Data flows start from raw inputs (CSV/JSONL/PDF). Preprocessing notebooks ingest and clean this data.
- The output from the preprocessing is fed into the Query Expansion module, where the input queries are augmented using the LLM.
- Expanded queries and preprocessed test data are then passed into the RAG pipeline:
- The embedding model converts both queries and documents into vectors.
- The vectordb module performs similarity search to retrieve top-k relevant documents.
- A re-ranking step refines these results further.
- Finally, the LLM (along with prompt inputs from src/prompt.py) generates the final text responses.
- Final outputs (e.g., submission CSV files in the submissions directory) represent the culmination of the process.

##### Architectural Patterns and Design Principles
- Modular Design: Each stage is encapsulated as a module (preprocessing, query expansion, retrieval, generation), promoting separation of concerns.
- Pipeline Architecture: Data flows in a sequential manner from data ingestion to generation, with clear intermediate steps.
- Integration of External Services: The project leverages Hugging Face models for language generation, leveraging external libraries (e.g., flash-attn) for performance.
- Use of a Vector Search (RAG) Pattern: Transforming natural language into vector embeddings and then retrieving similar documents is typical in modern retrieval-based systems.
- Extensibility: The design is modular so that one can swap out models (embedding, re-ranker, LLM) or update preprocessing steps as needed.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DS["Data Sources (CSV, JSONL, PDFs)"]:::data
    end

    %% Preprocessing
    subgraph "Preprocessing"
        NP["Preprocessing Notebooks"]:::processing
    end

    %% Query Expansion
    subgraph "Query Expansion"
        QE["Query Expansion Module"]:::processing
    end

    %% RAG Pipeline
    subgraph "RAG Pipeline"
        OP["Pipeline Orchestration (main.py)"]:::processing
        Emb["Embedding Model (KURE-v1)"]:::model
        Vect["Vector Similarity Search (vectordb)"]:::processing
        RR["Re-ranking Module (bge-reranker)"]:::processing
        LLM["LLM Component (rtzr/ko-gemma-2-9b-it)"]:::model
        Prompt["Prompt Design (prompt.py)"]:::model
    end

    %% Utility Components
    subgraph "Utility Components"
        Args["Arguments (arguments.py)"]:::utility
        Uti["Utils (utils.py)"]:::utility
    end

    %% Final Output
    subgraph "Final Output"
        FO["Final Submission (Submission CSV)"]:::output
    end

    %% Data Flow
    DS -->|"rawData"| NP
    NP -->|"cleanedData"| QE
    QE -->|"expandedQueries"| OP
    OP -->|"vectorEmbeddings"| Emb
    Emb -->|"searchVectors"| Vect
    Vect -->|"retrievedDocs"| RR
    RR -->|"filteredDocs"| LLM
    LLM -->|"promptInput"| Prompt
    Prompt -->|"generatedOutput"| FO

    %% Utility integration
    OP --- Args
    OP --- Uti

    %% Styles
    classDef data fill:#FCE5CD,stroke:#B45F06,stroke-width:2px;
    classDef processing fill:#D9EAD3,stroke:#38761D,stroke-width:2px;
    classDef model fill:#C9DAF8,stroke:#1155CC,stroke-width:2px;
    classDef utility fill:#FFF2CC,stroke:#BF9000,stroke-width:2px;
    classDef output fill:#D0E0E3,stroke:#0B5394,stroke-width:2px;

    %% Click Events
    click DS "https://github.com/whybe-choi/dacon-hansoldeco-season3/tree/main/data/"
    click NP "https://github.com/whybe-choi/dacon-hansoldeco-season3/tree/main/notebooks/"
    click QE "https://github.com/whybe-choi/dacon-hansoldeco-season3/blob/main/src/expand_query.py"
    click OP "https://github.com/whybe-choi/dacon-hansoldeco-season3/blob/main/src/main.py"
    click Vect "https://github.com/whybe-choi/dacon-hansoldeco-season3/blob/main/src/vectordb.py"
    click LLM "https://github.com/whybe-choi/dacon-hansoldeco-season3/blob/main/src/llm.py"
    click Prompt "https://github.com/whybe-choi/dacon-hansoldeco-season3/blob/main/src/prompt.py"
    click Args "https://github.com/whybe-choi/dacon-hansoldeco-season3/blob/main/src/arguments.py"
    click Uti "https://github.com/whybe-choi/dacon-hansoldeco-season3/blob/main/src/utils.py"
    click FO "https://github.com/whybe-choi/dacon-hansoldeco-season3/tree/main/submissions/"
```
