How whybe-choi/dacon-hansoldeco-season3 works
The dacon-hansoldeco-season3 project is an NLP pipeline developed for a construction accident prevention and response generation challenge. Its core functionality involves processing raw construction safety documents and test data, then applying query expansion and a Retrieval Augmented Generation (RAG) pipeline built around language models and re-ranking.
Main Components
• Data Sources and Data Layer:
• Raw datasets found in CSV files (e.g., train.csv, test.csv, test_preprocessed.csv) and JSONL files in the data/documents folder that include various safety guidelines.
• Additional reference documents like PDFs in the “건설안전지침” subdirectory present historical or traditional file sources.
• Preprocessing:
• Notebooks under the “notebooks” directory (e.g., data-preprocessing-csv.ipynb, data-preprocessing-pdf.ipynb) are used to clean, transform, and prepare data for subsequent stages.
• Query Expansion Module:
• The script src/expand_query.py is responsible for enhancing queries with additional context using an LLM (rtzr/ko-gemma-2-9b-it). This module takes preprocessed test data and outputs the expanded queries.
• Retrieval Augmented Generation (RAG) Pipeline:
• The main pipeline is executed via src/main.py. It orchestrates the following sub-components:
• Embedding Model (using nlpai-lab/KURE-v1) to convert documents and queries into vector space.
• A vector database-like module implemented in src/vectordb.py that likely helps to perform similarity match between the expanded queries and indexed document embeddings.
• The LLM component (rtzr/ko-gemma-2-9b-it) for answer generation along with prompt design (src/prompt.py).
• A re-ranking module leveraging a re-ranker model (dragonkue/bge-reranker-v2-m3-ko) to choose the best suggestions among top-k documents.
• Utility Components:
• Supporting scripts like src/arguments.py, src/utils.py, and src/llm.py for orchestration, parameter passing, logging, and LLM interactions.
Relationships and Data Flow
• Data flows start from raw inputs (CSV/JSONL/PDF). Preprocessing notebooks ingest and clean this data.
• The output from the preprocessing is fed into the Query Expansion module, where the input queries are augmented using the LLM.
• Expanded queries and preprocessed test data are then passed into the RAG pipeline:
• The embedding model converts both queries and documents into vectors.
• The vectordb module performs similarity search to retrieve top-k relevant documents.
• A re-ranking step refines these results further.
• Finally, the LLM (along with prompt inputs from src/prompt.py) generates the final text responses.
• Final outputs (e.g., submission CSV files in the submissions directory) represent the culmination of the process.
Architectural Patterns and Design Principles
• Modular Design: Each stage is encapsulated as a module (preprocessing, query expansion, retrieval, generation), promoting separation of concerns.
• Pipeline Architecture: Data flows in a sequential manner from data ingestion to generation, with clear intermediate steps.
• Integration of External Services: The project leverages Hugging Face models for language generation, leveraging external libraries (e.g., flash-attn) for performance.
• Use of a Vector Search (RAG) Pattern: Transforming natural language into vector embeddings and then retrieving similar documents is typical in modern retrieval-based systems.
• Extensibility: The design is modular so that one can swap out models (embedding, re-ranker, LLM) or update preprocessing steps as needed.