How vera-bernhard/swisstextsdg works
Main Components
• Data Input and Storage:
• “data/raw”, “data/preprocessed”, and other data sources.
• Configuration files (e.g., sdg_label_mapping.json, sgd_label_subtargets.json, settings_dist.py) that are used to define mappings/settings.
• Data Processing and Preprocessing:
• Notebooks like preprocessing.ipynb and scripts in “scripts/” and “src/data/” (e.g., text_preprocessing.py, tokenizer.py, dataset.py) that handle cleaning, tokenizing, and data configuration.
• Model Training and Evaluation:
• “src/models/” splits into subfolders: Llama2, Mistral, RAG, ensemble, mbert, qlora.
• Each subdirectory contains scripts and notebooks for training, testing, and evaluation.
• Specific notebooks (e.g., eval_classification_with_Llama.ipynb, run_ensemble_predict.py) signify separate training/testing branches.
• Ensemble and Voting Mechanisms:
• Look at the ensemble directories under “results/” and “src/models/ensemble/” showing majority/soft voting implementations.
• Helpers and Utilities:
• “src/helpers” containing logging, model, path helpers, etc.
• Utility scripts in “data/” (e.g., extract_fulltext.py, helper.py) that support the data layer.
• Results and Analysis:
• “results/” directory with structured output such as CSV prediction logs.
• Notebooks in “scripts/” for further analysis (e.g., analyze_data.ipynb, confusion_matrices.ipynb).
Relationships and Interactions
• Data flows from the raw sources through the preprocessing modules:
• Raw data → Preprocessing (data cleaning, tokenizing) → Preprocessed files.
• Preprocessed data then feeds into various model-specific pipelines:
• Each model (Llama2, Mistral, etc.) reads preprocessed data, trains or performs predictions.
• The outputs from individual model runs are logged into the “results/” directory.
• Ensemble modules combine results from different models:
• Ensemble scripts in “src/models/ensemble” merge predictions (using majority or soft voting) and produce final outputs.
• Analysis notebooks consume both raw predictions and processed results for evaluation.
• Helper modules support each of these stages with shared utilities and configurations.
Architectural Patterns and Design Principles
• Modular Pipeline: A clear separation of concerns where data ingestion, preprocessing, model training/evaluation, and ensemble analysis are isolated, making it easier to update one section without impacting others.
• Experimentation Focus: Multiple directories for different model architectures and integration of ensemble methods indicate an extensible research pipeline.
• Reusability & Utility: The presence of helper modules for logging, paths, and seed management shows an emphasis on reproducibility.
• Integration with External Tools: The README shows guidelines for working in Colab/GitHub, suggesting that the design must support remote and collaborative development.
Relevant Technologies/Frameworks
• Python, Jupyter Notebooks, and Google Colab integration.
• Machine Learning frameworks (likely PyTorch for transformer models, as suggested by files named pytorch_model.py and run_transformer_training.py).
• Data format handling with JSONL, CSV, and Pickle.
• Possibly dependency management via pyproject.toml and requirements.txt.
• Data Inputs:
• Raw Data Files (from “data/raw”)
• Configuration Files (json and settings scripts)
• Preprocessing Layer:
• Preprocessor scripts and notebooks (“src/data/”, “preprocessing.ipynb”)
• Model Layers:
• Individual Model Pipelines: Llama2, Mistral, RAG, MBERT, QLORA, etc. (with scripts and notebooks indicating training and prediction)
• Ensemble Methods (both with and without QLORA, majority and soft voting approaches)
• Post-Processing and Evaluation:
• Results Folder: CSV logs and evaluation metrics
• Analysis Notebooks (from “scripts/”)
• Support Utilities:
• Helper Modules (“src/helpers”)