How amitbcp/multilingual_profanity works
Project type and purpose
• This is an open-source benchmarking toolkit (“SweEval-Bench”) for evaluating LLMs on safety by prompting them with multilingual swear-word instructions and measuring harmful outputs. It is not a web or mobile app but rather a data-science/ML evaluation pipeline comprised of Python scripts, Jupyter notebooks, data files, and metric outputs.
File structure
• notebooks/
• dataset/Swear_Words_Dataset.ipynb – exploration and construction of the prompt dataset
• inference/multi_experiment.ipynb, multi_experiment_2.ipynb – interactive model inference and result visualization
• src/
• downloader.py – retrieves models or data artifacts if needed
• load_dataset.py, dataset_config.py – loads prompt files and case-specific JSONs (case_1.json, case_2.json, case_3.json)
• load_model.py, infer_model.py – instantiates LLMs (e.g., via HuggingFace or API) and runs inference
• annotate_inference.py, eval_utils.py – applies manual or semi-automated annotation logic to mark harmful responses
• calculate_metrics.py, utils.py – computes harmful rates and other metrics, writes to Excel (src/metrics/*.xlsx)
• paths.py – centralizes directory paths for dataset, outputs, logs
• src/metrics/ – stores raw metric spreadsheets for each case
• requirements.txt – lists Python dependencies
• README.md – project overview, methodology, results, and proposed final structure
README for additional insights
• Architecture is implied as a linear data-processing pipeline: Dataset formation → Model inference → Manual annotation → Metric computation → Visualization/analysis.