How humanai-foundation/issr works
Overview & Project Purpose
• ISSR is an umbrella repository supporting multiple research projects in the social sciences. Each project focuses on different aspects—from analyzing language evolution on the Dark Web (using clustering, topic modeling, and deep learning) to gender roles analysis in educational choices, mixed-method data prediction, and fatigue/distracted driving detection using computer vision.
• The purpose is to provide reproducible analyses, state-of-the-art ML pipelines (e.g., BERT models, LSTM, LightGBM), and data visualizations that support interdisciplinary research.
Main Architectural Components
a. Projects as Modules:
• ISSR_Dark_Web_Domenico_Lacavalla: Contains sub-components for content analysis, thread analysis, topic modeling (BERTopic, LLAMA), and even computer vision aspects (for visual topic models).
• ISSR_Gender_Roles_Career_Rashi_Gupta: Houses analysis notebooks, reports, and presentations related to grade-wise educational data.
• ISSR_Improve_Accuracy_Mixed_Data_Shao_Jin: Comprises regression and classification analyses on mixed data, including raw dataset input and output analysis notebooks.
• ISSR_Fatigue_detection: Focuses on driver fatigue detection using computer vision; it includes YOLO, Mediapipe, and CLIP-based pipelines.
b. Common Components Across Projects:
• Data Sources & Datasets: Each project includes its own raw/input data files as well as processed datasets (e.g., CSVs, Excel files, and specialized .safetensors for ML models).
• Preprocessing & Utility Modules: The Util directory includes utility scripts for text preprocessing, clustering, and model helper functions.
• ML Models & Analysis Scripts: Multiple directories (such as MLModelsBERT, MergedModelBERT, and notebooks in each project) show the separation of training, saved models, and inference pipelines.
• Visualization & Reporting: The ShowResultsHTML directory and presentation files (e.g., PowerPoints) indicate a reporting layer used to share reproducible results.
• Experimentation & Grid Search: Directories like ResultsGridSearchBERTopic indicate components dedicated to hyperparameter tuning and analysis.
Relationships & Data Flow
• Data is ingested from various datasets (raw text files, annotated CSVs from computer vision experiments, XLSX files for quantitative analysis).
• Data Preprocessing & Utility Layer:
• Raw data is cleaned/preprocessed using scripts in the Util directory and separate notebooks in each analysis folder.
• ML Model Training & Analysis:
• Preprocessed data is fed into topic modeling (BERTopic, LLAMA), deep learning models (LSTM, YOLO, LightGBM, etc.), and other classification/regression models. Each project’s “Models” or “SavedModels” directories indicate training outputs.
• There’s further merging of results, as seen in the MergedModelBERT module, where content and thread analysis outcomes are combined.
• Result Generation & Visualization:
• Outputs (CSV results, HTML reproducibility reports, and interactive notebooks) are then visualized and shared. The ShowResultsHTML directory and various Jupyter notebooks support this stage.
• Feedback Loops: