How antonisraf/nlp-tagging-consumer-complaints works
1. What this repo is
ComplaintFlow is a Python-based NLP system for CFPB student-loan complaint tagging. Its core architecture is a supervised text-classification pipeline that turns free-text complaint narratives into hierarchical labels, then routes uncertain or disputed predictions to a human review loop.
2. Primary architectural layers
The repository is split into four main layers: data exploration, model preparation/training, model evaluation, and a Streamlit application. The same text-cleaning logic is intended to be reused across training and inference to avoid training-serving skew.
3. Data boundary
Raw complaint data lives under data/ as CSV files, with a cleaned NLP-ready export also present there and in NLP_Model_training/DATA/. The raw dataset is the source of truth for EDA and preprocessing, while the cleaned dataset is the input boundary for vectorization and model training.
4. EDA subsystem
EDA/EDA.ipynb and EDA/Report_EDA.md represent the analysis stage. This is where the dataset was filtered, label distributions were examined, low-signal metadata was pruned, and the label hierarchy and quality gates were determined. The EDA outputs drive later design choices but are not part of runtime inference.
5. Preprocessing and feature extraction
NLP_Model_training/vectorizer.py and app/model_pipeline.py define the text feature boundary. The pipeline applies aggressive normalization, lemmatization, token filtering, and TF-IDF vectorization. The trained vectorizer is an artifact shared between training and app inference so the same feature space is used end to end.
6. Model training subsystem
Training artifacts are created in app/train_and_save_models.py and supported by NLP_Model_training/utils_NLP.py. The project uses hierarchical classification with a coarse group prediction followed by a finer sub-issue prediction. The readme indicates ensemble classifiers built from Logistic Regression and calibrated LinearSVC, with out-of-fold probabilities used to prevent leakage in the cascade.
7. Model evaluation and experiment boundary
NLP_Model_testing/hierarchical_baseline_experiment.py and NLP_Model_testing/model_evaluation.py form the validation layer. The baseline script explores alternative groupings and grid search settings, while the evaluation script locks in the chosen architecture, computes metrics, and determines the confidence threshold used to send low-confidence cases to review.
8. Application runtime
The app/ package is the operational layer. streamlit_app.py is the entry point, model_pipeline.py handles inference, complaint_generator.py produces synthetic complaints, and human_review_section.py manages reviewer corrections. This layer combines model scoring, dashboard visualization, and workflow control in one UI.
9. Human-in-the-loop workflow
The app is explicitly designed as a review system, not just a classifier. Predictions below the configured confidence threshold are flagged, and the review queue can also force cases into review when the top-level label is wrong. Once a human finalizes a decision, those reviewed rows are treated as corrected and the app recomputes metrics.
10. External dependencies and operational constraints
The app can call Groq’s llama-3.3-70b-versatile model to synthesize complaint text, using few-shot examples from anonymized narratives. API usage is rate-limited per session, and the API key is provided through .env or entered interactively. Streamlit is the runtime UI framework, and the project is configured through .streamlit/config.toml.
11. Artifacts and outputs
Model and preprocessing artifacts are generated rather than hand-edited, then consumed by the app at runtime. Plots and markdown reports under plots/, EDA/, NLP_Model_training/, NLP_Model_testing/, and app/ document the pipeline and provide traceability for the architectural decisions.
12. End-to-end data flow
The overall flow is: raw CFPB complaints → EDA filtering and label engineering → text cleaning and TF-IDF vectorization → hierarchical model training and threshold tuning → saved artifacts → Streamlit inference on synthetic or real text → human review for uncertain or incorrect predictions → updated results dashboard.
Main entry points: streamlit_app.py Streamlit app · EDA.ipynb EDA notebook · train_and_save_models.py Train models · model_evaluation.py Model evaluation
Components 18
Data & EDA
Raw complaints, cleaned exports, and exploratory analysis that define the label schema and preprocessing choices.
- Raw complaints · Source-of-truth CFPB student-loan complaint narratives and metadata.
data/student_loan_complaints.csv - Cleaned data · NLP-ready complaint dataset used as the boundary between exploration and modeling.
data/student_loan_nlp_clean.csv - EDA notebook · Explores labels, filters low-signal fields, and informs hierarchy and quality gates.
EDA/EDA.ipynb - EDA report · Narrative summary of exploratory findings and dataset decisions.
EDA/Report_EDA.md
Training & Evaluation
Model preparation, vectorization, hierarchical training, and experiment evaluation.
- Vectorizer · Text normalization and TF-IDF feature extraction shared by training and inference.
NLP_Model_training/vectorizer.py - NLP utils · Reusable text-cleaning and NLP utility functions supporting the model pipeline.
NLP_Model_training/utils_NLP.py - Train models · Builds hierarchical classifiers and persists the trained artifacts for runtime use.
app/train_and_save_models.py - Baseline experiment · Explores grouping and grid-search settings for hierarchical classification.
NLP_Model_testing/hierarchical_baseline_experiment.py - Model evaluation · Computes metrics and selects the confidence threshold for routing cases to review.
NLP_Model_testing/model_evaluation.py - Testing report · Documented validation results and experiment conclusions.
NLP_Model_testing/Report_NLP_Model_testing.md
Application Runtime
Streamlit UI, inference pipeline, synthetic complaint generation, and human review loop.
- Model pipeline · Loads artifacts, preprocesses text, vectorizes input, and returns hierarchical predictions.
app/model_pipeline.py - Streamlit app · Operational dashboard for scoring, review, and metrics visualization.
app/streamlit_app.py - Complaint generator · Creates example complaints for demo and testing workflows.
app/complaint_generator.py - Review section · Handles reviewer corrections, disputed predictions, and queue management.
app/human_review_section.py - App report · Notes about the runtime workflow and app behavior.
app/Report_app.md
Artifacts
Saved models, plots, and reports consumed or produced across the pipeline.
- Training export · Cleaned dataset copy stored with training assets for reproducible model fitting.
NLP_Model_training/DATA/student_loan_nlp_clean.csv - Plots · Saved visual artifacts that document data exploration and model evaluation.
plots - Project README · Top-level repository description and usage guidance.
README.md
Connections 20
- Raw complaints → EDA notebook: explore · EDA begins from raw complaint records.
- EDA notebook → EDA report: summarize · Findings are captured in the EDA report.
- EDA notebook → Cleaned data: filter/clean · Exploration drives label engineering and dataset pruning.
- Cleaned data → Training export: copy · Cleaned data is exported for training-time use.
- Cleaned data → Vectorizer: fit input · Clean text is the input boundary for feature extraction.
- NLP utils → Vectorizer: normalize · Shared NLP helpers support the feature pipeline.
- Vectorizer → Train models: features · Vectorized text feeds model training.
- NLP utils → Train models: preprocess · Training reuses the same text-cleaning logic.
- Train models → Model evaluation: validate · Trained models are evaluated and thresholded.
- Baseline experiment → Model evaluation: compare · Baseline experiments inform the final evaluation setup.
- Model evaluation → Testing report: record · Metrics and threshold decisions are documented.
- Model evaluation → Plots: visualize · Evaluation produces supporting plots.
- Train models → Model pipeline: save artifacts · Inference loads the persisted models and vectorizer.
- Model pipeline → Streamlit app: serve predictions · The UI calls the pipeline for scoring.
- Complaint generator → Streamlit app: demo input · Synthetic complaints can be generated inside the app.
- Review section → Streamlit app: corrections · Human review updates disputed or low-confidence cases.
- Streamlit app → Review section: route low confidence · Cases below threshold are sent to review.
- App report → Streamlit app: document · App behavior and workflow are described in the app report.
- Project README → EDA notebook: orient · Repository documentation contextualizes the analysis stage.
- Project README → Streamlit app: orient · Repository documentation contextualizes the runtime app.