How rajpandeygithub/automated-bilingual-complaint-system works
The repository implements a bilingual banking-complaint system: users submit English or Hindi complaints, which the API validates and transforms, checks for drift, classifies by product and department, and records. Separate Airflow data-preparation and Kubeflow model-training workflows support the serving system. The README also describes language-aware agent assignment, but the supplied source does not show its implementation. Edges reflect shown calls or documented workflow; unsampled subsystems are included without asserting unverified connections.
Main entry points: submit_train_pipeline.py Training orchestrator · airflow.py Preprocessing DAGs · server.py Prediction API · frontend.py Complaint portal
Components 30
Complaint serving
- Complaint portal
frontend/frontend.py - API client
frontend/backend.py - Prediction API
backend/server.py - Text validation and cleanup
backend/preprocessing.py - Model inference
backend/inference.py - Product and department models
- Prediction records
- BigQuery writer
backend/bigquery_operations.py - Language-aware agent assignment
Data preparation
- Preprocessing DAGs
data_preprocessing_pipeline/dags/airflow.py - Data quality checks
data_preprocessing_pipeline/dags/scripts/data_quality.py - Data de-identification
data_preprocessing_pipeline/dags/scripts/deidentification.py - Batch cleaning
data_preprocessing_pipeline/dags/scripts/preprocessing.py - Statistics generation
data_preprocessing_pipeline/dags/scripts/statistics_generation.py - Agent dataset preparation
data_preprocessing_pipeline/dags/scripts/agent_dataset.py
Model lifecycle
- Training orchestrator
train/submit_train_pipeline.py - Training data acquisition
train/components/get_data.py - Tokenized dataset preparation
train/components/prepare_data.py - Model training
train/components/hf_model_train.py - Model evaluation
train/components/hf_model_test.py - Best model selection
train/components/select_best_model.py - Bias assessment
train/components/bias_detection.py - Model registry
train/components/model_registry.py - Model deployment
train/components/model_deployment.py
Monitoring and operations
- Drift detection
data_drift/drift_cloud_fn_hindi.py - English drift support
data_drift/Data_drift.py - Pipeline email alerts
data_preprocessing_pipeline/dags/scripts/failure_email.py - Pipeline success alerts
data_preprocessing_pipeline/dags/scripts/success_email.py - Slack notifications
Other components
- Complaint submitter
Connections 27
- Complaint submitter → Complaint portal: submits complaint
README.md - Complaint portal → API client: sends text
frontend/frontend.py - API client → Prediction API: posts complaint
frontend/backend.py - Prediction API → Text validation and cleanup: validates and cleans
backend/server.py - Prediction API → Model inference: requests predictions
backend/server.py - Model inference → Product and department models: calls endpoints
backend/inference.py - Prediction API → BigQuery writer: writes prediction
backend/server.py - BigQuery writer → Prediction records: inserts record
backend/bigquery_operations.py - Prediction API → Drift detection: checks complaint
- Prediction API → English drift support: checks complaint
- Preprocessing DAGs → Data quality checks: runs checks
data_preprocessing_pipeline/dags/airflow.py - Preprocessing DAGs → Data de-identification: runs de-identification
data_preprocessing_pipeline/dags/airflow.py - Preprocessing DAGs → Batch cleaning: runs cleaning
data_preprocessing_pipeline/dags/airflow.py - Preprocessing DAGs → Statistics generation: generates statistics
data_preprocessing_pipeline/dags/airflow.py - Preprocessing DAGs → Pipeline email alerts: sends failure alerts
data_preprocessing_pipeline/dags/airflow.py - Preprocessing DAGs → Pipeline success alerts: sends success alerts
data_preprocessing_pipeline/dags/airflow.py - Preprocessing DAGs → Agent dataset preparation: prepares agent data
- Training orchestrator → Training data acquisition: acquires data
train/submit_train_pipeline.py - Training orchestrator → Tokenized dataset preparation: prepares datasets
train/submit_train_pipeline.py - Training orchestrator → Model training: trains models
train/submit_train_pipeline.py - Training orchestrator → Model evaluation: tests models
train/submit_train_pipeline.py - Training orchestrator → Best model selection: selects best model
train/submit_train_pipeline.py - Training orchestrator → Bias assessment: assesses bias
train/submit_train_pipeline.py - Training orchestrator → Model registry: registers model
train/submit_train_pipeline.py - Training orchestrator → Model deployment: deploys conditionally
train/submit_train_pipeline.py - Training orchestrator → Slack notifications: notifies pipeline
- Complaint portal → Language-aware agent assignment: supports assignment