How aaravkataria24/xtesting-backend works
What this project is (type + purpose)
From the README and file tree, this is a Python backend service for “TweetLab”, deployed on Google Cloud, that exposes HTTP API endpoints:
• /api/tweets (retrieve tweet data)
• /api/analyze (sentiment analysis)
• /api/stats (tweet statistics)
Architecturally it’s also an ML-backed service: it loads multiple pre-trained artifacts at runtime (RoBERTa fine-tuned model + several scikit-learn regressors + TF-IDF vectorizer). The repo also contains offline ML pipeline scripts for scraping, preprocessing, fine-tuning, evaluation, and validation.
How to read the file structure (what maps to architecture)
Top-level signals:
Online service / runtime
• app.py (likely the web server + route handlers; README says main.py but tree shows app.py—verify which is entrypoint)
• predict.py (likely inference entrypoint used by handlers)
• model.py, model_colab.py (model loading / wrappers)
• requirements.txt (runtime deps)
• Dockerfile (containerization)
• .env variables in README: API_KEY, MODEL_PATH
Offline ML + data workflow (batch jobs, not part of serving path)
• Scraping / data acquisition:
• scrape_tabasco_1000.py, save_cookies.py
• Preprocessing / dataset creation:
• preprocess_tweets.py, data_preprocessing.py, merge_csvs.py, temp_all.py
• Training / fine-tuning:
• fine_tune_roberta.py, colab_fine_tune_roberta.py
• archive/* includes iterative experiments: regressor training, scalers, continuing finetune, etc.
• Evaluation / QA:
• evaluate_model.py, validate_model.py, analyze_errors.py
• evaluation_results/ (outputs)
Required runtime artifacts (from README)
• finetuned_twitter_roberta_multi.pt (PyTorch/Transformers)
• regressor_model.pkl, replies_model.pkl, retweets_model.pkl, likes_model.pkl (sklearn)
• tfidf_vectorizer.pkl
API Layer (routes/controllers)
• /api/tweets handler (in app.py)
• /api/analyze handler
• /api/stats handler
Inference Layer
• predict.py (request-to-prediction orchestration)
• model.py (model loading, tokenization, forward pass)
• “Feature extraction” (TF-IDF vectorizer)
• “Regressors” (likes/retweets/replies models)
Data Access / Acquisition
• If live: a “Tweet Fetcher” module (Twitter API client or scraping wrapper)
• If offline/static: “Dataset Store” (CSV/DB) — determine from code
Shared Utilities
• Preprocessing functions (if reused at runtime): preprocess_tweets.py / data_preprocessing.py
• Client → /api/analyze → predict.py → RoBERTa (finetuned_twitter_roberta_multi.pt) → Response
• Client → /api/stats → predict.py → TF-IDF (tfidf_vectorizer.pkl) → regressors (*_model.pkl) → Response
• Client → /api/tweets → Tweet Fetcher (Twitter API/scraper) OR Dataset Store → Response
Data ingestion
• scrape_tabasco_1000.py, save_cookies.py
Preprocessing / dataset build
• preprocess_tweets.py, data_preprocessing.py, merge_csvs.py
Model training
• Fine-tune transformer: fine_tune_roberta.py / colab_fine_tune_roberta.py
• Train regressors: (some are in archive/ like train_regressor.py)
Evaluation / validation
• evaluate_model.py, validate_model.py, analyze_errors.py
Artifact output
• .pt + .pkl files (the exact list in README)
• Offline pipeline → Model Artifact Storage → Online backend loads artifacts
This separation is the key architectural truth of this repo.
Relationships and interactions to emphasize (what makes this system “itself”)
• “Model-as-a-dependency”: the backend is not “just API code”; it is a thin HTTP layer around multiple ML artifacts.
• Multi-model inference:
• Transformer model for sentiment (RoBERTa)
• TF-IDF + regressors for engagement/stat predictions
• Config-driven artifact loading via MODEL_PATH
• Cloud deployment boundary (Google Cloud) with container/runtime and environment variables
Before finalizing, quickly confirm these in app.py / entrypoint:
• Framework (Flask vs FastAPI) and how routes are registered
• Whether /api/tweets reads from:
• external API,
• scraping at runtime,
• a local CSV,
• or a database (none is visible in tree, so likely not DB-backed)
• Where artifacts are loaded from (local path vs download)
• Actual entrypoint (app.py vs a missing main.py referenced by README)
What you find by adjusting:
• “Tweet Fetcher” component (present/absent)
• “Persistent DB” (likely absent; don’t invent it)
• Exact GCP runtime (App Engine vs Cloud Run via Docker)
This yields a detailed, accurate architecture while keeping the system understandable: online inference service powered by offline-produced model artifacts, deployed on Google Cloud, with clearly labeled API flows and ML dependencies.