# aaravkataria24/xtesting-backend architecture

> Architecture diagram of the GitHub repository aaravkataria24/xtesting-backend, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aaravkataria24/xtesting-backend
- Repository: https://github.com/aaravkataria24/xtesting-backend

## Overview

##### What this project is (type + purpose)
From the README and file tree, this is a **Python backend service** for “TweetLab”, deployed on **Google Cloud**, that exposes HTTP API endpoints:

- `/api/tweets` (retrieve tweet data)
- `/api/analyze` (sentiment analysis)
- `/api/stats` (tweet statistics)

Architecturally it’s also an **ML-backed service**: it loads multiple pre-trained artifacts at runtime (RoBERTa fine-tuned model + several scikit-learn regressors + TF-IDF vectorizer). The repo also contains **offline ML pipeline scripts** for scraping, preprocessing, fine-tuning, evaluation, and validation.

##### How to read the file structure (what maps to architecture)
Top-level signals:

##### Online service / runtime
- `app.py` (likely the web server + route handlers; README says `main.py` but tree shows `app.py`—verify which is entrypoint)
- `predict.py` (likely inference entrypoint used by handlers)
- `model.py`, `model_colab.py` (model loading / wrappers)
- `requirements.txt` (runtime deps)
- `Dockerfile` (containerization)
- `.env` variables in README: `API_KEY`, `MODEL_PATH`

##### Offline ML + data workflow (batch jobs, not part of serving path)
- Scraping / data acquisition:
- `scrape_tabasco_1000.py`, `save_cookies.py`
- Preprocessing / dataset creation:
- `preprocess_tweets.py`, `data_preprocessing.py`, `merge_csvs.py`, `temp_all.py`
- Training / fine-tuning:
- `fine_tune_roberta.py`, `colab_fine_tune_roberta.py`
- `archive/*` includes iterative experiments: regressor training, scalers, continuing finetune, etc.
- Evaluation / QA:
- `evaluate_model.py`, `validate_model.py`, `analyze_errors.py`
- `evaluation_results/` (outputs)

##### Required runtime artifacts (from README)
- `finetuned_twitter_roberta_multi.pt` (PyTorch/Transformers)
- `regressor_model.pkl`, `replies_model.pkl`, `retweets_model.pkl`, `likes_model.pkl` (sklearn)
- `tfidf_vectorizer.pkl`

##### API Layer (routes/controllers)
- `/api/tweets` handler (in `app.py`)
- `/api/analyze` handler
- `/api/stats` handler

##### Inference Layer
- `predict.py` (request-to-prediction orchestration)
- `model.py` (model loading, tokenization, forward pass)
- “Feature extraction” (TF-IDF vectorizer)
- “Regressors” (likes/retweets/replies models)

##### Data Access / Acquisition
- If live: a “Tweet Fetcher” module (Twitter API client or scraping wrapper)
- If offline/static: “Dataset Store” (CSV/DB) — determine from code

##### Shared Utilities
- Preprocessing functions (if reused at runtime): `preprocess_tweets.py` / `data_preprocessing.py`

- Client → `/api/analyze` → `predict.py` → RoBERTa (`finetuned_twitter_roberta_multi.pt`) → Response
- Client → `/api/stats` → `predict.py` → TF-IDF (`tfidf_vectorizer.pkl`) → regressors (`*_model.pkl`) → Response
- Client → `/api/tweets` → Tweet Fetcher (Twitter API/scraper) OR Dataset Store → Response

##### Data ingestion
- `scrape_tabasco_1000.py`, `save_cookies.py`
##### Preprocessing / dataset build
- `preprocess_tweets.py`, `data_preprocessing.py`, `merge_csvs.py`
##### Model training
- Fine-tune transformer: `fine_tune_roberta.py` / `colab_fine_tune_roberta.py`
- Train regressors: (some are in `archive/` like `train_regressor.py`)
##### Evaluation / validation
- `evaluate_model.py`, `validate_model.py`, `analyze_errors.py`
##### Artifact output
- `.pt` + `.pkl` files (the exact list in README)

- Offline pipeline → Model Artifact Storage → Online backend loads artifacts

This separation is the key architectural truth of this repo.

##### Relationships and interactions to emphasize (what makes this system “itself”)
- **“Model-as-a-dependency”**: the backend is not “just API code”; it is a thin HTTP layer around multiple ML artifacts.
- **Multi-model inference**:
- Transformer model for sentiment (RoBERTa)
- TF-IDF + regressors for engagement/stat predictions
- **Config-driven artifact loading** via `MODEL_PATH`
- **Cloud deployment boundary** (Google Cloud) with container/runtime and environment variables

Before finalizing, quickly confirm these in `app.py` / entrypoint:
- Framework (Flask vs FastAPI) and how routes are registered
- Whether `/api/tweets` reads from:
- external API,
- scraping at runtime,
- a local CSV,
- or a database (none is visible in tree, so likely not DB-backed)
- Where artifacts are loaded from (local path vs download)
- Actual entrypoint (`app.py` vs a missing `main.py` referenced by README)

What you find by adjusting:
- “Tweet Fetcher” component (present/absent)
- “Persistent DB” (likely absent; don’t invent it)
- Exact GCP runtime (App Engine vs Cloud Run via Docker)

This yields a detailed, accurate architecture while keeping the system understandable: **online inference service** powered by **offline-produced model artifacts**, deployed on **Google Cloud**, with clearly labeled API flows and ML dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TD

%% =========================
%% DIAGRAM A — SYSTEM CONTEXT
%% =========================
subgraph "A) System Context"
direction TB

U["TweetLab Frontend / Client"]:::external

subgraph "Google Cloud"
direction TB
BCTX["TweetLab Backend API (Python service)"]:::runtime
end

X["Twitter/X Data Source (API or scraping target)"]:::external
GCP["Google Cloud Managed Services\n(Run/App Engine, Storage, Logging, Secrets)"]:::cloud

U -->|"HTTPS/REST (/api/*)"| BCTX
BCTX -->|"fetches_tweets (optional)"| X
BCTX -->|"deploys_on/uses"| GCP
end

%% =========================
%% DIAGRAM B — CONTAINER VIEW
%% =========================
subgraph "B) Container Diagram (Deployable Units + Storage)"
direction TB

subgraph "Google Cloud Runtime Boundary"
direction TB

C1["HTTP Serving Container\n(Python API Service)"]:::runtime
CFG["Secrets/Config\n(Env vars / Secret Manager)\nAPI_KEY, MODEL_PATH"]:::cloud
LOG["Observability\n(Cloud Logging/Monitoring)"]:::cloud
ARTS[("Model Artifact Storage\n(GCS bucket or container filesystem)")]:::storage

end

U2["Client (Browser/App)"]:::external

U2 -->|"HTTPS/REST"| C1
C1 -->|"reads_env"| CFG
C1 -->|"writes_logs/metrics"| LOG
C1 -->|"loads_artifacts_at_startup (MODEL_PATH)"| ARTS

subgraph "Build Inputs"
direction TB
DOCKER["Dockerfile"]:::build
REQS["requirements.txt"]:::build
end

N1["Docker Image Build"]:::build
DOCKER -->|"used_by"| N1
REQS -->|"used_by"| N1
N1 -->|"produces"| C1
end

%% ==================================================
%% DIAGRAM C — BACKEND COMPONENTS + OFFLINE PIPELINE
%% ==================================================
subgraph "C) Backend Components (Online) + Offline ML Pipeline (Separate Lane)"
direction TB

%% ---- Online Inference Serving ----
subgraph "Online: Inference Serving Path (Runtime)"
direction TB

subgraph "API Layer"
direction TB
API["API Service / HTTP Server\nRoutes: /api/tweets, /api/analyze, /api/stats"]:::runtime
end

subgraph "Inference Layer (Multi-model)"
direction TB
PRED["Inference Orchestration\n(predict.py)"]:::ml
MOD["Model Wrapper / Loader\n(model.py)"]:::ml
MODC["Alternate Model Wrapper (Experiment)\n(model_colab.py)"]:::ml

SENT["Sentiment Model\nRoBERTa fine-tuned (.pt)"]:::ml
VEC["Feature Extraction\nTF-IDF Vectorizer (.pkl)"]:::ml
REG["Engagement Regressors\nlikes/retweets/replies/regressor (.pkl)"]:::ml
end

subgraph "Data Acquisition (Runtime)"
direction TB
FETCH["Tweet Fetcher\n(Twitter API client or scraping wrapper)"]:::runtime
end

subgraph "Shared Utilities (Optional Runtime Reuse)"
direction TB
PREU["Preprocessing Utilities\n(token cleanup/transforms)"]:::util
end

ARTS2[("Model Artifacts\n.pt + .pkl files")]:::storage
CFG2["Runtime Config\n(API_KEY, MODEL_PATH)"]:::cloud

API -->|"calls"| PRED
API -->|"get_tweets"| FETCH
API -->|"uses"| PREU

PRED -->|"loads/uses"| MOD
PRED -.->|"optional_alternate_loader"| MODC
MOD -->|"runs_inference"| SENT
PRED -->|"vectorizes"| VEC
PRED -->|"predicts_stats"| REG

MOD -->|"reads"| ARTS2
VEC -->|"loads"| ARTS2
REG -->|"loads"| ARTS2
API -->|"reads_env"| CFG2

FETCH -->|"optional_external_calls"| X
end

%% ---- Offline ML Pipeline (separate lane) ----
subgraph "Offline: Data + Training Pipeline (Batch / Not serving path)"
direction TB
OFFENV["Offline Training Environment\n(Local / Colab)"]:::offline

subgraph "1) Data Ingestion"
direction TB
SCRAPE["Scraping Job\n(scrape_tabasco_1000.py)"]:::offline
COOK["Cookie/Session Helper\n(save_cookies.py)"]:::offline
end

subgraph "2) Preprocessing / Dataset Build"
direction TB
PREP1["Preprocess Tweets\n(preprocess_tweets.py)"]:::offline
PREP2["Transform Utilities\n(data_preprocessing.py)"]:::offline
MERGE["Merge/Consolidate CSVs\n(merge_csvs.py)"]:::offline
TEMP["Batch Workflow Script\n(temp_all.py)"]:::offline
end

subgraph "3) Training / Fine-tuning"
direction TB
FT["Fine-tune RoBERTa\n(fine_tune_roberta.py)"]:::offline
FTCOL["Colab Fine-tune Variant\n(colab_fine_tune_roberta.py)"]:::offline

subgraph "Archived Experiments\n(archive/)"
direction TB
ARCHDIR["Archive Directory\n(archive/)"]:::offline
AR_TR["Train Regressor\n(archive/train_regressor.py)"]:::offline
AR_CONT["Continue Fine-tune\n(archive/continue_finetune_roberta.py)"]:::offline
AR_SCAL["Create Target Scaler\n(archive/create_target_scaler.py)"]:::offline
AR_EVAL["Evaluate Model\n(archive/evaluate_model.py)"]:::offline
AR_ANALYZE["Analyze Model\n(archive/analyze_model.py)"]:::offline
AR_MAIN["Archive Main\n(archive/main.py)"]:::offline
AR_SAVE["Save Fine-tuned RoBERTa\n(archive/save_finetuned_roberta.py)"]:::offline
end
end

subgraph "4) Evaluation / Validation"
direction TB
EVAL["Evaluate Model\n(evaluate_model.py)"]:::offline
VAL["Validate Model\n(validate_model.py)"]:::offline
ERR["Analyze Errors\n(analyze_errors.py)"]:::offline
EOUT[("Evaluation Outputs\n(evaluation_results/)")]:::storage
end

subgraph "Artifacts Produced"
direction TB
OUTART[("Trained Artifacts\nfinetuned_*.pt + tfidf_vectorizer.pkl + *_model.pkl")]:::storage
end

OFFENV --> SCRAPE
OFFENV --> PREP1
OFFENV --> FT
OFFENV --> EVAL

COOK --> SCRAPE
SCRAPE -->|"raw_tweets"| PREP1
PREP1 --> PREP2
PREP2 --> MERGE
MERGE --> FT
TEMP --> MERGE

FT -->|"checkpoint"| OUTART
FTCOL -->|"checkpoint"| OUTART
ARCHDIR -->|"experimental_models"| OUTART

FT --> EVAL
EVAL --> EOUT
VAL --> EOUT
ERR --> EOUT

end

%% ---- Offline -> Online linkage (dashed) ----
OUTART -.->|"publish/copy_to_artifact_store"| ARTS2
end

%% =================
%% CLICK EVENTS
%% =================
click API "https://github.com/aaravkataria24/xtesting-backend/blob/main/app.py"
click PRED "https://github.com/aaravkataria24/xtesting-backend/blob/main/predict.py"
click MOD "https://github.com/aaravkataria24/xtesting-backend/blob/main/model.py"
click MODC "https://github.com/aaravkataria24/xtesting-backend/blob/main/model_colab.py"

click DOCKER "https://github.com/aaravkataria24/xtesting-backend/tree/main/Dockerfile"
click REQS "https://github.com/aaravkataria24/xtesting-backend/blob/main/requirements.txt"

click SCRAPE "https://github.com/aaravkataria24/xtesting-backend/blob/main/scrape_tabasco_1000.py"
click COOK "https://github.com/aaravkataria24/xtesting-backend/blob/main/save_cookies.py"
click PREP1 "https://github.com/aaravkataria24/xtesting-backend/blob/main/preprocess_tweets.py"
click PREP2 "https://github.com/aaravkataria24/xtesting-backend/blob/main/data_preprocessing.py"
click MERGE "https://github.com/aaravkataria24/xtesting-backend/blob/main/merge_csvs.py"
click FT "https://github.com/aaravkataria24/xtesting-backend/blob/main/fine_tune_roberta.py"
click FTCOL "https://github.com/aaravkataria24/xtesting-backend/blob/main/colab_fine_tune_roberta.py"
click EVAL "https://github.com/aaravkataria24/xtesting-backend/blob/main/evaluate_model.py"
click VAL "https://github.com/aaravkataria24/xtesting-backend/blob/main/validate_model.py"
click ERR "https://github.com/aaravkataria24/xtesting-backend/blob/main/analyze_errors.py"
click EOUT "https://github.com/aaravkataria24/xtesting-backend/tree/main/evaluation_results/"
click ARCHDIR "https://github.com/aaravkataria24/xtesting-backend/tree/main/archive/"
click TEMP "https://github.com/aaravkataria24/xtesting-backend/blob/main/temp_all.py"

click AR_TR "https://github.com/aaravkataria24/xtesting-backend/blob/main/archive/train_regressor.py"
click AR_CONT "https://github.com/aaravkataria24/xtesting-backend/blob/main/archive/continue_finetune_roberta.py"
click AR_SCAL "https://github.com/aaravkataria24/xtesting-backend/blob/main/archive/create_target_scaler.py"
click AR_EVAL "https://github.com/aaravkataria24/xtesting-backend/blob/main/archive/evaluate_model.py"
click AR_ANALYZE "https://github.com/aaravkataria24/xtesting-backend/blob/main/archive/analyze_model.py"
click AR_MAIN "https://github.com/aaravkataria24/xtesting-backend/blob/main/archive/main.py"
click AR_SAVE "https://github.com/aaravkataria24/xtesting-backend/blob/main/archive/save_finetuned_roberta.py"

%% =================
%% STYLES
%% =================
classDef runtime fill:#2b6cb0,stroke:#1a365d,color:#ffffff,stroke-width:1.5px
classDef ml fill:#6b46c1,stroke:#3c1a7a,color:#ffffff,stroke-width:1.5px
classDef storage fill:#dd6b20,stroke:#7b341e,color:#ffffff,stroke-width:1.5px
classDef external fill:#4a5568,stroke:#1a202c,color:#ffffff,stroke-width:1.5px
classDef cloud fill:#718096,stroke:#2d3748,color:#ffffff,stroke-width:1.5px
classDef offline fill:#9b2c2c,stroke:#63171b,color:#ffffff,stroke-width:1.5px
classDef util fill:#2f855a,stroke:#1c4532,color:#ffffff,stroke-width:1.5px
classDef build fill:#2c5282,stroke:#1a365d,color:#ffffff,stroke-width:1.5px
```
