# imdibr/adshield_tn architecture

> Architecture diagram of the GitHub repository imdibr/adshield_tn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/imdibr/adshield_tn
- Repository: https://github.com/imdibr/adshield_tn

## Overview

##### Project type and purpose
- This is a data-driven ML pipeline (“Adshield”) for real-time scam-ad detection and user profiling on social media.
- It scrapes ads & user data from Facebook, Twitter, Instagram; stores & preprocesses it; trains ML models; serves live inference; and issues alerts.

##### File structure
- Top level:
- data/ (raw CSVs)
- model_training/ (training & evaluation scripts)
- src_userprofilling/ (all ETL, scraping, preprocessing, inference code)
- config.ini, requirements.txt (env & dependency setup)
- test_data.json (small sample)
- Scraper modules: twitter_scrapping.py, facebook_userprofiling.py, scrap_instgram_userdata.py, scrappy.py
- Pipeline orchestration: src_userprofilling/main.py, userprofiling.py
- Modeling: src_userprofilling/model.py, modelscamllm.py, scam_detection_text_bert.py
- Training: model_training/model_training.py, model_eval.py

##### Key architectural insights from README
- Data collection via APIs & scraping
- Storage in MongoDB/PostgreSQL
- ML techniques: Random Forest, Gradient Boosting, neural nets, BERT/RoBERTa
- Real-time alerting via WebSockets, Kafka/RabbitMQ

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        FB["Facebook API"]:::external
        TWAPI["Twitter API"]:::external
        TWSCR["Twitter Scraper"]:::external
        IG["Instagram Web Scraper"]:::external
    end

    %% Ingestion Layer
    subgraph "Ingestion Layer"
        ORCH["Scraper Orchestration"]:::compute
        PIPE["User Profiling Pipeline"]:::compute
    end

    %% Messaging Layer
    MessageBroker["Kafka/RabbitMQ"]:::message

    %% Storage Layer
    subgraph "Storage Layer"
        RawStore["MongoDB"]:::database
        ProcStore["PostgreSQL"]:::database
    end

    %% Configuration
    Config["config.ini"]:::config

    %% Data Processing & Feature Engineering
    subgraph "Data Processing & Feature Engineering"
        ETL["ETL Pipeline (main & userprofiling)"]:::compute
        NLP["NLP Feature Extractor"]:::compute
        LLM["LLM-based Feature Extractor"]:::compute
        ModelIF["Core Model Interface"]:::compute
    end

    %% Model Training
    subgraph "Offline Model Training"
        Train["Offline Model Training"]:::compute
        Eval["Model Evaluation"]:::compute
        AuxScrap["Auxiliary Scraper"]:::compute
        Registry["Model Registry"]:::artifact
    end

    %% Model Serving & Inference
    PredictionService["Prediction Service"]:::compute

    %% Real-Time Alerting
    NotificationService["Real-Time Alerting Service"]:::compute
    Client["Dashboard / Browser Extension"]:::client

    %% Test Data & Dependencies
    TestRoot["test_data.json"]:::config
    TestTrain["model_training/test_data.json"]:::config
    Req["requirements.txt"]:::config

    %% Connections
    FB -->|collects data| ORCH
    TWAPI -->|collects data| ORCH
    TWSCR -->|collects data| ORCH
    IG -->|collects data| ORCH

    ORCH -->|publishes raw| MessageBroker
    PIPE -->|publishes raw| MessageBroker

    MessageBroker -->|writes raw| RawStore
    RawStore -->|reads raw| ETL
    ETL -->|writes processed| ProcStore

    ProcStore -->|source for training| Train
    ProcStore -->|source for evaluation| Eval
    AuxScrap -->|generates training data| ProcStore

    Train -->|stores model| Registry
    Eval -->|updates metrics| Registry

    Registry -->|latest model| PredictionService
    MessageBroker -->|streams incoming| PredictionService
    PredictionService -->|predictions| NotificationService
    NotificationService -->|alerts via WebSocket| Client

    %% Test Data & Dependencies Links
    TestRoot --> ETL
    TestTrain --> Eval
    Req -->|env dependencies| ORCH

    %% Click Events
    click FB "https://github.com/imdibr/adshield_tn/blob/main/src_userprofilling/facebook_userprofiling.py"
    click TWAPI "https://github.com/imdibr/adshield_tn/blob/main/src_userprofilling/twitter_api_.py"
    click TWSCR "https://github.com/imdibr/adshield_tn/blob/main/src_userprofilling/twitter_scrapping.py"
    click IG "https://github.com/imdibr/adshield_tn/blob/main/src_userprofilling/scrap_instgram_userdata.py"
    click ORCH "https://github.com/imdibr/adshield_tn/blob/main/src_userprofilling/main.py"
    click PIPE "https://github.com/imdibr/adshield_tn/blob/main/src_userprofilling/userprofiling.py"
    click NLP "https://github.com/imdibr/adshield_tn/blob/main/src_userprofilling/scam_detection_text_bert.py"
    click LLM "https://github.com/imdibr/adshield_tn/blob/main/src_userprofilling/modelscamllm.py"
    click ModelIF "https://github.com/imdibr/adshield_tn/blob/main/src_userprofilling/model.py"
    click NotificationService "https://github.com/imdibr/adshield_tn/blob/main/src_userprofilling/mobilescamalert.py"
    click Train "https://github.com/imdibr/adshield_tn/blob/main/model_training/model_training.py"
    click Eval "https://github.com/imdibr/adshield_tn/blob/main/model_training/model_eval.py"
    click AuxScrap "https://github.com/imdibr/adshield_tn/blob/main/model_training/scrappy.py"
    click TestRoot "https://github.com/imdibr/adshield_tn/blob/main/test_data.json"
    click TestTrain "https://github.com/imdibr/adshield_tn/blob/main/model_training/test_data.json"
    click Req "https://github.com/imdibr/adshield_tn/blob/main/requirements.txt"
    click Config "https://github.com/imdibr/adshield_tn/blob/main/config.ini"

    %% Styles
    classDef external fill:#f2f2f2,stroke:#333,stroke-width:1px
    classDef compute fill:#9f6,stroke:#333,stroke-width:1px
    classDef database fill:#6cf,stroke:#333,stroke-width:1px
    classDef message fill:#fc6,stroke:#333,stroke-width:1px
    classDef artifact fill:#8df,stroke:#333,stroke-width:1px
    classDef config fill:#ccc,stroke:#333,stroke-width:1px
    classDef client fill:#eef,stroke:#333,stroke-width:1px
```
