# karthik-md07/ai-web-vuln_scanner architecture

> Architecture diagram of the GitHub repository karthik-md07/ai-web-vuln_scanner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/karthik-md07/ai-web-vuln_scanner
- Repository: https://github.com/karthik-md07/ai-web-vuln_scanner

## Overview

##### What this project is (type + purpose)
This is a **Django monolith web application** that provides an **AI-assisted web vulnerability scanning service**. Users submit a target URL via a web UI, the server crawls/pages the target, runs a set of vulnerability checks, applies an **ML model (Random Forest)** to classify/score risk, stores scan results/history in the DB, and generates a **PDF report**.

- **Web UI (templates + JS)** → **Django web app** → **Scanning/ML/PDF subsystems** → **DB + files** (+ **external target websites**)

##### Core Django project
- `manage.py` — entry point
- `config/` — Django project config: settings, URLs, WSGI
- `config/settings.py` — installed apps, DB config (sqlite), static/templates config
- `config/urls.py` — root routing (likely includes `scanner/urls.py`)

##### Django apps (important: there appear to be two)
- `scanner/` — looks like the primary app containing the actual scanner logic:
- `views.py`, `urls.py` — HTTP endpoints / request handling
- `models.py` — persistence for scans/results/history
- `crawler.py` — discovers pages/links on the target
- `scanners.py` — likely rule-based vulnerability checks (the 5 vuln types)
- `ml_engine.py` — feature extraction + model inference
- `pdf_report.py` — PDF generation (ReportLab)
- `scanner/ml_data/training_data.csv` — training dataset (not runtime-critical unless retraining)
- `scanners/` — a second Django app (name is confusingly close to `scanner/`):
- `apps.py`, `models.py`, `views.py`, `tests.py`

##### Data/artifacts
- `db.sqlite3` — runtime persistence
- `ml_data/vuln_model.pkl` — packaged trained model artifact used by ML engine
- `.env` — environment config (secrets, toggles)
- `templates/dashboard.html` — UI entry point

##### User Browser
- Loads `dashboard.html`
- Submits target URL (form POST or AJAX)
##### Django Routing
- `config/urls.py` → `scanner/urls.py` → `scanner/views.py`
##### Scan Orchestration (inside scanner app)
- `scanner/views.py` calls into scanning pipeline, typically:
- `crawler.py` to fetch and enumerate pages/links
- `scanners.py` to run vulnerability checks per URL/page
- `ml_engine.py` to compute risk score / classify vulnerabilities using `ml_data/vuln_model.pkl`
##### Persistence
- `scanner/models.py` writes Scan + Findings + Score + Timestamps to `db.sqlite3`
##### Response
- Returns JSON/HTML with results and risk score, updates UI

##### External dependency boundary
- All crawling/scanning uses:
- `requests` (HTTP)
- `BeautifulSoup4` (HTML parsing)
- The **target website(s)** are external systems and must be drawn as a separate boundary.

##### Architectural patterns/design principles to emphasize (specific to this project)
- **Monolithic Django architecture**: single deployable unit; internal modules separate concerns.
- **Pipeline pattern** for scanning:
- Crawl → Detect (rules) → Score (ML) → Persist → Report
- **Separation of concerns**:
- Views/URLs handle HTTP
- Dedicated modules for crawler/scanners/ML/PDF
- Models isolate DB schema

If there’s “real-time scanning”, represent it as:
- synchronous request (browser waits) **or**
- pseudo-real-time updates (polling endpoints / AJAX)
Since no Celery/Redis is visible, assume **synchronous + frontend updates**, unless you confirm otherwise.

- Browser → view → crawler → scanners → ml_engine → models → view → Browser
- `Scan`, `Finding`, `Report` etc. (names to be confirmed by reading the file)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% =========================================================
    %% Level 1: System Context / Container View (vertical)
    %% =========================================================
    U["User Browser<br/>(UI: dashboard + JS)"]:::external

    subgraph "Your System Boundary (Django Monolith)"
        M["Django Entry Point<br/>(manage.py)"]:::runtime
        subgraph "Django Project: config/"
            CFGS["Settings<br/>(config/settings.py)"]:::config
            CFGU["Root URLs<br/>(config/urls.py)"]:::config
            CFGW["WSGI App<br/>(config/wsgi.py)"]:::config
        end

        subgraph "Primary App: scanner/"
            SURL["URLConf<br/>(scanner/urls.py)"]:::django
            SVIEW["HTTP Endpoints / Controllers<br/>(scanner/views.py)"]:::django

            ORCH["Scan Orchestrator<br/>(pipeline: crawl→detect→score→persist)"]:::service

            CRAWL["Crawler<br/>(scanner/crawler.py)"]:::service
            DETECT["Rule-based Vulnerability Checks<br/>(scanner/scanners.py)"]:::service
            ML["ML Inference / Risk Scoring<br/>(scanner/ml_engine.py)"]:::ml
            PDF["PDF Generator (ReportLab)<br/>(scanner/pdf_report.py)"]:::report

            MODEL["Persistence Models<br/>(scanner/models.py)"]:::data
            MIG["DB Migrations<br/>(scanner/migrations/)"]:::data
        end

        subgraph "Secondary App: scanners/ (optional/legacy?)"
            S2APP["AppConfig<br/>(scanners/apps.py)"]:::optional
            S2VIEW["Views<br/>(scanners/views.py)"]:::optional
            S2MODEL["Models<br/>(scanners/models.py)"]:::optional
            S2TEST["Tests<br/>(scanners/tests.py)"]:::optional
        end

        ENV["Environment Config<br/>(.env)"]:::config

        subgraph "Local Data & Artifacts"
            DB[("SQLite DB<br/>(db.sqlite3)")]:::database
            PKL[("ML Model Artifact<br/>(ml_data/vuln_model.pkl)")]:::artifact
            CSV[("Training Dataset (offline)<br/>(scanner/ml_data/training_data.csv)")]:::artifact
            TPL["Template<br/>(templates/dashboard.html)"]:::frontend
        end
    end

    EXT["Target Website(s)<br/>(External Systems on Internet)"]:::external
    LIB["HTTP + HTML Parsing<br/>(requests, BeautifulSoup4)"]:::library

    %% ---------------------------------------------------------
    %% Context-level interactions
    %% ---------------------------------------------------------
    U -->|"GET dashboard / submit target URL / download PDF"| CFGU
    CFGU -->|"route to app"| SURL
    SURL -->|"dispatch request"| SVIEW
    SVIEW -->|"render UI"| TPL
    TPL -->|"form POST/AJAX: target URL"| SVIEW

    SVIEW -->|"start scan"| ORCH
    ORCH -->|"crawl target"| CRAWL
    CRAWL -->|"HTTP requests"| LIB
    LIB -->|"GET/POST to target"| EXT
    EXT -->|"HTML responses"| LIB
    LIB -->|"parsed HTML / links"| CRAWL
    CRAWL -->|"discovered URLs + page content"| ORCH

    ORCH -->|"run checks per page"| DETECT
    DETECT -->|"findings + features"| ORCH

    ORCH -->|"score/classify"| ML
    ML -->|"load model for inference"| PKL
    ML -->|"risk score + probabilities"| ORCH

    ORCH -->|"persist scan + findings + score"| MODEL
    MIG -.->|"schema evolution"| MODEL
    MODEL <-->|"DB read/write"| DB

    SVIEW -->|"fetch history/results"| MODEL
    MODEL -->|"query results"| SVIEW
    SVIEW -->|"JSON/HTML results"| U

    U -->|"Download PDF report"| SVIEW
    SVIEW -->|"build report from results"| PDF
    PDF -->|"read scan data"| MODEL
    PDF -->|"PDF bytes (application/pdf)"| SVIEW
    SVIEW -->|"stream PDF"| U

    CFGS -.->|"INSTALLED_APPS includes scanner + scanners?"| S2APP
    CSV -.->|"offline training input (not runtime-critical)"| ML

    ENV -.->|"secrets/toggles/config"| CFGS

    %% =========================================================
    %% Click Events (from component_mapping)
    %% =========================================================
    click M "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/manage.py"
    click CFGS "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/config/settings.py"
    click CFGU "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/config/urls.py"
    click CFGW "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/config/wsgi.py"
    click SURL "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanner/urls.py"
    click SVIEW "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanner/views.py"
    click MODEL "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanner/models.py"
    click MIG "https://github.com/karthik-md07/ai-web-vuln_scanner/tree/main/scanner/migrations/"
    click CRAWL "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanner/crawler.py"
    click DETECT "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanner/scanners.py"
    click ML "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanner/ml_engine.py"
    click PDF "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanner/pdf_report.py"
    click DB "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/db.sqlite3"
    click PKL "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/ml_data/vuln_model.pkl"
    click CSV "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanner/ml_data/training_data.csv"
    click TPL "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/templates/dashboard.html"
    click ENV "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/.env"
    click S2APP "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanners/apps.py"
    click S2VIEW "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanners/views.py"
    click S2MODEL "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanners/models.py"
    click S2TEST "https://github.com/karthik-md07/ai-web-vuln_scanner/blob/main/scanners/tests.py"

    %% =========================================================
    %% Styles (colors + shapes via classDefs)
    %% =========================================================
    classDef external fill:#2b2d42,stroke:#10121a,color:#ffffff,stroke-width:1.5px
    classDef frontend fill:#3a86ff,stroke:#1d4ed8,color:#ffffff,stroke-width:1.5px
    classDef django fill:#4361ee,stroke:#1e40af,color:#ffffff,stroke-width:1.5px
    classDef service fill:#4cc9f0,stroke:#0ea5b7,color:#06212a,stroke-width:1.5px
    classDef ml fill:#06d6a0,stroke:#047857,color:#06211a,stroke-width:1.5px
    classDef report fill:#ffd166,stroke:#b45309,color:#3b2200,stroke-width:1.5px
    classDef database fill:#9b5de5,stroke:#5b21b6,color:#ffffff,stroke-width:1.5px
    classDef artifact fill:#f15bb5,stroke:#9d174d,color:#ffffff,stroke-width:1.5px
    classDef config fill:#8d99ae,stroke:#334155,color:#0b1220,stroke-width:1.5px
    classDef runtime fill:#ef476f,stroke:#9f1239,color:#ffffff,stroke-width:1.5px
    classDef optional fill:#adb5bd,stroke:#6b7280,color:#111827,stroke-width:1.5px
    classDef library fill:#c7f9cc,stroke:#22c55e,color:#052e12,stroke-width:1.5px
```
