# yeshapan/website-categorizer architecture

> Architecture diagram of the GitHub repository yeshapan/website-categorizer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yeshapan/website-categorizer
- Repository: https://github.com/yeshapan/website-categorizer

## Overview

##### Project type and purpose
- This is a lightweight, full-stack Python web application that uses machine learning to classify arbitrary website URLs into categories.
- The frontend is built with Streamlit (app/app.py), and the “backend” consists of scraping, preprocessing, and a pre-trained Logistic Regression model (categorizer/).

##### File structure
- Top-level files:
- README.md, requirements.txt, pyproject.toml – project metadata and dependencies
- main.py – (optionally) an entry point or orchestrator
- websites_dataset.csv – sample data used during model training
- app/ directory – the Streamlit UI (app.py)
- categorizer/ directory – core ML pipeline:
- scraper.py – fetches and extracts text from the target website
- preprocess.py – cleans and stems the raw text
- dataset.py – (used for initial dataset handling)
- model.py – loads the vectorizer and classifier (.joblib artifacts) and runs inference
- website_vectorizer.joblib & website_categorizer_model.joblib – persisted scikit-learn artifacts

##### README for additional insights
- The README confirms the tech stack: Streamlit frontend + Logistic Regression classifier
- It describes the data flow: URL → scrape → clean → vectorize → predict → display

##### Main components and their relationships
a. Components:
1) Streamlit UI (app/app.py)
2) Scraper (categorizer/scraper.py)
3) Preprocessor (categorizer/preprocess.py)
4) Vectorizer + ML Model (categorizer/model.py + .joblib files)
5) External website (HTTP source of HTML/text)
b. Interactions:
- User enters URL in the Streamlit UI
↓
- UI calls the Scraper to fetch and parse text
↓
- Raw text passes to the Preprocessor (cleaning, stemming)
↓
- Preprocessed tokens feed into the Vectorizer, then to Logistic Regression for prediction
↓
- Predicted category and a preview of scraped text are returned to the UI
c. Architectural patterns:
- Monolithic, single-process Python app
- Pipeline pattern within the categorizer module (scrape → preprocess → vectorize → predict)
- Separation of concerns: UI vs. core logic vs. persisted artifacts
d. Key technologies:
- Python 3.x, Streamlit for UI
- requests + BeautifulSoup (in scraper.py) for HTML extraction
- scikit-learn for vectorization and classification (joblib for model persistence)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External service
    External["External Website"]:::external

    %% UI Layer
    UI["Streamlit UI<br>(app/app.py)"]:::ui
    click UI "https://github.com/yeshapan/website-categorizer/blob/main/app/app.py"

    %% Core Pipeline
    Scraper["Scraper<br>(categorizer/scraper.py)"]:::core
    click Scraper "https://github.com/yeshapan/website-categorizer/blob/main/categorizer/scraper.py"
    Preprocessor["Preprocessor<br>(categorizer/preprocess.py)"]:::core
    click Preprocessor "https://github.com/yeshapan/website-categorizer/blob/main/categorizer/preprocess.py"
    Model["Vectorizer & Model<br>(categorizer/model.py)"]:::core
    click Model "https://github.com/yeshapan/website-categorizer/blob/main/categorizer/model.py"

    %% Persisted Artifacts
    Joblib1["website_vectorizer.joblib"]:::storage
    click Joblib1 "https://github.com/yeshapan/website-categorizer/blob/main/categorizer/website_vectorizer.joblib"
    Joblib2["website_categorizer_model.joblib"]:::storage
    click Joblib2 "https://github.com/yeshapan/website-categorizer/blob/main/categorizer/website_categorizer_model.joblib"

    %% Entry Point
    Entry["Entry Point<br>(main.py)"]:::core
    click Entry "https://github.com/yeshapan/website-categorizer/blob/main/main.py"

    %% Training Subgraph
    subgraph "Training Pipeline"
        Dataset["Dataset Handler<br>(categorizer/dataset.py)"]:::storage
        click Dataset "https://github.com/yeshapan/website-categorizer/blob/main/categorizer/dataset.py"
        Dataset -->|"builds"| Joblib1
        Dataset -->|"builds"| Joblib2
    end

    %% Runtime Flow
    UI -->|"enters URL"| Scraper
    Scraper -->|"HTTP GET raw HTML"| External
    External -->|"raw HTML"| Scraper
    Scraper -->|"raw text"| Preprocessor
    Preprocessor -->|"clean text"| Model
    Model -->|"predicted category & preview"| UI

    %% Optional orchestrator path
    Entry --> UI

    %% Styles
    classDef ui fill:#D6E4FF,stroke:#0366d6,color:#0366d6;
    classDef core fill:#E2F0D9,stroke:#2e7d32,color:#2e7d32;
    classDef external fill:#F2F2F2,stroke:#6a737d,color:#6a737d;
    classDef storage fill:#FFF2CC,stroke:#b08900,color:#b08900;
```
