How yeshapan/website-categorizer works
Project type and purpose
• This is a lightweight, full-stack Python web application that uses machine learning to classify arbitrary website URLs into categories.
• The frontend is built with Streamlit (app/app.py), and the “backend” consists of scraping, preprocessing, and a pre-trained Logistic Regression model (categorizer/).
File structure
• Top-level files:
• README.md, requirements.txt, pyproject.toml – project metadata and dependencies
• main.py – (optionally) an entry point or orchestrator
• websites_dataset.csv – sample data used during model training
• app/ directory – the Streamlit UI (app.py)
• categorizer/ directory – core ML pipeline:
• scraper.py – fetches and extracts text from the target website
• preprocess.py – cleans and stems the raw text
• dataset.py – (used for initial dataset handling)
• model.py – loads the vectorizer and classifier (.joblib artifacts) and runs inference
• website_vectorizer.joblib & website_categorizer_model.joblib – persisted scikit-learn artifacts
README for additional insights
• The README confirms the tech stack: Streamlit frontend + Logistic Regression classifier
• It describes the data flow: URL → scrape → clean → vectorize → predict → display
Main components and their relationships
a. Components:
1) Streamlit UI (app/app.py)
2) Scraper (categorizer/scraper.py)
3) Preprocessor (categorizer/preprocess.py)
4) Vectorizer + ML Model (categorizer/model.py + .joblib files)
5) External website (HTTP source of HTML/text)
b. Interactions:
• User enters URL in the Streamlit UI
↓
• UI calls the Scraper to fetch and parse text
↓
• Raw text passes to the Preprocessor (cleaning, stemming)
↓
• Preprocessed tokens feed into the Vectorizer, then to Logistic Regression for prediction
↓
• Predicted category and a preview of scraped text are returned to the UI
c. Architectural patterns:
• Monolithic, single-process Python app
• Pipeline pattern within the categorizer module (scrape → preprocess → vectorize → predict)
• Separation of concerns: UI vs. core logic vs. persisted artifacts
d. Key technologies:
• Python 3.x, Streamlit for UI
• requests + BeautifulSoup (in scraper.py) for HTML extraction
• scikit-learn for vectorization and classification (joblib for model persistence)