# stadiello/article_extractor architecture

> Architecture diagram of the GitHub repository stadiello/article_extractor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stadiello/article_extractor
- Repository: https://github.com/stadiello/article_extractor

## Overview

##### Project type and purpose
- This is a single-page Streamlit web application (“article_extractor”) that scrapes web articles, feeds them into an AI model, and returns answers to a predefined list of questions.
- Key indicators:
- `src/extractor/main.py` – Streamlit entry point
- `src/extractor/scrap.py` – Selenium-based scraping
- `src/extractor/bot.py` – Ollama AI calls
- `data/questions.txt` – the question set

##### File structure
- Top-level:
- `data/` – static resource (questions.txt)
- `src/extractor/` – core application modules
- `tests/` – unit-test harness
- Entry point: `main.py`
- Third-party integrations: Selenium (in `scrap.py`), Ollama (in `bot.py`), Streamlit (in `main.py`)
- Config: `pyproject.toml` for dependencies

##### README for additional insights
- Usage: `streamlit run src/extractor/main.py`
- AI model: Ollama’s `deepseek-r1:8b`
- Scraping: Selenium WebDriver
- No external database—questions are read from a plain text file

##### Detailed breakdown
- Browser/Client (blue)
– “Connect to http://localhost:8501”
↓
- Streamlit Server (green rectangle)
– Entry point: `main.py`
– Reads `questions.txt`
– Orchestrates two calls:
→ Scraper
→ Bot
- Scraper Module (green rectangle)
– `scrap.py` uses Selenium WebDriver
– Fetches and cleans HTML/text from target URL
- Bot Module (green rectangle)
– `bot.py` invokes Ollama CLI/API with model `deepseek-r1:8b`
– Input: scraped text + questions
– Output: structured answers
– `data/questions.txt`
- External Website (orange cloud)
– Any URL to be scraped
- Ollama Inference Engine (orange cloud)
– Local or remote Ollama server running the model
↓
- Streamlit renders answers → Browser

## Diagram (Mermaid)

```mermaid
flowchart TB
    Client(["User Browser"]):::ui
    subgraph "App Backend"
        Streamlit["Streamlit Server\n(src/extractor/main.py)"]:::module
        Scraper["Scraper Module\n(src/extractor/scrap.py)"]:::module
        Bot["Bot Module\n(src/extractor/bot.py)"]:::module
        DataStore["Questions Data Store\n(data/questions.txt)"]:::datastore
        Config["Configuration\n(pyproject.toml)"]:::config
    end
    ExternalSite(("External Website")):::external
    Ollama(("Ollama Inference Engine")):::external
    TestHarness(["Unit Test Harness\n(tests/)"]):::external

    Client -->|"HTTP/WS"| Streamlit
    Streamlit -->|"Reads questions"| DataStore
    Streamlit -->|"Uses config"| Config
    Streamlit -->|"Calls Scraper"| Scraper
    Scraper -->|"WebDriver fetch"| ExternalSite
    Streamlit -->|"Calls Bot"| Bot
    Bot -->|"CLI/API"| Ollama
    Ollama -->|"Responses"| Bot
    Bot -->|"Returns answers"| Streamlit
    Streamlit -->|"Renders answers"| Client

    click Streamlit "https://github.com/stadiello/article_extractor/blob/main/src/extractor/main.py"
    click Scraper "https://github.com/stadiello/article_extractor/blob/main/src/extractor/scrap.py"
    click Bot "https://github.com/stadiello/article_extractor/blob/main/src/extractor/bot.py"
    click DataStore "https://github.com/stadiello/article_extractor/blob/main/data/questions.txt"
    click Config "https://github.com/stadiello/article_extractor/blob/main/pyproject.toml"
    click TestHarness "https://github.com/stadiello/article_extractor/tree/main/tests/"

    classDef ui fill:#D0E7FF,stroke:#023E8A,color:#03045E,stroke-width:2px,shape:ellipse
    classDef module fill:#D8F3DC,stroke:#2F5233,color:#1B4332,stroke-width:2px
    classDef datastore fill:#F0EFEF,stroke:#6C757D,color:#495057,stroke-width:2px,shape:hexagon
    classDef external fill:#FFE5D9,stroke:#E76F51,color:#9B2226,stroke-width:2px,shape:cloud
    classDef config fill:#FFF3B0,stroke:#E9C46A,color:#A57F46,stroke-width:2px,shape:rect
```
