How stadiello/article_extractor works
Project type and purpose
• This is a single-page Streamlit web application (“article_extractor”) that scrapes web articles, feeds them into an AI model, and returns answers to a predefined list of questions.
• Key indicators:
• src/extractor/main.py – Streamlit entry point
• src/extractor/scrap.py – Selenium-based scraping
• src/extractor/bot.py – Ollama AI calls
• data/questions.txt – the question set
File structure
• Top-level:
• data/ – static resource (questions.txt)
• src/extractor/ – core application modules
• tests/ – unit-test harness
• Entry point: main.py
• Third-party integrations: Selenium (in scrap.py), Ollama (in bot.py), Streamlit (in main.py)
• Config: pyproject.toml for dependencies
README for additional insights
• Usage: streamlit run src/extractor/main.py
• AI model: Ollama’s deepseek-r1:8b
• Scraping: Selenium WebDriver
• No external database—questions are read from a plain text file
Detailed breakdown
• Browser/Client (blue)
– “Connect to http://localhost:8501”
↓
• Streamlit Server (green rectangle)
– Entry point: main.py
– Reads questions.txt
– Orchestrates two calls:
→ Scraper
→ Bot
• Scraper Module (green rectangle)
– scrap.py uses Selenium WebDriver
– Fetches and cleans HTML/text from target URL
• Bot Module (green rectangle)
– bot.py invokes Ollama CLI/API with model deepseek-r1:8b
– Input: scraped text + questions
– Output: structured answers
– data/questions.txt
• External Website (orange cloud)
– Any URL to be scraped
• Ollama Inference Engine (orange cloud)
– Local or remote Ollama server running the model
↓
• Streamlit renders answers → Browser