# geshan/scraper-with-llm-streamlit architecture

> Architecture diagram of the GitHub repository geshan/scraper-with-llm-streamlit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/geshan/scraper-with-llm-streamlit
- Repository: https://github.com/geshan/scraper-with-llm-streamlit

## Overview

##### Main Components Identification

a) Web Scraping Layer
- Beautiful Soup component
- URL handling and HTTP requests
- Raw data extraction

b) LLM Processing Layer
- Gemini API integration
- Ollama/Gemma integration via LiteLLM
- Text processing and transformation

c) Presentation Layer
- Streamlit interface
- Results display
- User interaction

**Websites → Scraper → LLM Processing → Streamlit Interface**

##### Specific Components

- Target Websites (External)
- Beautiful Soup Scraper (scraper.py)
- LLM Services:
- Gemini (app-gemini.py)
- Ollama/Gemma via LiteLLM (app-ollama-litellm.py)
- Streamlit Interface
- Docker Container (shown as a boundary)

##### Additional Elements

- Docker container boundary (shown by Dockerfile and docker-compose.yml)
- Input/Output formats at each stage
- API endpoints and connections

##### Notes for Implementation Details

Annotations for:
- Beautiful Soup scraping logic
- LLM processing steps
- Different app versions (Gemini vs. Ollama)
- Streamlit interface features

- The pipeline nature of the application
- The modularity between scraping, LLM processing, and presentation
- The containerized deployment structure
- The ability to switch between different LLM providers

Keep the abstraction level focused on the main components while showing the key interactions between the scraper, LLM services, and the user interface. This will help developers understand both the high-level architecture and the specific implementation details of this modern scraping solution.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph External["External Services"]
        direction TB
        Websites["Target Websites"]:::external
        GeminiAPI["Google Gemini API"]:::external
        OllamaAPI["Ollama API"]:::external
    end

    subgraph Processing["Processing Layer"]
        direction TB
        Scraper["Beautiful Soup Scraper"]:::processing
        
        subgraph LLM["LLM Processing"]
            direction TB
            GeminiApp["Gemini Processor"]:::llm
            OllamaApp["Ollama/LiteLLM Processor"]:::llm
        end
    end

    subgraph Interface["Interface Layer"]
        StreamlitUI["Streamlit Interface"]:::ui
    end

    subgraph Infrastructure["Docker Infrastructure"]
        direction TB
        Docker["Dockerfile"]:::infra
        Compose["Docker Compose"]:::infra
        DockerIgnore[".dockerIgnore"]:::infra
        Requirements["Requirements"]:::infra
    end

    %% Relationships
    Websites --> Scraper
    Scraper --> GeminiApp
    Scraper --> OllamaApp
    GeminiApp --> StreamlitUI
    OllamaApp --> StreamlitUI
    GeminiApp <--> GeminiAPI
    OllamaApp <--> OllamaAPI

    %% Click Events
    click Scraper "https://github.com/geshan/scraper-with-llm-streamlit/blob/master/scraper.py"
    click GeminiApp "https://github.com/geshan/scraper-with-llm-streamlit/blob/master/app-gemini.py"
    click OllamaApp "https://github.com/geshan/scraper-with-llm-streamlit/blob/master/app-ollama-litellm.py"
    click StreamlitUI "https://github.com/geshan/scraper-with-llm-streamlit/blob/master/app-ollama-litellm-st.py"
    click Docker "https://github.com/geshan/scraper-with-llm-streamlit/tree/master/Dockerfile"
    click Compose "https://github.com/geshan/scraper-with-llm-streamlit/blob/master/docker-compose.yml"
    click DockerIgnore "https://github.com/geshan/scraper-with-llm-streamlit/blob/master/.dockerignore"
    click Requirements "https://github.com/geshan/scraper-with-llm-streamlit/blob/master/requirements.txt"

    %% Styling
    classDef external fill:#ADD8E6,stroke:#333
    classDef processing fill:#90EE90,stroke:#333
    classDef llm fill:#FFB6C1,stroke:#333
    classDef ui fill:#FFA500,stroke:#333
    classDef infra fill:#D3D3D3,stroke:#333

    %% Legend
    subgraph Legend
        direction TB
        E[External Services]:::external
        P[Processing Components]:::processing
        L[LLM Services]:::llm
        U[User Interface]:::ui
        I[Infrastructure]:::infra
    end
```
