# anduriroshan/carer-med-scraper architecture

> Architecture diagram of the GitHub repository anduriroshan/carer-med-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anduriroshan/carer-med-scraper
- Repository: https://github.com/anduriroshan/carer-med-scraper

## Overview

##### Main Components
- Entry Point & Controller: The main entry is likely “app.py”, which (most probably) orchestrates the scraping operations, initializes modules, and possibly handles scheduling or routing.
- Crawling Modules:
- Article Crawlers: Contained in the “crawl_article” directory, with various Python scripts (e.g., crawl_article_allergy_sql.py, crawl_article_cardiology_sql.py, etc.) that handle scraping article data for each medical specialty.
- Page Crawlers: Found in the “crawl_page” directory, with a similar structure but designed for scraping page-based information.
- Utility Components: There are several “utils.py” files (in the top-level, crawl_article, and crawl_page directories) that likely provide common functionality to process and abstract the underlying scraping logic.
- Data/Processing Components: Files like “merging_collections.py”, “update.py”, and “query_search.py” indicate components that handle post-scraping data operations such as merging scraped data collections, updating existing data, and facilitating search queries.
- Auxiliary/Advanced Functionality: The “rag.py” file suggests there might be an integration with advanced processing such as a retrieval-augmented generation approach for content or search enhancement.
- Constants: “constanst.py” is likely used to store configuration or constant values used across the application.

##### System Interactions & Data Flow
- The flow likely begins at the main controller (app.py), which may determine what specialty or type of content to scrape.
- The controller routes the request to the appropriate crawling module:
- Article scraping modules (from the crawl_article directory) will be invoked when the system needs to scrape article-based content for a specific medical specialty.
- Similarly, page scraping modules (from the crawl_page directory) are called for page-based contents.
- Each of these modules uses common utility functions to handle tasks like HTTP requests, parsing, and error handling.
- After crawling, the data likely flows into a processing pipeline where merging (merging_collections.py) and updates (update.py) are performed. The query_search.py module might interface with either an internal index or an external search service.
- The advanced “rag.py” module, if integrated, might process the scraped data to enhance search relevance or content summarization.

##### Architectural Patterns and Design Principles
- Separation of Concerns: The clear split into article versus page crawler directories indicates a modular design where each component concerns itself with a distinct functionality.
- Reusability: The presence of multiple “utils.py” files in different directories suggests that common functionality is abstracted for reuse across various modules.
- Scalability: With each medical specialty having its own module, it’s straightforward to add new specialties or modify existing ones without disrupting the rest of the system.
- Maintainability: Use of a main entry point (app.py) to orchestrate actions and constants defined in “constanst.py” helps centralize configuration and keep the code maintainable.

## Diagram (Mermaid)

```mermaid
flowchart TB
    A["Main Application / Controller (app.py)"]:::controller
    B["Constants (constanst.py)"]:::config

    subgraph "Article Scraper Components"
        A1["Allergy Scraper (crawl_article/crawl_article_allergy_sql.py)"]:::scraper
        A2["Cardiology Scraper (crawl_article/crawl_article_cardiology_sql.py)"]:::scraper
        AUtil["Article Utilities (crawl_article/utils.py)"]:::utility
    end

    subgraph "Page Scraper Components"
        P1["Allergy Page Scraper (crawl_page/crawl_page_allergy.py)"]:::scraper
        P2["Cardiology Page Scraper (crawl_page/crawl_page_cardiology.py)"]:::scraper
        PUtil["Page Utilities (crawl_page/utils.py)"]:::utility
    end

    C["Shared Utilities (utils.py)"]:::utility

    subgraph "Data Processing Components"
        D1["Merging Collections (merging_collections.py)"]:::processing
        D2["Update Process (update.py)"]:::processing
        D3["Query Search (query_search.py)"]:::processing
        D4["Retrieval Augmented Generation (rag.py)"]:::processing
    end

    DStore["Data Storage (Database)"]:::database

    %% Connections from Main Application
    A -->|"reads"| B
    A -->|"executes"| A1
    A -->|"executes"| A2
    A -->|"executes"| P1
    A -->|"executes"| P2

    %% Connections within Article Scraper Components
    A1 -->|"uses"| AUtil
    A2 -->|"uses"| AUtil

    %% Connections within Page Scraper Components
    P1 -->|"uses"| PUtil
    P2 -->|"uses"| PUtil

    %% Linking Utilities to Shared Utilities
    AUtil -->|"aggregates"| C
    PUtil -->|"aggregates"| C

    %% Data flow from Crawlers to Data Processing
    A1 -->|"sends_data"| D1
    A2 -->|"sends_data"| D1
    P1 -->|"sends_data"| D1
    P2 -->|"sends_data"| D1

    %% Shared Utilities supporting Processing
    C -->|"supports"| D2

    %% Data Processing Components feeding into Data Storage
    D1 -->|"feeds"| DStore
    D2 -->|"feeds"| DStore
    D3 -->|"feeds"| DStore
    D4 -->|"feeds"| DStore

    %% Styling
    classDef controller fill:#ffaaaa,stroke:#990000,stroke-width:2px;
    classDef config fill:#ffeeaa,stroke:#aa6600,stroke-width:2px;
    classDef scraper fill:#aaffaa,stroke:#009900,stroke-width:2px;
    classDef utility fill:#aaaaff,stroke:#000099,stroke-width:2px;
    classDef processing fill:#ffffaa,stroke:#999900,stroke-width:2px;
    classDef database fill:#ffddaa,stroke:#aa5500,stroke-width:2px;

    %% Click Events
    click A "https://github.com/anduriroshan/carer-med-scraper/blob/main/app.py"
    click B "https://github.com/anduriroshan/carer-med-scraper/blob/main/constanst.py"
    click A1 "https://github.com/anduriroshan/carer-med-scraper/blob/main/crawl_article/crawl_article_allergy_sql.py"
    click A2 "https://github.com/anduriroshan/carer-med-scraper/blob/main/crawl_article/crawl_article_cardiology_sql.py"
    click AUtil "https://github.com/anduriroshan/carer-med-scraper/blob/main/crawl_article/utils.py"
    click P1 "https://github.com/anduriroshan/carer-med-scraper/blob/main/crawl_page/crawl_page_allergy.py"
    click P2 "https://github.com/anduriroshan/carer-med-scraper/blob/main/crawl_page/crawl_page_cardiology.py"
    click PUtil "https://github.com/anduriroshan/carer-med-scraper/blob/main/crawl_page/utils.py"
    click C "https://github.com/anduriroshan/carer-med-scraper/blob/main/utils.py"
    click D1 "https://github.com/anduriroshan/carer-med-scraper/blob/main/merging_collections.py"
    click D2 "https://github.com/anduriroshan/carer-med-scraper/blob/main/update.py"
    click D3 "https://github.com/anduriroshan/carer-med-scraper/blob/main/query_search.py"
    click D4 "https://github.com/anduriroshan/carer-med-scraper/blob/main/rag.py"
```
