# ait-rdp/rdp-data-crawler architecture

> Architecture diagram of the GitHub repository ait-rdp/rdp-data-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ait-rdp/rdp-data-crawler
- Repository: https://github.com/ait-rdp/rdp-data-crawler

## Overview

##### Project type and purpose
- This is a backend data‐crawler microservice written in Python, packaged as a Docker container.
- Its purpose is to read external data (forecasts, measurements) from a wide variety of sources (HTTP APIs, Modbus TCP, OPC UA, Redis streams), process it (polling or event‐driven), and write it into Redis streams or push it to external systems (Modbus, OPC UA).

##### File structure
- Top level:
- data_crawler/: the application package (entry point, CLI, plugin modules for “sources” and “sinks,” shared utilities).
- data/: example test data.
- docker/: Dockerfile and default configuration under /etc/data_crawler.
- docs/: installation and data‐source/sink specifications.
- test/: unit tests organized by module.
- config files (pyproject.toml, requirements-dev.txt, .gitlab-ci.yml).
- The package uses an abstract‐base‐class pattern under data_crawler/sources/abc and data_crawler/sinks/abc to load source/sink plugins dynamically via type names from the YAML configuration.
- CI and packaging scripts indicate a container‐first deployment and external dependency on Redis.

##### README for additional insights
- Configuration is entirely YAML‐driven, with “data sources” sections. Each section defines exactly one source plugin, optional polling parameters, and one sink plugin (or Redis shorthand).
- Runtime behavior: a single process can host multiple “channels,” each consisting of a source instance + scheduler + sink instance.
- Uses Redis streams as the default internal bus; optionally pushes out to industrial protocols (Modbus, OPC UA).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Docker Container"
        direction TB
        ContainerConfig["Configuration Loader"]:::internal
        CLI["CLI / Entry Point"]:::internal
        Orchestrator["Core Orchestrator"]:::internal
        Scheduler["Scheduler / Poller"]:::internal
        subgraph "Plugin Layer"
            direction TB
            ABCs["Abstract Plugin Base Classes"]:::internal
            Sources["Source Plugins"]:::internal
            Sinks["Sink Plugins"]:::internal
            Utils["Shared Utilities"]:::internal
        end
        Bus["Internal Messaging Bus (Redis Streams)"]:::datastore
    end

    subgraph "External Components"
        direction TB
        ForecastAPIs["HTTP Forecast APIs"]:::external_api
        IndustrialDevices["Industrial Devices (Modbus TCP, OPC UA)"]:::industrial
        RedisInstance["Redis Instance"]:::datastore
    end

    ContainerConfig -->|loads YAML config| Orchestrator
    CLI -->|invokes| ContainerConfig
    Orchestrator -->|spawns| Scheduler
    Scheduler -->|triggers| Sources
    Sources -->|fetch data| ExternalServices
    ExternalServices[[ ]]:::dummy
    ExternalServices --> ForecastAPIs
    ExternalServices --> IndustrialDevices
    Sources -->|emit data| Sinks
    Sinks -->|append/push| Bus
    Sinks -->|push| IndustrialDevices
    Sinks -->|append| RedisInstance
    Bus -->|stream| Sinks

    classDef internal fill:#D0E8FF,stroke:#005B9E,color:#003366;
    classDef datastore fill:#DFF2BF,stroke:#4F8A10,color:#4F8A10;
    classDef external_api fill:#FFEFCF,stroke:#B66E00,color:#B66E00;
    classDef industrial fill:#EEEEEE,stroke:#888888,color:#555555;
    classDef dummy fill:none,stroke:none;

    click ContainerConfig "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/__main__.py"
    click ContainerConfig "https://github.com/ait-rdp/rdp-data-crawler/blob/development/docker/etc/data_crawler/config.yml"
    click ContainerConfig "https://github.com/ait-rdp/rdp-data-crawler/blob/development/docker/etc/data_crawler/default.env"
    click CLI "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/cli.py"
    click Orchestrator "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/query_supervisor.py"
    click Orchestrator "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/query_executors.py"
    click Scheduler "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/query_supervisor.py"
    click ABCs "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/abc/abstract_source.py"
    click ABCs "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/abc/active_source_sync.py"
    click ABCs "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/abc/history.py"
    click ABCs "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/abc/http_cache.py"
    click ABCs "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/abc/message.py"
    click ABCs "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sinks/abc/abstract_sink.py"
    click Sources "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/weatherbit.py"
    click Sources "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/yr_no.py"
    click Sources "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/knmi.py"
    click Sources "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/entsoe_da.py"
    click Sources "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/fronius.py"
    click Sources "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/redis.py"
    click Sources "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/modbus.py"
    click Sources "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/teltonika_modbus.py"
    click Sources "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/opc_ua.py"
    click Sinks "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sinks/redis.py"
    click Sinks "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sinks/modbus.py"
    click Sinks "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sinks/opc_ua.py"
    click Sinks "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sinks/websocket.py"
    click Utils "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/shared/modbus.py"
    click Utils "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/shared/opcua.py"
    click Utils "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/access/storage.py"
    click Utils "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/access/jsonpath.py"
    click Utils "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/access/guards.py"
    click Bus "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sinks/redis.py"
    click Bus "https://github.com/ait-rdp/rdp-data-crawler/blob/development/data_crawler/sources/redis.py"
```
