How ait-rdp/rdp-data-crawler works
Project type and purpose
• This is a backend data‐crawler microservice written in Python, packaged as a Docker container.
• Its purpose is to read external data (forecasts, measurements) from a wide variety of sources (HTTP APIs, Modbus TCP, OPC UA, Redis streams), process it (polling or event‐driven), and write it into Redis streams or push it to external systems (Modbus, OPC UA).
File structure
• Top level:
• data_crawler/: the application package (entry point, CLI, plugin modules for “sources” and “sinks,” shared utilities).
• data/: example test data.
• docker/: Dockerfile and default configuration under /etc/data_crawler.
• docs/: installation and data‐source/sink specifications.
• test/: unit tests organized by module.
• config files (pyproject.toml, requirements-dev.txt, .gitlab-ci.yml).
• The package uses an abstract‐base‐class pattern under data_crawler/sources/abc and data_crawler/sinks/abc to load source/sink plugins dynamically via type names from the YAML configuration.
• CI and packaging scripts indicate a container‐first deployment and external dependency on Redis.
README for additional insights
• Configuration is entirely YAML‐driven, with “data sources” sections. Each section defines exactly one source plugin, optional polling parameters, and one sink plugin (or Redis shorthand).
• Runtime behavior: a single process can host multiple “channels,” each consisting of a source instance + scheduler + sink instance.
• Uses Redis streams as the default internal bus; optionally pushes out to industrial protocols (Modbus, OPC UA).