How lux4rd0/weatherflow-collector works
Main Components
• Data Collectors:
– The “src/collector” directory contains modules for various data collection mechanisms such as:
• REST Export (rest_export.py)
• REST Forecasts (rest_forecasts.py)
• REST Import (rest_import.py)
• REST Observations (separately for devices and stations)
• REST Stats (rest_stats.py)
• UDP Receiver (udp.py)
• WebSocket Collector (websocket.py)
– These correspond to different ways of fetching data from the WeatherFlow device/cloud (e.g., local-udp, remote-rest, remote-socket).
• Handlers:
– Within “src/handlers” there is a mirror of the collectors (along with additional features like system metrics) that likely perform request processing and routing for each data type.
– This indicates an architectural layer that decouples raw data ingestion from business logic.
• Data Processor:
– The “src/processor” folder (with collector_data.py and export.py) signals an intermediate processing/transformation component.
– This part of the system packages or formats the data for storage and export, potentially handling CSV export and other transformations.
• Data Storage:
– The “src/storage” directory shows support for two types of persistence:
• File Storage (file.py)
• InfluxDB Storage (influxdb.py and influxdb_delete.py)
– InfluxDB is used as the primary time series database for historical and real-time weather data.
• Provider/Server:
– The “src/provider” with websocket_server.py indicates that the system can provide live updates or serve data to connected clients via a WebSocket server.
– This is the outward-facing API layer for real-time data dissemination.
• Utility & Configuration Components:
– Files like “src/config.py”, “src/config_validator.py”, “src/event_manager.py”, and “src/logger.py” help manage settings, validate configurations, handle events, and log system activities.
– “src/utils” contains helper functions (e.g., calculate_weather_metrics.py) to derive or manipulate weather data.
• External Integrations:
– Grafana is heavily featured – with dedicated dashboard JSON files (under the “grafana” directory) and datasource YAML files – meaning the system architecture must show integration with Grafana and InfluxDB.
– Environment variables (as outlined in the README) provide runtime configuration for API tokens, InfluxDB credentials, WebSocket settings, etc.
Relationships and Interactions
• Ingestion Layer:
– Data flows from external sources (WeatherFlow devices or WeatherFlow cloud) via multiple collectors (UDP, REST, WebSocket).
• Processing and Transformation:
– From the handlers, data is routed to the processor for any required computations (e.g., calculating derived weather metrics).
• Persistence Layer:
– The processed data is sent to the storage modules.
– Illustrate two storage pathways: one for persisting to InfluxDB (for time series queries and Grafana use) and another (optional) for file-based storage for exports/imports.
– Clearly mark the configuration-driven choice in storage (via environment variables).
• Provider and Visualization:
– The provider component, namely the WebSocket server, serves processed data out to clients, such as Grafana dashboards.
• Configuration and Monitoring:
– Illustrate the usage of the configuration (config.py) and event manager modules that monitor health-checks (using defined environmental health check intervals) and manage system restarts.
– A side block can represent logging (via logger.py) that tracks system events and potential errors.
Architectural Patterns and Design Principles
• Modular Design:
– The clearly separated directories (collector, handlers, processor, storage, provider, utils) support a modular architecture.
– Emphasize the separation of concerns; each module is responsible for a specific task.
• Configurability:
– The project is highly configurable via environmental variables – making it flexible for different deployment environments.
• Extensibility:
– The design supports multiple data collection methods (local-udp, remote-rest, remote-socket, etc.) and can support more collectors or processing pipelines as needed.
– It also supports multiple devices and time zone adjustments for different geographic locations.
• Containerization: