# lux4rd0/weatherflow-collector architecture

> Architecture diagram of the GitHub repository lux4rd0/weatherflow-collector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lux4rd0/weatherflow-collector
- Repository: https://github.com/lux4rd0/weatherflow-collector

## Overview

##### Main Components

- Data Collectors:
– The “src/collector” directory contains modules for various data collection mechanisms such as:
- REST Export (rest_export.py)
- REST Forecasts (rest_forecasts.py)
- REST Import (rest_import.py)
- REST Observations (separately for devices and stations)
- REST Stats (rest_stats.py)
- UDP Receiver (udp.py)
- WebSocket Collector (websocket.py)
– These correspond to different ways of fetching data from the WeatherFlow device/cloud (e.g., local-udp, remote-rest, remote-socket).

- Handlers:
– Within “src/handlers” there is a mirror of the collectors (along with additional features like system metrics) that likely perform request processing and routing for each data type.
– This indicates an architectural layer that decouples raw data ingestion from business logic.

- Data Processor:
– The “src/processor” folder (with collector_data.py and export.py) signals an intermediate processing/transformation component.
– This part of the system packages or formats the data for storage and export, potentially handling CSV export and other transformations.

- Data Storage:
– The “src/storage” directory shows support for two types of persistence:
- File Storage (file.py)
- InfluxDB Storage (influxdb.py and influxdb_delete.py)
– InfluxDB is used as the primary time series database for historical and real-time weather data.

- Provider/Server:
– The “src/provider” with websocket_server.py indicates that the system can provide live updates or serve data to connected clients via a WebSocket server.
– This is the outward-facing API layer for real-time data dissemination.

- Utility & Configuration Components:
– Files like “src/config.py”, “src/config_validator.py”, “src/event_manager.py”, and “src/logger.py” help manage settings, validate configurations, handle events, and log system activities.
– “src/utils” contains helper functions (e.g., calculate_weather_metrics.py) to derive or manipulate weather data.

- External Integrations:
– Grafana is heavily featured – with dedicated dashboard JSON files (under the “grafana” directory) and datasource YAML files – meaning the system architecture must show integration with Grafana and InfluxDB.
– Environment variables (as outlined in the README) provide runtime configuration for API tokens, InfluxDB credentials, WebSocket settings, etc.

##### Relationships and Interactions

- Ingestion Layer:
– Data flows from external sources (WeatherFlow devices or WeatherFlow cloud) via multiple collectors (UDP, REST, WebSocket).

- Processing and Transformation:
– From the handlers, data is routed to the processor for any required computations (e.g., calculating derived weather metrics).

- Persistence Layer:
– The processed data is sent to the storage modules.
– Illustrate two storage pathways: one for persisting to InfluxDB (for time series queries and Grafana use) and another (optional) for file-based storage for exports/imports.
– Clearly mark the configuration-driven choice in storage (via environment variables).

- Provider and Visualization:
– The provider component, namely the WebSocket server, serves processed data out to clients, such as Grafana dashboards.

- Configuration and Monitoring:
– Illustrate the usage of the configuration (config.py) and event manager modules that monitor health-checks (using defined environmental health check intervals) and manage system restarts.
– A side block can represent logging (via logger.py) that tracks system events and potential errors.

##### Architectural Patterns and Design Principles

- Modular Design:
– The clearly separated directories (collector, handlers, processor, storage, provider, utils) support a modular architecture.
– Emphasize the separation of concerns; each module is responsible for a specific task.

- Configurability:
– The project is highly configurable via environmental variables – making it flexible for different deployment environments.

- Extensibility:
– The design supports multiple data collection methods (local-udp, remote-rest, remote-socket, etc.) and can support more collectors or processing pipelines as needed.
– It also supports multiple devices and time zone adjustments for different geographic locations.

- Containerization:

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Sources and Systems
    WF["WeatherFlow Devices"]:::external
    Influx["InfluxDB"]:::database
    GrafanaDS["Grafana Datasource"]:::external
    GrafanaDB["Grafana Dashboards"]:::external

    %% Docker Container Boundary
    subgraph "Docker Container"
        direction TB
        %% Data Ingestion Pipeline
        subgraph "Data Ingestion Pipeline"
            direction TB
            Collectors["Data Collectors (REST, UDP, WebSocket)"]:::internal
            Handlers["Handlers"]:::internal
            Processor["Data Processor"]:::internal
            %% Branch: Two pipelines from Processor
            Storage["Data Storage (InfluxDB/File)"]:::internal
            Provider["WebSocket Provider"]:::internal
            
            %% Connections within pipeline
            Collectors -->|"collects via UDP/REST/WebSocket"| Handlers
            Handlers -->|"routes data"| Processor
            Processor -->|"transformed data"| Storage
            Processor -->|"real-time feed"| Provider
        end

        %% Utility & Configuration Components
        subgraph "Utility & Config"
            direction TB
            Config["Config"]:::utility
            ConfigValidator["Config Validator"]:::utility
            EventManager["Event Manager"]:::utility
            Logger["Logger"]:::utility
            Utils["Utils"]:::utility
        end
    end

    %% External Integrations Connections
    Influx -->|"stores time-series data"| GrafanaDS
    Influx -->|"stores time-series data"| GrafanaDB
    Provider -->|"pushes live updates"| GrafanaDB
    GrafanaDB -->|"queries data"| Influx

    %% Data flow from external source into Docker Container
    WF -->|"sends raw data"| Collectors

    %% Utility influence (dashed lines)
    Config ---|"configures"| Collectors
    Config ---|"configures"| Handlers
    Config ---|"configures"| Processor
    Config ---|"configures"| Storage
    Config ---|"configures"| Provider

    ConfigValidator ---|"validates"| Config
    EventManager ---|"monitors"| Processor
    Logger ---|"logs events from"| Processor
    Utils ---|"assists"| Processor

    %% Click Events for Component Mapping
    click Collectors "https://github.com/lux4rd0/weatherflow-collector/tree/main/src/collector"
    click Handlers "https://github.com/lux4rd0/weatherflow-collector/tree/main/src/handlers"
    click Processor "https://github.com/lux4rd0/weatherflow-collector/tree/main/src/processor"
    click Storage "https://github.com/lux4rd0/weatherflow-collector/tree/main/src/storage"
    click Provider "https://github.com/lux4rd0/weatherflow-collector/blob/main/src/provider/websocket_server.py"
    click Config "https://github.com/lux4rd0/weatherflow-collector/blob/main/src/config.py"
    click ConfigValidator "https://github.com/lux4rd0/weatherflow-collector/blob/main/src/config_validator.py"
    click EventManager "https://github.com/lux4rd0/weatherflow-collector/blob/main/src/event_manager.py"
    click Logger "https://github.com/lux4rd0/weatherflow-collector/blob/main/src/logger.py"
    click Utils "https://github.com/lux4rd0/weatherflow-collector/tree/main/src/utils"
    click GrafanaDB "https://github.com/lux4rd0/weatherflow-collector/tree/main/grafana/dashboards"
    click GrafanaDS "https://github.com/lux4rd0/weatherflow-collector/blob/main/grafana/datasources/websocket-weatherflow-collector.yaml"
    click DockerContainer "https://github.com/lux4rd0/weatherflow-collector/tree/main/Dockerfile"

    %% Styles
    classDef internal fill:#bbf,stroke:#2f73b2,stroke-width:2px;
    classDef utility fill:#ffeb3b,stroke:#c8a700,stroke-width:2px;
    classDef external fill:#fdd,stroke:#d14,stroke-width:2px;
    classDef database fill:#f9c,stroke:#a60,stroke-width:2px;
    
    %% Dummy node to attach Dockerfile click event (not rendered as visible node)
    DockerContainer["Docker Container"]:::internal
    
    %% Link Dockerfile click: Using invisible connection to Docker Container boundary
    DockerContainer --- Collectors
```
