# solectrus/ingest architecture

> Architecture diagram of the GitHub repository solectrus/ingest, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/solectrus/ingest
- Repository: https://github.com/solectrus/ingest

## Overview

##### Main Components Identification
- Ingest Service (the core service, with its main entry point in app.rb and further configured via config.ru and routes)
- HTTP API Endpoints:
- POST /api/v2/write – receives incoming sensor data.
- GET / — exposes a stats page (with basic auth if enabled).
- GET /up — a simple health check endpoint.
- Buffer/Queue Mechanism:
- Local SQLite storage (data directory) for buffering data with a 12-hour retention.
- Background workers such as the cleanup worker (lib/cleanup_worker.rb) and outbox worker (lib/outbox_worker.rb) that process data in batches.
- House Power Calculation Module:
- House power recalculation and interpolation paths (modules such as lib/house_power_calculator.rb, lib/house_power_formula.rb, lib/interpolator.rb, and sensor configuration support in lib/sensor_env_config.rb).
- InfluxDB Interaction:
- The module responsible for writing data to InfluxDB (lib/influx_writer.rb).

##### Relationships and Data Flow
- Collectors represented externally (labeled CollectorA, CollectorB, CollectorC) as pushing data to the Ingest Service.
- The Ingest service receives incoming data through the /api/v2/write route. Immediately within the service, the following happens:
- Data is persisted into a local SQLite database (buffering).
- If the data includes any of the eight relevant sensors, triggers are initiated for the house power recalculation via the calculation modules.
- After processing, the service forwards the (potentially modified) data in batches to InfluxDB.
- InfluxDB then serves as the database for visualization (Dashboard pulls data from InfluxDB).
- A background cleanup worker periodically removes old data.
- Optionally, show the stats endpoint and health check endpoint as additional monitoring interfaces.

##### Architectural Patterns & Design Principles
- Proxy Pattern: The Ingest service acts as an intermediary layer between the collectors and InfluxDB.
- Buffering / Queue-based Processing: Incoming data is first persisted into SQLite before being forwarded, which decouples data ingestion from the database write operations.
- Modular Design: Distinct responsibilities are separated into routes (API endpoints), models (data structures), and libraries (business logic like the house power calculation, logging, etc.).

##### Technology/Framework Highlights
- Ruby: The whole project is built in Ruby (as shown by Gemfile, app.rb, and Ruby-specific tools like Guardfile and RSpec).
- SQLite: For buffered storage with a 12-hour retention policy.
- InfluxDB: Destination database and as part of the external system.
- Docker: Configuration files (Dockerfile, example Docker Compose snippet in README) indicate containerized deployment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Collectors
    subgraph "External Collectors"
        C1("Collector A"):::external
        C2("Collector B"):::external
        C3("Collector C"):::external
    end

    %% Ingest Service Layer
    subgraph "Ingest Service"
        IMain("Main Ingest Service"):::service
        API_Write("HTTP API - POST /api/v2/write"):::api
        API_Stats("HTTP API - GET /"):::api
        API_Health("HTTP API - GET /up"):::api
        Buffer("SQLite Buffer"):::database
        HouseCalc("House Power Calculator"):::module
        Cleanup("Cleanup Worker"):::worker
        Outbox("Outbox Worker"):::worker
        InfluxWriter("Influx Writer"):::module
    end

    %% External Systems Layer
    subgraph "External Systems"
        InfluxDB("InfluxDB"):::database
        Dashboard("Dashboard"):::external
    end

    %% Data Flow from External Collectors to Ingest Service
    C1 -->|"IncomingData"| API_Write
    C2 -->|"IncomingData"| API_Write
    C3 -->|"IncomingData"| API_Write

    %% Routing inside Ingest Service
    IMain -->|"routes"| API_Stats
    IMain -->|"routes"| API_Health
    IMain -->|"routes"| API_Write

    %% Processing Flow within Ingest Service
    API_Write -->|"PersistData"| Buffer
    API_Write -->|"TriggerRecalc"| HouseCalc
    HouseCalc -->|"ForwardData"| InfluxWriter
    Buffer -->|"BatchData"| InfluxWriter
    Cleanup -->|"Cleanup"| Buffer
    Outbox -->|"ProcessBatch"| Buffer

    %% Data Flow to External Systems
    InfluxWriter -->|"WriteData"| InfluxDB
    InfluxDB -->|"DataPull"| Dashboard

    %% Click Events for Component Mapping
    click IMain "https://github.com/solectrus/ingest/blob/develop/app.rb"
    click API_Write "https://github.com/solectrus/ingest/blob/develop/routes/write.rb"
    click API_Stats "https://github.com/solectrus/ingest/blob/develop/routes/stats.rb"
    click API_Health "https://github.com/solectrus/ingest/blob/develop/routes/up.rb"
    click Buffer "https://github.com/solectrus/ingest/tree/develop/data/"
    click Cleanup "https://github.com/solectrus/ingest/blob/develop/lib/cleanup_worker.rb"
    click Outbox "https://github.com/solectrus/ingest/blob/develop/lib/outbox_worker.rb"
    click HouseCalc "https://github.com/solectrus/ingest/blob/develop/lib/house_power_calculator.rb"
    click InfluxWriter "https://github.com/solectrus/ingest/blob/develop/lib/influx_writer.rb"

    %% Styles
    classDef external fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef service fill:#AED6F1,stroke:#2980B9,stroke-width:2px;
    classDef api fill:#D5F5E3,stroke:#27AE60,stroke-width:2px;
    classDef database fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef module fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
    classDef worker fill:#F5CBA7,stroke:#E67E22,stroke-width:2px;
```
