How solectrus/ingest works
Main Components Identification
• Ingest Service (the core service, with its main entry point in app.rb and further configured via config.ru and routes)
• HTTP API Endpoints:
• POST /api/v2/write – receives incoming sensor data.
• GET / — exposes a stats page (with basic auth if enabled).
• GET /up — a simple health check endpoint.
• Buffer/Queue Mechanism:
• Local SQLite storage (data directory) for buffering data with a 12-hour retention.
• Background workers such as the cleanup worker (lib/cleanup_worker.rb) and outbox worker (lib/outbox_worker.rb) that process data in batches.
• House Power Calculation Module:
• House power recalculation and interpolation paths (modules such as lib/house_power_calculator.rb, lib/house_power_formula.rb, lib/interpolator.rb, and sensor configuration support in lib/sensor_env_config.rb).
• InfluxDB Interaction:
• The module responsible for writing data to InfluxDB (lib/influx_writer.rb).
Relationships and Data Flow
• Collectors represented externally (labeled CollectorA, CollectorB, CollectorC) as pushing data to the Ingest Service.
• The Ingest service receives incoming data through the /api/v2/write route. Immediately within the service, the following happens:
• Data is persisted into a local SQLite database (buffering).
• If the data includes any of the eight relevant sensors, triggers are initiated for the house power recalculation via the calculation modules.
• After processing, the service forwards the (potentially modified) data in batches to InfluxDB.
• InfluxDB then serves as the database for visualization (Dashboard pulls data from InfluxDB).
• A background cleanup worker periodically removes old data.
• Optionally, show the stats endpoint and health check endpoint as additional monitoring interfaces.
Architectural Patterns & Design Principles
• Proxy Pattern: The Ingest service acts as an intermediary layer between the collectors and InfluxDB.
• Buffering / Queue-based Processing: Incoming data is first persisted into SQLite before being forwarded, which decouples data ingestion from the database write operations.
• Modular Design: Distinct responsibilities are separated into routes (API endpoints), models (data structures), and libraries (business logic like the house power calculation, logging, etc.).
Technology/Framework Highlights
• Ruby: The whole project is built in Ruby (as shown by Gemfile, app.rb, and Ruby-specific tools like Guardfile and RSpec).
• SQLite: For buffered storage with a 12-hour retention policy.
• InfluxDB: Destination database and as part of the external system.
• Docker: Configuration files (Dockerfile, example Docker Compose snippet in README) indicate containerized deployment.