How daq-tools/kotori works
1. Kotori is a Python telemetry broker built around Twisted’s async runtime. The repo centers on acquiring sensor data from multiple protocols, normalizing it into a common channel model, storing it in time-series databases, and exposing it for export and visualization.
2. The main runtime entry points are in kotori/core.py and the DAQ application layer under kotori/daq/application/. These application modules (basic, composite, mqttkit) assemble a deployment from configuration: protocol listeners, decoders, storage backends, and optional graphing/export services.
3. Ingestion is split across protocol adapters in kotori/daq/intercom/ and kotori/io/protocol/. Supported ingress paths include MQTT, HTTP, UDP, and WAMP. The repository also includes a Crossbar config, which indicates WAMP/Crossbar is a first-class integration point for some deployments.
4. Incoming payloads are routed through a channel/path abstraction in kotori/io/router/path.py and then normalized by decoders in kotori/daq/decoder/. The decoder layer contains vendor-specific parsers for Airrohr, Tasmota, and The Things Network uplinks, plus schema helpers for turning heterogeneous payloads into a common measurement structure.
5. The normalized data model appears to be channel-oriented and time-series-friendly: data is associated with hierarchical channel paths, tags, and timestamps. The strategy modules (lan, wan) suggest different addressing/routing conventions for local-network versus wide-area sensor deployments.
6. Persistence is handled primarily by kotori/daq/storage/influx.py, with utility code around it. InfluxDB is the core storage backend in the architecture; the docs and tests also show the project’s time-series export and graphing workflows are built around Influx-compatible data.
7. Visualization and dashboard management live in kotori/daq/graphing/grafana/. This subsystem talks to Grafana through its API, manages dashboards/panels/annotations, and packages Grafana resource templates. That makes Grafana an external service Kotori can provision or update automatically rather than merely query.
8. Data export is a separate subsystem under kotori/io/export/. It supports tabular output, HTML rendering, plots, and database-backed export. This is the main boundary for “read-only” retrieval of historical measurements, distinct from the ingestion/storage path.
9. There is a small frontend package in kotori/frontend/ with templates, static assets, and a server/app wrapper. It looks like a lightweight admin or status UI rather than the core data path, so it sits beside the backend services instead of in-line with acquisition.
10. Firmware support is another explicit boundary: kotori/firmware/ contains builder/service code, and the repo includes Arduino and PHP client examples under clients/. This indicates Kotori is not only a server but also a toolchain for generating or supporting device-side payload producers.
11. Configuration is file-driven. The etc/ tree contains base, production, test, and example .ini files for different deployment profiles, plus vendor-specific presets. That configuration layer is the main way the same codebase is adapted to different telemetry scenarios.
12. Deployment and packaging are a major part of the repo architecture: Dockerfiles, docker-compose.yml, systemd units, Debian packaging scripts, and wheel build scripts are all present. The project is designed to run either as a packaged service or inside containerized infrastructure.
13. The doc/ tree is extensive and effectively documents the system’s intended topology: sensor clients send MQTT/HTTP payloads into Kotori, Kotori stores and transforms them, and Grafana/InfluxDB provide the downstream analytics and visualization stack. The tests mirror this architecture with protocol, decoder, storage, and export coverage.
Main entry points: core.py Core · path.py Path router · etc Config & delivery · basic.py Basic app
Components 18
Runtime
Core process entry points and application assembly
- Core · Main runtime bootstrap and orchestration
kotori/core.py - Basic app · Minimal telemetry deployment wiring
kotori/daq/application/basic.py - Composite app · Multi-service deployment wiring
kotori/daq/application/composite.py - MQTT kit · MQTT-oriented deployment profile
kotori/daq/application/mqttkit.py
Ingest & Normalize
Protocol listeners, routing, decoders, and addressing strategies
- MQTT ingest · MQTT listeners and runtime adapters
kotori/daq/intercom/mqtt - HTTP ingest · HTTP acquisition and forwarding path
kotori/io/protocol/http.py - UDP/WAMP · Datagram and bus ingress paths
kotori/daq/intercom/udp.py - WAMP bus · Crossbar/WAMP integration layer
kotori/bus/wamp.py - Path router · Channel and path normalization
kotori/io/router/path.py - Decoders · Vendor-specific payload parsers
kotori/daq/decoder - Strategies · LAN and WAN addressing conventions
kotori/daq/strategy
Storage & Viz
Persistence, exports, and Grafana integration
- Influx storage · Primary persistence backend
kotori/daq/storage/influx.py - Export tools · Tabular, HTML, plot, and database export
kotori/io/export - Grafana mgr · Grafana API, dashboards, panels, and resources
kotori/daq/graphing/grafana/manager.py - Grafana API · External Grafana service interaction
kotori/daq/graphing/grafana/api.py
Interfaces
Frontend UI and firmware support boundaries
Config & Delivery
Deployment, packaging, and configuration assets
- Config & delivery · INI configs, Docker, systemd, and packaging scripts
etc
Connections 20
- Core → Basic app: boots · Core starts the basic deployment profile
- Core → Composite app: boots · Core starts the composite deployment profile
- Core → MQTT kit: boots · Core starts the MQTT-kit deployment profile
- Basic app → MQTT ingest: wires · Assembles MQTT listeners into the runtime
- Basic app → HTTP ingest: wires · Adds HTTP acquisition endpoints
- Composite app → UDP/WAMP: wires · Combines datagram and bus ingress
- Composite app → WAMP bus: integrates · Connects to Crossbar/WAMP deployments
- MQTT ingest → Path router: routes · Maps incoming messages to channel paths
- HTTP ingest → Path router: routes · Normalizes HTTP payload paths
- UDP/WAMP → Path router: routes · Routes UDP and WAMP payloads into channels
- Path router → Decoders: feeds · Hands normalized payloads to decoders
- Decoders → Strategies: interprets · Applies LAN/WAN naming and routing conventions
- Decoders → Influx storage: writes · Produces measurements for time-series persistence
- Influx storage → Export tools: serves · Provides read-side data for export workflows
- Influx storage → Grafana mgr: powers · Backs dashboard and annotation provisioning
- Grafana mgr → Grafana API: calls · Uses the external Grafana API
- Export tools → Frontend: renders · Feeds admin or status presentation paths
- Config & delivery → Core: configures · Deployment files and packaging shape runtime startup
- Config & delivery → Frontend: packages · Frontend assets and service wiring are deployed alongside the backend
- Config & delivery → Firmware: supports · Includes device-side support material and builders