# someengineering/fixinventory architecture

> Architecture diagram of the GitHub repository someengineering/fixinventory, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/someengineering/fixinventory
- Repository: https://github.com/someengineering/fixinventory

## Overview

##### Project type and purpose
- Fix Inventory is a multi‐service, full-stack Python application organized as a single Git monorepo. It provides:
- A CLI (“fixshell”) for querying and visualizing cloud inventory
- A REST API/service (“fixcore”) that orchestrates inventory collection and normalization
- A worker service (“fixworker”) that runs collection jobs in parallel
- A metrics service (“fixmetrics”) that gathers runtime telemetry
- A plugin architecture under plugins/* to talk to AWS, GCP, Azure, Kubernetes, etc.
- Under the hood, it stores data in a graph database (ArangoDB) and a time-series database, and exposes metrics for Prometheus.

##### File structure
- Top-level folders:
- fixcore/ – core API, scheduler, graph manager, DB access layers, web handlers, task queue
- fixworker/ – worker process that picks tasks from the internal message bus and calls plugin collectors
- fixmetrics/ – service that collects runtime metrics and exports them (Prometheus endpoint)
- fixshell/ – interactive CLI front-end (cmd interpreter, autocomplete, rendering)
- plugins/* – dozens of cloud-provider collectors, each a self-contained package
- docker-related directories and docker-compose.yaml – one compose for dev and CI with containers for API, worker, metrics, ArangoDB, TSDB, DNS, logging, Prometheus
- Look for message_bus.py, task/, web/api.py, db/ directories—these signal a microservice + message-driven architecture.

##### Read README for architecture cues
- Three phases: Collect → Normalize → Triage
- Data flows from cloud APIs → plugin collectors → normalized graph → query/policies → CLI or exported workflows
- Graph database is central for dependencies; TSDB + Prometheus for metrics.

##### Main components & interactions
1. fixshell CLI
2. fixcore API & scheduler
3. Internal Message Bus / Task Queue
4. fixworker Worker Pool
5. fixmetrics Metrics Exporter
6. Graph Database (ArangoDB)
7. Time-Series DB (Prometheus TSDB or equivalent)
8. Prometheus Server (scrapes metrics endpoints)
9. External Cloud Provider APIs (AWS, GCP, Azure, K8s, etc.)
10. DNSmasq & syslog-ng (in Docker Compose for service discovery & logging)

- User runs fixshell → HTTP(S) calls → fixcore REST API
- fixcore API posts tasks to internal message bus
- fixworker nodes subscribe → dequeue tasks → invoke plugin collectors → call External Cloud APIs
- fixworker receives raw data → normalize via fixcore’s graph_manager → write nodes & edges to ArangoDB
- fixcore and fixworker expose Prometheus metrics → scraped by Prometheus Server → stored in TSDB
- fixmetrics service queries ArangoDB and internal stats → exposes additional metrics endpoints for Prometheus
- CLI or user dashboards query fixcore REST API → fetch normalized graph data or reports

C. Architectural patterns & design principles
- Microservices separated by concern (API, workers, metrics)
- Message-driven task distribution (pub/sub or queue)
- Plugin-based extensibility (each provider collector as its own package)
- Graph data model for unified resource relationships
- Polyglot persistence: document-graph DB + time-series DB
- Containerized deployment (Docker Compose, devcontainer, CI workflows)

D. Key technologies & libraries
- Python 3 packages: fixcore, fixworker, fixmetrics, fixshell, fixlib
- ArangoDB accessed via async_arangodb
- Prometheus client libs for metrics
- Docker / Docker Compose for local dev & CI
- GitHub Actions for build/test/publish
- Plugin entry‐points via setuptools for dynamic collector loading

##### Elements
- Component labels (e.g. “fixcore REST API”, “ArangoDB”)
– Blue rectangles for services
- Optional annotations: ports (e.g. 8080, 9090), protocol (HTTP, AMQP/Redis), library names

##### Detailed capture of essential elements
- Don’t forget internal modules like graph_manager, message_bus, task scheduler, plugin loader
- That each plugin (AWS, GCP, Azure, etc.) is dynamically loaded by fixworker
- Prometheus scraping both fixcore and fixworker, plus fixmetrics as a metrics pipeline
- DNSmasq & syslog-ng only if they matter for deployment/service discovery

## Diagram (Mermaid)

```mermaid
graph TD
    %% Frontend
    CLI["fixshell CLI"]:::service
    CLI -->|"HTTP(S) requests"| CORE

    %% Docker Compose Boundary
    subgraph "Docker Compose Cluster"
      direction TB
      CORE["fixcore REST API & Scheduler"]:::service
      BUS["Internal Message Bus / Task Queue"]:::service
      WORKER["fixworker Worker Pool"]:::service
      METRICS["fixmetrics Metrics Exporter"]:::service
      ARANGODB["ArangoDB (Graph DB)"]:::database
      TSDB["Time-Series DB (Prometheus TSDB)"]:::database
      PROMSERVER["Prometheus Server"]:::service
    end

    %% External
    EXTERNAL["External Cloud Provider APIs"]:::cloud

    %% Data Flows
    CORE -->|"POST task→queue"| BUS
    BUS -->|"dequeue tasks"| WORKER
    WORKER -->|"invoke collectors"| PLUGINS
    PLUGINS["Plugin Architecture"]:::service
    PLUGINS -->|"API calls"| EXTERNAL
    WORKER -->|"normalize & write"| ARANGODB
    CORE -->|"write metrics"| PROMSERVER
    WORKER -->|"write metrics"| PROMSERVER
    METRICS -->|"queries DB & stats"| ARANGODB
    METRICS -->|"exposes metrics"| PROMSERVER
    CLI -->|"query graph data"| CORE

    %% Classes
    classDef service fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef database fill:#98FB98,stroke:#000,stroke-width:1px;
    classDef cloud fill:#FFD580,stroke:#000,stroke-width:1px;

    %% Click Events - fixshell CLI
    click CLI "https://github.com/someengineering/fixinventory/tree/main/fixshell/"
    click CLI "https://github.com/someengineering/fixinventory/blob/main/fixshell/fixshell/__main__.py"

    %% Click Events - fixcore
    click CORE "https://github.com/someengineering/fixinventory/tree/main/fixcore/"
    click CORE "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/web/api.py"
    click CORE "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/web/auth.py"
    click CORE "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/task/scheduler.py"
    click CORE "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/task/job_handler.py"
    click CORE "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/task/subscribers.py"
    click CORE "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/system_start.py"
    click CORE "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/__main__.py"

    %% Click Events - Message Bus
    click BUS "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/message_bus.py"
    click BUS "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/worker_task_queue.py"
    click BUS "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/task/task_description.py"
    click BUS "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/task/task_dependencies.py"

    %% Click Events - fixworker
    click WORKER "https://github.com/someengineering/fixinventory/tree/main/fixworker/"
    click WORKER "https://github.com/someengineering/fixinventory/blob/main/fixworker/fixworker/__main__.py"
    click WORKER "https://github.com/someengineering/fixinventory/blob/main/fixworker/fixworker/collect.py"
    click WORKER "https://github.com/someengineering/fixinventory/blob/main/fixworker/fixworker/pluginloader.py"
    click WORKER "https://github.com/someengineering/fixinventory/blob/main/fixworker/fixworker/fixcore.py"

    %% Click Events - fixmetrics
    click METRICS "https://github.com/someengineering/fixinventory/tree/main/fixmetrics/"
    click METRICS "https://github.com/someengineering/fixinventory/blob/main/fixmetrics/fixmetrics/metrics.py"
    click METRICS "https://github.com/someengineering/fixinventory/blob/main/fixmetrics/fixmetrics/config.py"
    click METRICS "https://github.com/someengineering/fixinventory/blob/main/fixmetrics/fixmetrics/default_metrics.yaml"
    click METRICS "https://github.com/someengineering/fixinventory/blob/main/fixmetrics/fixmetrics/search.py"
    click METRICS "https://github.com/someengineering/fixinventory/blob/main/fixmetrics/fixmetrics/__main__.py"

    %% Click Events - Plugins
    click PLUGINS "https://github.com/someengineering/fixinventory/tree/main/plugins/"
    click PLUGINS "https://github.com/someengineering/fixinventory/blob/main/plugins/aws/fix_plugin_aws/collector.py"
    click PLUGINS "https://github.com/someengineering/fixinventory/blob/main/plugins/gcp/fix_plugin_gcp/collector.py"
    click PLUGINS "https://github.com/someengineering/fixinventory/blob/main/plugins/azure/fix_plugin_azure/collector.py"
    click PLUGINS "https://github.com/someengineering/fixinventory/blob/main/plugins/k8s/fix_plugin_k8s/collector.py"

    %% Click Events - ArangoDB
    click ARANGODB "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/db/async_arangodb.py"
    click ARANGODB "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/db/graphdb.py"
    click ARANGODB "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/db/entitydb.py"
    click ARANGODB "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/db/deferrededgesdb.py"
    click ARANGODB "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/graph_manager/graph_manager.py"

    %% Click Events - TSDB
    click TSDB "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/db/timeseriesdb.py"
    click TSDB "https://github.com/someengineering/fixinventory/blob/main/fixcore/fixcore/web/tsdb.py"

    %% Click Events - Prometheus Server & Compose
    click PROMSERVER "https://github.com/someengineering/fixinventory/blob/main/docker-compose.yaml"
    click PROMSERVER "https://github.com/someengineering/fixinventory/blob/main/docker/prometheus.yml"
    click PROMSERVER "https://github.com/someengineering/fixinventory/blob/main/dockerV2/prometheus.yml"
```
