How someengineering/fixinventory works
Project type and purpose
• Fix Inventory is a multi‐service, full-stack Python application organized as a single Git monorepo. It provides:
• A CLI (“fixshell”) for querying and visualizing cloud inventory
• A REST API/service (“fixcore”) that orchestrates inventory collection and normalization
• A worker service (“fixworker”) that runs collection jobs in parallel
• A metrics service (“fixmetrics”) that gathers runtime telemetry
• A plugin architecture under plugins/* to talk to AWS, GCP, Azure, Kubernetes, etc.
• Under the hood, it stores data in a graph database (ArangoDB) and a time-series database, and exposes metrics for Prometheus.
File structure
• Top-level folders:
• fixcore/ – core API, scheduler, graph manager, DB access layers, web handlers, task queue
• fixworker/ – worker process that picks tasks from the internal message bus and calls plugin collectors
• fixmetrics/ – service that collects runtime metrics and exports them (Prometheus endpoint)
• fixshell/ – interactive CLI front-end (cmd interpreter, autocomplete, rendering)
• plugins/* – dozens of cloud-provider collectors, each a self-contained package
• docker-related directories and docker-compose.yaml – one compose for dev and CI with containers for API, worker, metrics, ArangoDB, TSDB, DNS, logging, Prometheus
• Look for message_bus.py, task/, web/api.py, db/ directories—these signal a microservice + message-driven architecture.
Read README for architecture cues
• Three phases: Collect → Normalize → Triage
• Data flows from cloud APIs → plugin collectors → normalized graph → query/policies → CLI or exported workflows
• Graph database is central for dependencies; TSDB + Prometheus for metrics.
Main components & interactions
1. fixshell CLI
2. fixcore API & scheduler
3. Internal Message Bus / Task Queue
4. fixworker Worker Pool
5. fixmetrics Metrics Exporter
6. Graph Database (ArangoDB)
7. Time-Series DB (Prometheus TSDB or equivalent)
8. Prometheus Server (scrapes metrics endpoints)
9. External Cloud Provider APIs (AWS, GCP, Azure, K8s, etc.)
10. DNSmasq & syslog-ng (in Docker Compose for service discovery & logging)
• User runs fixshell → HTTP(S) calls → fixcore REST API
• fixcore API posts tasks to internal message bus
• fixworker nodes subscribe → dequeue tasks → invoke plugin collectors → call External Cloud APIs
• fixworker receives raw data → normalize via fixcore’s graph_manager → write nodes & edges to ArangoDB
• fixcore and fixworker expose Prometheus metrics → scraped by Prometheus Server → stored in TSDB
• fixmetrics service queries ArangoDB and internal stats → exposes additional metrics endpoints for Prometheus
• CLI or user dashboards query fixcore REST API → fetch normalized graph data or reports
C. Architectural patterns & design principles
• Microservices separated by concern (API, workers, metrics)
• Message-driven task distribution (pub/sub or queue)
• Plugin-based extensibility (each provider collector as its own package)
• Graph data model for unified resource relationships
• Polyglot persistence: document-graph DB + time-series DB
• Containerized deployment (Docker Compose, devcontainer, CI workflows)
D. Key technologies & libraries
• Python 3 packages: fixcore, fixworker, fixmetrics, fixshell, fixlib
• ArangoDB accessed via async_arangodb
• Prometheus client libs for metrics
• Docker / Docker Compose for local dev & CI
• GitHub Actions for build/test/publish
• Plugin entry‐points via setuptools for dynamic collector loading
Elements
• Component labels (e.g. “fixcore REST API”, “ArangoDB”)
– Blue rectangles for services
• Optional annotations: ports (e.g. 8080, 9090), protocol (HTTP, AMQP/Redis), library names
Detailed capture of essential elements
• Don’t forget internal modules like graph_manager, message_bus, task scheduler, plugin loader
• That each plugin (AWS, GCP, Azure, etc.) is dynamically loaded by fixworker
• Prometheus scraping both fixcore and fixworker, plus fixmetrics as a metrics pipeline
• DNSmasq & syslog-ng only if they matter for deployment/service discovery