How soumilshah1995/s3tables-storage-insights works
Project type and purpose
• This is a monitoring/observability solution for AWS S3Tables (“S3Tables Storage Insights”).
• It consists of a Python-based metric‐collection job, a Prometheus-based metrics ingestion layer, and a Grafana-based visualization layer.
File structure
• Top‐level files:
• run.py – Python metric‐collector entry point
• requirements.txt – Python dependencies
• docker-compose.yml – orchestrates Prometheus & Grafana containers
• prometheus/prometheus.yml – Prometheus scrape configuration
• grafana/provisioning/… – Grafana datasources & dashboard provisioning
• No “frontend” or “backend” code per se; rather a pipeline of components.
• Deployment is containerized via Docker Compose.
README for additional insights
• run.py is invoked manually (or via cron) with AWS credentials, region, Prometheus Pushgateway address, etc.
• It computes metrics (partition counts, file counts, bytes/records, etc.) and pushes them to a Prometheus gateway endpoint.
Main components & their interactions
a. AWS S3Tables
• Source of truth; the collector queries S3Tables metadata and statistics.
b. Metric Collector (Python job)
• run.py: polls S3Tables, computes metrics, sends them via HTTP to the Prometheus Pushgateway (or Prometheus remote write).
c. Prometheus Pushgateway
• Receives pushed metrics; makes them available to Prometheus server.
d. Prometheus Server
• Scrapes the Pushgateway (per prometheus/prometheus.yml), stores time‐series data.
e. Grafana
• Queries Prometheus for metrics, renders dashboards (pre‐provisioned under grafana/provisioning).
f. Docker Compose
• Defines and runs the Grafana and Prometheus containers, wires up the networking and volumes.
Architectural patterns & design principles
• “Pull + Push” hybrid for Prometheus: collector pushes to Pushgateway; Prometheus pulls from it.
• Separation of concerns:
• Collector: metric extraction
• Storage: Prometheus
• Visualization: Grafana
• Infrastructure as Code: provisioning Grafana datasources/dashboards via YAML/JSON.