# soumilshah1995/s3tables-storage-insights architecture

> Architecture diagram of the GitHub repository soumilshah1995/s3tables-storage-insights, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/soumilshah1995/s3tables-storage-insights
- Repository: https://github.com/soumilshah1995/s3tables-storage-insights

## Overview

##### Project type and purpose
- This is a monitoring/observability solution for AWS S3Tables (“S3Tables Storage Insights”).
- It consists of a Python-based metric‐collection job, a Prometheus-based metrics ingestion layer, and a Grafana-based visualization layer.

##### File structure
- Top‐level files:
- run.py – Python metric‐collector entry point
- requirements.txt – Python dependencies
- docker-compose.yml – orchestrates Prometheus & Grafana containers
- prometheus/prometheus.yml – Prometheus scrape configuration
- grafana/provisioning/… – Grafana datasources & dashboard provisioning
- No “frontend” or “backend” code per se; rather a pipeline of components.
- Deployment is containerized via Docker Compose.

##### README for additional insights
- run.py is invoked manually (or via cron) with AWS credentials, region, Prometheus Pushgateway address, etc.
- It computes metrics (partition counts, file counts, bytes/records, etc.) and pushes them to a Prometheus gateway endpoint.

##### Main components & their interactions
a. AWS S3Tables
- Source of truth; the collector queries S3Tables metadata and statistics.
b. Metric Collector (Python job)
- run.py: polls S3Tables, computes metrics, sends them via HTTP to the Prometheus Pushgateway (or Prometheus remote write).
c. Prometheus Pushgateway
- Receives pushed metrics; makes them available to Prometheus server.
d. Prometheus Server
- Scrapes the Pushgateway (per prometheus/prometheus.yml), stores time‐series data.
e. Grafana
- Queries Prometheus for metrics, renders dashboards (pre‐provisioned under grafana/provisioning).
f. Docker Compose
- Defines and runs the Grafana and Prometheus containers, wires up the networking and volumes.

##### Architectural patterns & design principles
- “Pull + Push” hybrid for Prometheus: collector pushes to Pushgateway; Prometheus pulls from it.
- Separation of concerns:
- Collector: metric extraction
- Storage: Prometheus
- Visualization: Grafana
- Infrastructure as Code: provisioning Grafana datasources/dashboards via YAML/JSON.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Source
    AWS["AWS S3Tables"]:::datasource

    %% Collector
    Collector["Metric Collector (run.py)"]:::processing
    Requirements["requirements.txt"]:::infra

    %% Docker Compose Group
    subgraph "Docker Compose"
        direction TB
        Pushgateway["Prometheus Pushgateway"]:::metrics
        Prometheus["Prometheus Server"]:::metrics
        PromTSDB[(Prometheus TSDB)]:::metrics
        PrometheusCfg["prometheus.yml"]:::infra
        Grafana["Grafana"]:::visualization
        GrafanaDS["prometheus.yaml"]:::infra
        GrafanaDB1["dashboards.yaml"]:::infra
        GrafanaDB2["s3_table_stats.json"]:::infra
    end

    %% Data Flow
    AWS -->|"API Query"| Collector
    Collector -->|"push metrics"| Pushgateway
    Prometheus -->|"scrape metrics"| Pushgateway
    Grafana -->|"query metrics"| Prometheus

    %% Auxiliary connections
    Collector --> Requirements
    Prometheus --> PromTSDB
    Prometheus --> PrometheusCfg
    Grafana --> GrafanaDS
    Grafana --> GrafanaDB1
    Grafana --> GrafanaDB2

    %% Click Events
    click Collector "https://github.com/soumilshah1995/s3tables-storage-insights/blob/main/run.py"
    click Requirements "https://github.com/soumilshah1995/s3tables-storage-insights/blob/main/requirements.txt"
    click DockerCompose "https://github.com/soumilshah1995/s3tables-storage-insights/blob/main/docker-compose.yml"
    click PrometheusCfg "https://github.com/soumilshah1995/s3tables-storage-insights/blob/main/prometheus/prometheus.yml"
    click GrafanaDS "https://github.com/soumilshah1995/s3tables-storage-insights/blob/main/grafana/provisioning/datasources/prometheus.yaml"
    click GrafanaDB1 "https://github.com/soumilshah1995/s3tables-storage-insights/blob/main/grafana/provisioning/dashboards/dashboards.yaml"
    click GrafanaDB2 "https://github.com/soumilshah1995/s3tables-storage-insights/blob/main/grafana/provisioning/dashboards/s3_table_stats.json"

    %% Styles
    classDef datasource fill:#ADD8E6,stroke:#000
    classDef processing fill:#90EE90,stroke:#000
    classDef metrics fill:#FFA500,stroke:#000
    classDef visualization fill:#9370DB,stroke:#000
    classDef infra fill:#D3D3D3,stroke:#000
```
