# tableau/altimeter architecture

> Architecture diagram of the GitHub repository tableau/altimeter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tableau/altimeter
- Repository: https://github.com/tableau/altimeter

## Overview

##### Project type and purpose
- Altimeter is a Python-based tool for scanning AWS resources (across Accounts/Organizations), generating an RDF graph of those resources, and loading it into a triplestore (e.g., AWS Neptune or Blazegraph).
- It also includes a “Query Job” microservice (QJ) that exposes a REST API, persists requests/results in Postgres, invokes serverless workers (Lambdas), and notifies users of results.

##### File structure to uncover architectural elements
- Top-level “altimeter/aws” – the scanner engine, AWS SDK wrappers, resource definitions, multiplexers (local vs. Lambda), and Step-Function samplers.
- “altimeter/core” – graph-modeling, RDF writer/reader, Neptune client, pruner logic.
- “altimeter/qj” – Query Job library: API endpoints, ORM (SQLAlchemy + Alembic migrations under services/qj/alembic), Lambdas (executor, pruner, publish, query, remediator), client code, schemas, notification logic.
- “bin/” and Dockerfiles – entry-points for CLI, Lambda handlers, and containerized deployment artifacts.
- “services/qj” – Flask/Gunicorn REST service, local run scripts, RDS schema migrations.

##### README.md for workflow insights
- The CLI (`altimeter <config>`) drives the scan, producing an RDF file.
- That RDF is then loaded into a triplestore (Neptune/Blazegraph).
- QJ handles querying that graph via jobs, storing metadata in Postgres, and running actual graph queries against Neptune.

##### Main components & their interactions
a. Scanner Component
- CLI or Lambda‐based orchestrator (via AWS Step Functions)
- AWS SDK clients (in `altimeter/aws/auth` & `altimeter/aws/scan`)
- Resource modules model each AWS service type
- Outputs RDF artifact via `altimeter/core/artifact_io`
- (Optional) stores artifact in S3 or local file system
b. Triplestore
- AWS Neptune (or Blazegraph) where the RDF graph is loaded
c. QueryJob Service (QJ)
- REST API (Flask/Gunicorn in services/qj)
- Relational DB (Postgres/RDS) – job and result metadata (SQLAlchemy + Alembic)
- AWS SQS/SNS (for job queuing and notifications)
- Worker Lambdas (`altimeter/qj/lambdas`)
– executor: reads job, queries Neptune via SPARQL, writes result to storage
– query: smaller or ad-hoc graph queries
– pruner: cleans up old artifacts/DB entries
– publish: pushes notifications (SNS, email)
– remediator: handles errors or retries
d. Orchestration & Deployment
- AWS Step Functions coordinate scanning sub-steps (compile graph spec, scan, prune, load).
- Docker containers for altimeter-lambda, QJ API, local testing.
- CI/CD workflows (GitHub Actions in .github/workflows/ci.yml).

##### Architectural patterns & design principles
- Separation of concerns – scanner vs. graph core vs. query job service
- Event-driven microservices – REST API + SQS/SNS + Lambdas
- Polyglot persistence – graph DB (Neptune) + relational DB (Postgres)
- Serverless orchestration via Step Functions + Lambdas
- Config-driven behavior (TOML files in `conf/`)

##### Detail
- Error-handling and retry (via remediator Lambda).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External
    GH["GitHub Actions (CI/CD)"]:::external
    GH --> CLI

    %% Scanner Subsystem
    subgraph "Scanner Subsystem"
        CLI["CLI (altimeter)"]:::custom
        SF["Step Functions"]:::aws
        CLI -->|"invoke"| SF

        subgraph "Scanner Lambdas"
            ScanLambda["Scan Lambda"]:::custom
        end
        SF -->|"Step Functions execution"| ScanLambda

        Auth["AWS SDK Clients & Auth"]:::custom
        ScanLambda -->|"boto3 calls"| Auth

        ResDefs["AWS Resource Definitions"]:::custom
        Auth --> ResDefs

        Engine["Scanner Engine & Multiplexers"]:::custom
        ResDefs --> Engine

        ArtifactIO["Artifact I/O (RDF)"]:::custom
        Engine -->|"generate RDF via writer"| ArtifactIO

        Storage["Artifact Storage (S3/File)"]:::aws
        ArtifactIO -->|"write artifact"| Storage

        Triplestore["Triplestore (Neptune)"]:::aws
        Storage -->|"load RDF"| Triplestore
    end

    %% QueryJob Subsystem
    subgraph "QueryJob Service"
        Client["Client"]:::external
        API["QJ REST API (Flask/Gunicorn)"]:::custom
        DB["QJ Postgres DB (RDS)"]:::aws
        Msg["Messaging (SQS/SNS)"]:::aws

        Client -->|"HTTP"| API
        API -->|"SQLAlchemy"| DB
        API -->|"enqueue job"| Msg

        subgraph "QJ Worker Lambdas"
            Exec["Executor Lambda"]:::custom
            Query["Query Lambda"]:::custom
            Pruner["Pruner Lambda"]:::custom
            Publish["Publish Lambda"]:::custom
            Remediator["Remediator Lambda"]:::custom
        end

        Msg --> Exec
        Msg --> Query
        Msg --> Pruner
        Msg --> Publish
        Msg --> Remediator

        Exec -->|"SPARQL Query"| Triplestore
        Exec -->|"store metadata"| DB
        Exec -->|"notify"| Publish
        Publish -->|"SNS Notification"| Client
    end

    %% Click Events
    click CLI "https://github.com/tableau/altimeter/tree/master/bin/altimeter"
    click SF "https://github.com/tableau/altimeter/blob/master/bin/sfn_compile_graphs.py"
    click Auth "https://github.com/tableau/altimeter/blob/master/altimeter/aws/auth/accessor.py"
    click ResDefs "https://github.com/tableau/altimeter/blob/master/altimeter/aws/resource/__init__.py"
    click Engine "https://github.com/tableau/altimeter/blob/master/altimeter/aws/scan/scan.py"
    click ArtifactIO "https://github.com/tableau/altimeter/blob/master/altimeter/core/artifact_io/writer.py"
    click API "https://github.com/tableau/altimeter/blob/master/services/qj/main.py"
    click DB "https://github.com/tableau/altimeter/blob/master/altimeter/qj/models/job.py"
    click Exec "https://github.com/tableau/altimeter/blob/master/altimeter/qj/lambdas/executor.py"
    click Query "https://github.com/tableau/altimeter/blob/master/altimeter/qj/lambdas/query.py"
    click Pruner "https://github.com/tableau/altimeter/blob/master/altimeter/qj/lambdas/pruner.py"
    click Publish "https://github.com/tableau/altimeter/blob/master/altimeter/qj/lambdas/publish.py"
    click Remediator "https://github.com/tableau/altimeter/blob/master/altimeter/qj/lambdas/remediator.py"

    %% Styles
    classDef aws fill:#bbdefb,stroke:#2196f3,color:#0d47a1
    classDef custom fill:#c8e6c9,stroke:#388e3c,color:#1b5e20
    classDef external fill:#e0e0e0,stroke:#9e9e9e,color:#424242
```
