How tableau/altimeter works
Project type and purpose
• Altimeter is a Python-based tool for scanning AWS resources (across Accounts/Organizations), generating an RDF graph of those resources, and loading it into a triplestore (e.g., AWS Neptune or Blazegraph).
• It also includes a “Query Job” microservice (QJ) that exposes a REST API, persists requests/results in Postgres, invokes serverless workers (Lambdas), and notifies users of results.
File structure to uncover architectural elements
• Top-level “altimeter/aws” – the scanner engine, AWS SDK wrappers, resource definitions, multiplexers (local vs. Lambda), and Step-Function samplers.
• “altimeter/core” – graph-modeling, RDF writer/reader, Neptune client, pruner logic.
• “altimeter/qj” – Query Job library: API endpoints, ORM (SQLAlchemy + Alembic migrations under services/qj/alembic), Lambdas (executor, pruner, publish, query, remediator), client code, schemas, notification logic.
• “bin/” and Dockerfiles – entry-points for CLI, Lambda handlers, and containerized deployment artifacts.
• “services/qj” – Flask/Gunicorn REST service, local run scripts, RDS schema migrations.
README.md for workflow insights
• The CLI (altimeter <config>) drives the scan, producing an RDF file.
• That RDF is then loaded into a triplestore (Neptune/Blazegraph).
• QJ handles querying that graph via jobs, storing metadata in Postgres, and running actual graph queries against Neptune.
Main components & their interactions
a. Scanner Component
• CLI or Lambda‐based orchestrator (via AWS Step Functions)
• AWS SDK clients (in altimeter/aws/auth & altimeter/aws/scan)
• Resource modules model each AWS service type
• Outputs RDF artifact via altimeter/core/artifact_io
• (Optional) stores artifact in S3 or local file system
b. Triplestore
• AWS Neptune (or Blazegraph) where the RDF graph is loaded
c. QueryJob Service (QJ)
• REST API (Flask/Gunicorn in services/qj)
• Relational DB (Postgres/RDS) – job and result metadata (SQLAlchemy + Alembic)
• AWS SQS/SNS (for job queuing and notifications)
• Worker Lambdas (altimeter/qj/lambdas)
– executor: reads job, queries Neptune via SPARQL, writes result to storage
– query: smaller or ad-hoc graph queries
– pruner: cleans up old artifacts/DB entries
– publish: pushes notifications (SNS, email)
– remediator: handles errors or retries
d. Orchestration & Deployment
• AWS Step Functions coordinate scanning sub-steps (compile graph spec, scan, prune, load).
• Docker containers for altimeter-lambda, QJ API, local testing.
• CI/CD workflows (GitHub Actions in .github/workflows/ci.yml).
Architectural patterns & design principles
• Separation of concerns – scanner vs. graph core vs. query job service
• Event-driven microservices – REST API + SQS/SNS + Lambdas
• Polyglot persistence – graph DB (Neptune) + relational DB (Postgres)
• Serverless orchestration via Step Functions + Lambdas
• Config-driven behavior (TOML files in conf/)
Detail
• Error-handling and retry (via remediator Lambda).