How luizadevops/gcp-toolkit works
Project type and purpose
• This is an open-source, command-line Python toolkit (“GCP Services Toolkit”) for auditing and reporting on a Google Cloud Platform (GCP) project.
• It provides three pluggable services: a Firewall Rule Inspector/Cleaner, an IAM Policy Scanner, and a BigQuery Usage Reporter.
File structure
• Root files:
• main.py: CLI entry point that reads config, parses flags, and orchestrates all tools.
• config_sample.json (→ config.json): JSON configuration for project ID, thresholds, flags.
• Dockerfile / .dockerignore: containerization support.
• requirements.txt: Python dependencies.
• gcp_services_toolkit/: package containing each service as its own sub-package:
• cost_reporter/service.py
• firewall_inspector/service.py
• iam_scanner/service.py
• toolkit.py: registers and invokes each service.
• gcp_utils/: shared utility layer:
• clients.py: wraps Google API clients (Compute, IAM, BigQuery, Storage).
• utils.py: common functions (e.g., pagination, logging, error handling).
README for additional insights
• The three tools run by default when you execute main.py.
• Each tool:
• Invokes GCP REST/SDK APIs via gcp_utils.
• Processes & flags items based on config.json rules.
• Optionally deletes resources (--delete) or simulates (--dry-run).
• Prerequisites: Python ≥3.9, gcloud CLI, GCP APIs enabled.
a. Main components:
1) CLI Layer:
• User & flags (--config, --delete, --dry-run)
• main.py
2) Configuration:
• config.json
3) Orchestrator:
• toolkit.py (registers & dispatches each service)
4) Services (three modules):
• FirewallInspectorService
• IAMScannerService
• CostReporterService
5) Utility layer:
• GCPClientFactory (clients.py)
• SharedHelperFunctions (utils.py)
6) External Systems:
• GCP Compute API (firewall rules)
• GCP IAM & Cloud Storage APIs (bucket IAM)
• GCP BigQuery API (INFORMATION_SCHEMA query)
7) Output / Logs:
• stdout/log file, optional resource deletion.
b. Relationships & Data Flows:
1) CLI → Config Loader → Orchestrator
2) Orchestrator → each Service → gcp_utils clients → respective GCP API
3) Service receives API response → applies business rules (utils) → logs output / flags / deletes
4) All results returned to CLI → final summary printed.
c. Architectural patterns & design principles:
• Modular / Plugin-style: every tool sits in its own sub-package, and toolkit.py dynamically discovers or registers them.
• Single Responsibility: each service only handles one domain (firewall, IAM, cost).
• Layered design: CLI → Orchestrator → Service → Utilities → External APIs.
• Configuration-driven behavior: thresholds, regions, project IDs live in config.json.
d. Key technologies & frameworks:
• Python 3.9+ (type hints, Optional)
• google-cloud-bigquery, google-api-python-client
• Docker for containerization
• gcloud CLI for authentication (ADC)
• JSON for configuration