How playtikaoss/leaktopus works
Project Type and Purpose
• This is a full-stack, Docker-first security scanning platform (“Leaktopus”) with:
• A Python Flask API backend (with Celery workers).
• A Vue.js frontend served behind Nginx.
• A lightweight SQLite persistence layer plus Redis as a Celery broker.
• Optional ELK-stack (Elasticsearch, Logstash, Kibana) for indexing and visualization.
• External integrations: GitHub API, Pastebin API, Shhgit, TruffleHog, Slack, MS Teams, Cortex XSOAR.
High-Level File Structure Analysis
• Top-level orchestration: docker-compose.yml (+ override/examples), .env, scripts/, .github/workflows/
• Extensions/: Docker images & configs for ELK-extensions (apm-server, curator, enterprise-search, logspout).
• leaktopus_backend/:
• config/ (Flask, Celery, settings)
• leaktopus/app.py, routes/ (HTTP entrypoints)
• domain/ (entities, contracts, use-cases) — domain-driven design
• common/ (DB, Elasticsearch handlers, GitHub indexer, leak enhancer)
• services/ (alert, leak, secret, contributor, etc.) implementing provider-interface adapters
• tasks/ (Celery tasks & clients)
• SQLite providers, memory providers (for tests), tests/
• leaktopus_frontend/: Vue.js SPA, router, components, assets, Dockerfile, Nginx startup
Key Insights from README
• API-first architecture, all functionality exposed as REST.
• Async task dispatch via Celery + Redis.
• Built-in heuristic engine for filtering.
• ELK stack optional but pre-configured.
• Notification via Slack/MS Teams/Cortex.
Components
a. Orchestrator Layer (Docker-Compose)
– Defines containers & overlay network.
b. Frontend Layer
– Nginx container serving Vue.js SPA.
– Browser users → HTTP → Nginx → Vue App.
– Vue App → REST → Flask API.
c. Backend API
– Flask + Gunicorn container.
– Routes layer calls domain use-cases via services.
– Reads/writes SQLite DB.
– Enqueues tasks to Redis broker.
d. Worker Layer
– Celery worker containers.
– Subscribes to Redis.
– Executes scan, enhancement, notification tasks.
– Uses common/github_indexer, leak_enhancer, Shhgit/TruffleHog.
– Persists discoveries to SQLite & indexes to Elasticsearch.
e. Persistence & Messaging
– Redis (broker for Celery).
– SQLite (primary state store).
f. Optional ELK Stack
– Elasticsearch: indexes leaks + repo data.
– Logstash: collects & forwards logs to ES.
– Kibana: visualization.
– APM-Server, Curator, Enterprise-Search, Logspout extensions.
g. External Systems
– GitHub API, Pastebin API (scan sources).
– Slack/MS Teams/Cortex XSOAR (notifications).
h. CI/CD & Infrastructure
– GitHub Actions for testing.
– .env for configuration.
Relationships & Interactions
• Frontend → Backend API (REST).
• Backend → Redis (enqueue tasks).
• Worker → Redis (consume tasks).
• Worker → SQLite & Elasticsearch (write).
• Backend → SQLite & Elasticsearch (read).
• Worker → External scan sources.
• Worker → Notification endpoints.
• Logstash/Logspout → Elasticsearch (logs).
Architectural Patterns & Principles
• Hexagonal/Ports-&-Adapters: domain/use-cases ←→ service/provider interfaces.
• Layered: routes → domain → services → providers.
• Asynchronous messaging: Celery with Redis broker.
• Domain-Driven Design: domain/entities/use-cases separated from infra.
• Configuration as code: Docker-Compose, .env, config modules.