# playtikaoss/leaktopus architecture

> Architecture diagram of the GitHub repository playtikaoss/leaktopus, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/playtikaoss/leaktopus
- Repository: https://github.com/playtikaoss/leaktopus

## Overview

##### Project Type and Purpose
- This is a full-stack, Docker-first security scanning platform (“Leaktopus”) with:
- A Python Flask API backend (with Celery workers).
- A Vue.js frontend served behind Nginx.
- A lightweight SQLite persistence layer plus Redis as a Celery broker.
- Optional ELK-stack (Elasticsearch, Logstash, Kibana) for indexing and visualization.
- External integrations: GitHub API, Pastebin API, Shhgit, TruffleHog, Slack, MS Teams, Cortex XSOAR.

##### High-Level File Structure Analysis
- Top-level orchestration: docker-compose.yml (+ override/examples), .env, scripts/, .github/workflows/
- Extensions/: Docker images & configs for ELK-extensions (apm-server, curator, enterprise-search, logspout).
- leaktopus_backend/:
- config/ (Flask, Celery, settings)
- leaktopus/app.py, routes/ (HTTP entrypoints)
- domain/ (entities, contracts, use-cases) — domain-driven design
- common/ (DB, Elasticsearch handlers, GitHub indexer, leak enhancer)
- services/ (alert, leak, secret, contributor, etc.) implementing provider-interface adapters
- tasks/ (Celery tasks & clients)
- SQLite providers, memory providers (for tests), tests/
- leaktopus_frontend/: Vue.js SPA, router, components, assets, Dockerfile, Nginx startup

##### Key Insights from README
- API-first architecture, all functionality exposed as REST.
- Async task dispatch via Celery + Redis.
- Built-in heuristic engine for filtering.
- ELK stack optional but pre-configured.
- Notification via Slack/MS Teams/Cortex.

##### Components
a. Orchestrator Layer (Docker-Compose)
– Defines containers & overlay network.
b. Frontend Layer
– Nginx container serving Vue.js SPA.
– Browser users → HTTP → Nginx → Vue App.
– Vue App → REST → Flask API.
c. Backend API
– Flask + Gunicorn container.
– Routes layer calls domain use-cases via services.
– Reads/writes SQLite DB.
– Enqueues tasks to Redis broker.
d. Worker Layer
– Celery worker containers.
– Subscribes to Redis.
– Executes scan, enhancement, notification tasks.
– Uses common/github_indexer, leak_enhancer, Shhgit/TruffleHog.
– Persists discoveries to SQLite & indexes to Elasticsearch.
e. Persistence & Messaging
– Redis (broker for Celery).
– SQLite (primary state store).
f. Optional ELK Stack
– Elasticsearch: indexes leaks + repo data.
– Logstash: collects & forwards logs to ES.
– Kibana: visualization.
– APM-Server, Curator, Enterprise-Search, Logspout extensions.
g. External Systems
– GitHub API, Pastebin API (scan sources).
– Slack/MS Teams/Cortex XSOAR (notifications).
h. CI/CD & Infrastructure
– GitHub Actions for testing.
– .env for configuration.

##### Relationships & Interactions
- Frontend → Backend API (REST).
- Backend → Redis (enqueue tasks).
- Worker → Redis (consume tasks).
- Worker → SQLite & Elasticsearch (write).
- Backend → SQLite & Elasticsearch (read).
- Worker → External scan sources.
- Worker → Notification endpoints.
- Logstash/Logspout → Elasticsearch (logs).

##### Architectural Patterns & Principles
- Hexagonal/Ports-&-Adapters: domain/use-cases ←→ service/provider interfaces.
- Layered: routes → domain → services → providers.
- Asynchronous messaging: Celery with Redis broker.
- Domain-Driven Design: domain/entities/use-cases separated from infra.
- Configuration as code: Docker-Compose, .env, config modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestrator Layer
    subgraph "Orchestrator Layer" 
        style Orchestrator fill:#f9f,stroke:#333,stroke-width:2px
        DockerCompose["docker-compose.yml"]:::orchestrator
        Scripts["scripts/"]:::orchestrator
    end

    %% Frontend Layer
    subgraph "Frontend Layer" 
        style Frontend fill:#ccf,stroke:#333,stroke-width:2px
        Nginx["Nginx Container"]:::frontend
        VueApp["Vue.js App"]:::frontend
        FrontendDocker["Dockerfile"]:::frontend
        StartNginx["start-nginx.sh"]:::frontend
        VueMain["src/main.js"]:::frontend
        Router["src/router/index.js"]:::frontend
    end

    %% Backend API Layer
    subgraph "Backend API Layer"
        style Backend fill:#cfc,stroke:#333,stroke-width:2px
        FlaskAPI["Flask + Gunicorn"]:::backend
        API_Docker["Dockerfile"]:::backend
        AppPy["leaktopus/app.py"]:::backend
        CeleryApp["leaktopus/celery_app.py"]:::backend
        CeleryConfig["config/celery.py"]:::backend
    end

    %% Worker Layer
    subgraph "Worker Layer"
        style Worker fill:#cfc,stroke:#333,stroke-width:2px
        CeleryWorker["Celery Workers"]:::backend
        TaskPy["tasks/task.py"]:::backend
        ClientPy["clients/celery_client.py"]:::backend
        AlertsTask["send_alerts_notification_task.py"]:::backend
        TaskMgr["task_manager.py"]:::backend
    end

    %% Persistence & Messaging
    subgraph "Persistence & Messaging"
        style Persist fill:#fcf,stroke:#333,stroke-width:2px
        Redis["Redis (Broker)"]:::broker
        SQLite["SQLite DB"]:::db
        DataDir["data/"]:::db
    end

    %% Optional ELK Stack
    subgraph "Optional ELK Stack" 
        style ELK fill:#eee,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5
        Elasticsearch["Elasticsearch"]:::elk
        Logstash["Logstash"]:::elk
        Kibana["Kibana"]:::elk
        APM["extensions/apm-server"]:::elk
    end

    %% External Integrations
    subgraph "External Integrations"
        style External fill:#eec,stroke:#333,stroke-width:2px
        GitHubAPI["GitHub API"]:::external
        Pastebin["Pastebin API"]:::external
        Slack["Slack"]:::external
        MSTeams["MS Teams"]:::external
        XSOAR["Cortex XSOAR"]:::external
        GHIndexer["github_indexer.py"]:::external
        Enhancer["leak_enhancer.py"]:::external
    end

    %% Users & CI/CD
    User["Browser Users"]:::user
    KibanaUI["Kibana UI"]:::user
    CI["GitHub Actions"]:::infra
    Env[".env"]:::infra

    %% Connections
    User -->|HTTP| Nginx
    Nginx -->|serves SPA| VueApp
    VueApp -->|REST API| FlaskAPI
    FlaskAPI -->|enqueue tasks| Redis
    FlaskAPI -->|read/write| SQLite
    Redis -->|consume| CeleryWorker
    CeleryWorker -->|write| SQLite
    CeleryWorker -->|index| Elasticsearch
    CeleryWorker -->|scan| GitHubAPI
    CeleryWorker -->|scan| Pastebin
    CeleryWorker -->|notify| Slack
    CeleryWorker -->|notify| MSTeams
    CeleryWorker -->|notify| XSOAR
    Logstash -->|logs| Elasticsearch
    User -->|visualize| KibanaUI
    CI -->|tests| FlaskAPI
    CI -->|tests| CeleryWorker

    %% Click Events
    click DockerCompose "https://github.com/playtikaoss/leaktopus/blob/main/docker-compose.yml"
    click Scripts "https://github.com/playtikaoss/leaktopus/tree/main/scripts/"
    click FrontendDocker "https://github.com/playtikaoss/leaktopus/tree/main/leaktopus_frontend/Dockerfile"
    click StartNginx "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_frontend/start-nginx.sh"
    click VueMain "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_frontend/src/main.js"
    click Router "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_frontend/src/router/index.js"
    click API_Docker "https://github.com/playtikaoss/leaktopus/tree/main/leaktopus_backend/Dockerfile"
    click AppPy "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_backend/leaktopus/app.py"
    click CeleryApp "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_backend/leaktopus/celery_app.py"
    click CeleryConfig "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_backend/config/celery.py"
    click TaskPy "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_backend/leaktopus/tasks/task.py"
    click ClientPy "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_backend/leaktopus/tasks/clients/celery_client.py"
    click AlertsTask "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_backend/leaktopus/tasks/send_alerts_notification_task.py"
    click TaskMgr "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_backend/leaktopus/tasks/task_manager.py"
    click DataDir "https://github.com/playtikaoss/leaktopus/tree/main/leaktopus_backend/data/"
    click APM "https://github.com/playtikaoss/leaktopus/tree/main/extensions/apm-server"
    click GHIndexer "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_backend/leaktopus/common/github_indexer.py"
    click Enhancer "https://github.com/playtikaoss/leaktopus/blob/main/leaktopus_backend/leaktopus/common/leak_enhancer.py"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#004085,color:#004085
    classDef backend fill:#d4edda,stroke:#155724,color:#155724
    classDef broker fill:#fff3cd,stroke:#856404,color:#856404,shape:hexagon
    classDef db fill:#ffeeba,stroke:#856404,color:#856404,shape:cylinder
    classDef elk fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef external fill:#f5c6cb,stroke:#721c24,color:#721c24,shape:cloud
    classDef user fill:#d1ecf1,stroke:#0c5460,color:#0c5460,shape:oval
    classDef infra fill:#d6d8d9,stroke:#1b1e21,color:#1b1e21
    classDef orchestrator fill:#f5e1f0,stroke:#68217a,color:#68217a shape:stack
    class Orchestrator,Scripts orchestrator
    class FrontendDocker,StartNginx,VueMain,Router frontend
    class API_Docker,AppPy,CeleryApp,CeleryConfig backend
    class TaskPy,ClientPy,AlertsTask,TaskMgr backend
    class Redis broker
    class SQLite,DataDir db
    class Elasticsearch,Logstash,Kibana,APM elk
    class GitHubAPI,Pastebin,Slack,MSTeams,XSOAR,GHIndexer,Enhancer external
    class User,KibanaUI user
    class CI,Env infra
```
