# cert-polska/mwdb-core architecture

> Architecture diagram of the GitHub repository cert-polska/mwdb-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cert-polska/mwdb-core
- Repository: https://github.com/cert-polska/mwdb-core

## Overview

##### Project type and purpose
- This is a full-stack web application (“MWDB Core”) for storing, searching and sharing malware samples and their metadata.
- It consists of:
- A Python-based REST API and CLI (“mwdb/…”) with plugin and background-task support via Karton
- A React/TypeScript single-page application (“mwdb/web/…”)
- Persistent storage (PostgreSQL for relational data, JSONB for rich attributes, possibly Redis for counters)
- OAuth/OIDC authentication and optional external identity providers
- File/blob storage for malware binaries
- A search subsystem built on Lucene (via a Python wrapper)
- Metrics export (Prometheus) and integration points (webhooks, plugins)

##### File structure to infer the architecture
- Top-level “mwdb” folder
- mwdb/app.py, core/, resources/, schema/ → monolithic API server (Flask/Falcon style)
- mwdb/model/ + migrations → SQLAlchemy models and Alembic migrations
- mwdb/cli/ → command-line interface for admin tasks and migrations
- mwdb/web/ → separate React SPA client served by the API or static host
- docker/, deploy/, docker-compose*.yml → containerized deployment of API, web, DB, workers, cache, OIDC stub
- dev/oidc → a minimal local OIDC provider for development
- mwdb/core/karton.py → Karton framework for background task execution
- docs/ → architecture and integration guides

##### Key insights from README
- Emphasizes storage, relations graph, quick search, user management, plugin/webhook integration
- Mentions CERT.pl service under the hood → production deployment uses multiple containers/services

##### Main components
a. Frontend (React SPA)
b. API Server (Python mwdb/app.py + resources/ + core/)
c. Database (PostgreSQL)
d. Blob/File Storage (could be local FS, S3, or Docker volume)
e. Search Engine (Lucene index embedded or separate service)
f. Auth Provider (OIDC server / external IdP)
g. Background Worker Cluster (Karton workers)
h. Cache/Rate Limiter (Redis)
i. Metrics & Monitoring (Prometheus, optionally Grafana)
j. Plugin system / Webhook receivers (external analysis backends)

##### Relationships and interactions
- React SPA ↔ HTTP/HTTPS → API Server (REST endpoints)
- API Server ↔ SQL (PostgreSQL) → persistent metadata
- API Server ↔ Lucene library or service → indexing & search queries
- API Server ↔ Redis → rate limiting and counters
- API Server ↔ Blob storage → store/retrieve malware binaries
- API Server ↔ OIDC/IdP → OAuth flows for login & token validation
- API Server → publish tasks to Karton → Worker Cluster processes analysis, plugins, webhooks
- Workers ↔ Blob storage & Database (read/write)
- API Server → expose Prometheus metrics → Monitoring system
- Optional: API Server ↔ External webhooks / third-party sharing endpoints

##### Architectural patterns and design principles
- Monolithic API service with clearly separated layers: routing (resources/), business logic (core/), data models (model/)
- Async/background processing via Karton (task queue pattern)
- Plugin/extension points (mwdb/core/plugins and web’s commons/plugins)
- Configuration and environment–driven deployment (docker-compose, deploy/ folders)
- OAuth/OIDC integration for single sign-on

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Side
    subgraph "Client Side"
        React["React SPA"]:::frontend
    end

    %% Server Side
    subgraph "Server Side"
        %% API Layer
        subgraph "API Server"
            API_server["API Server (mwdb/app.py)"]:::core
            Resources["Routing & Resources"]:::core
            CoreLogic["Business Logic"]:::core
            Models["Data Models"]:::datastore
            Migrations["DB Migrations"]:::datastore
            CLI["CLI"]:::core
            Config["Configuration"]:::core
            Static["Static Defaults"]:::core

            API_server -->|includes| Resources
            API_server -->|includes| CoreLogic
            API_server -->|loads models| Models
            API_server -->|runs migrations| Migrations
            API_server -->|provides CLI| CLI
            API_server -->|reads config| Config
            API_server -->|reads static| Static
        end

        %% Data Layer
        subgraph "Data Layer"
            Postgres[(PostgreSQL)]:::datastore
            Redis[(Redis Cache)]:::datastore
            BlobStorage[(Blob Storage)]:::datastore
            Lucene[(Search Engine)]:::datastore
        end

        %% Auth Service
        subgraph "Auth Service"
            OAuthCore["OIDC Integration"]:::auth
            DevOIDC["Dev OIDC Provider"]:::auth
            ExternalIdP["External IdP"]:::auth
        end

        %% Background Processing
        subgraph "Background Processing"
            Karton["Karton Framework"]:::core
            Workers["Worker Cluster"]:::core
            Plugins["Plugin System"]:::core
        end

        %% Monitoring
        subgraph "Monitoring"
            Prometheus[(Prometheus)]:::monitoring
            Grafana[(Grafana)]:::monitoring
            MetricsCore["Metrics Export"]:::monitoring
            MetricsResource["Metrics Endpoints"]:::monitoring
        end
    end

    %% Interconnections
    React -->|REST API calls| API_server
    API_server -->|SQL queries| Postgres
    API_server -->|Cache operations| Redis
    API_server -->|Store/Retrieve| BlobStorage
    API_server -->|Search queries| Lucene
    API_server -->|OAuth flows| OAuthCore
    OAuthCore -->|dev/test| DevOIDC
    OAuthCore -->|prod| ExternalIdP
    API_server -->|Publish tasks| Karton
    Karton -->|Dispatch jobs| Workers
    Workers -->|Read/Write| Postgres
    Workers -->|Read/Write| BlobStorage
    Workers -->|Invoke plugins| Plugins
    API_server -->|Expose metrics| Prometheus
    Prometheus -->|Visualize| Grafana
    API_server -->|Metrics API| MetricsResource
    MetricsCore -->|collects| Redis

    %% Click Events
    click React "https://github.com/cert-polska/mwdb-core/tree/master/mwdb/web/"
    click API_server "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/app.py"
    click Resources "https://github.com/cert-polska/mwdb-core/tree/master/mwdb/resources/"
    click CoreLogic "https://github.com/cert-polska/mwdb-core/tree/master/mwdb/core/"
    click Models "https://github.com/cert-polska/mwdb-core/tree/master/mwdb/model/"
    click Migrations "https://github.com/cert-polska/mwdb-core/tree/master/mwdb/model/migrations/"
    click CLI "https://github.com/cert-polska/mwdb-core/tree/master/mwdb/cli/"
    click BlobModel "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/model/blob.py"
    click BlobResource "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/resources/blob.py"
    click Search "https://github.com/cert-polska/mwdb-core/tree/master/mwdb/core/search/"
    click OAuthCore "https://github.com/cert-polska/mwdb-core/tree/master/mwdb/core/oauth/"
    click DevOIDC "https://github.com/cert-polska/mwdb-core/tree/master/dev/oidc/"
    click Karton "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/core/karton.py"
    click Plugins "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/core/plugins.py"
    click Remotes "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/resources/remotes.py"
    click RateLimit "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/core/rate_limit.py"
    click RedisCounter "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/core/metrics/redis_counter.py"
    click MetricsCore "https://github.com/cert-polska/mwdb-core/tree/master/mwdb/core/metrics/"
    click MetricsResource "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/resources/metrics.py"
    click Config "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/core/config.py"
    click Static "https://github.com/cert-polska/mwdb-core/blob/master/mwdb/core/static.py"
    click Deploy "https://github.com/cert-polska/mwdb-core/tree/master/deploy/"
    click Docker "https://github.com/cert-polska/mwdb-core/tree/master/docker/"

    %% Styles
    classDef frontend fill:#D0E6FF,stroke:#0366D6,stroke-width:1px;
    classDef core fill:#E0F8E0,stroke:#2E8B57,stroke-width:1px;
    classDef datastore fill:#FFE5B4,stroke:#DAA520,stroke-width:1px;
    classDef auth fill:#E6DAF0,stroke:#663399,stroke-width:1px;
    classDef monitoring fill:#F0F0F0,stroke:#666666,stroke-width:1px;
```
