How cert-polska/mwdb-core works
Project type and purpose
• This is a full-stack web application (“MWDB Core”) for storing, searching and sharing malware samples and their metadata.
• It consists of:
• A Python-based REST API and CLI (“mwdb/…”) with plugin and background-task support via Karton
• A React/TypeScript single-page application (“mwdb/web/…”)
• Persistent storage (PostgreSQL for relational data, JSONB for rich attributes, possibly Redis for counters)
• OAuth/OIDC authentication and optional external identity providers
• File/blob storage for malware binaries
• A search subsystem built on Lucene (via a Python wrapper)
• Metrics export (Prometheus) and integration points (webhooks, plugins)
File structure to infer the architecture
• Top-level “mwdb” folder
• mwdb/app.py, core/, resources/, schema/ → monolithic API server (Flask/Falcon style)
• mwdb/model/ + migrations → SQLAlchemy models and Alembic migrations
• mwdb/cli/ → command-line interface for admin tasks and migrations
• mwdb/web/ → separate React SPA client served by the API or static host
• docker/, deploy/, docker-compose*.yml → containerized deployment of API, web, DB, workers, cache, OIDC stub
• dev/oidc → a minimal local OIDC provider for development
• mwdb/core/karton.py → Karton framework for background task execution
• docs/ → architecture and integration guides
Key insights from README
• Emphasizes storage, relations graph, quick search, user management, plugin/webhook integration
• Mentions CERT.pl service under the hood → production deployment uses multiple containers/services
Main components
a. Frontend (React SPA)
b. API Server (Python mwdb/app.py + resources/ + core/)
c. Database (PostgreSQL)
d. Blob/File Storage (could be local FS, S3, or Docker volume)
e. Search Engine (Lucene index embedded or separate service)
f. Auth Provider (OIDC server / external IdP)
g. Background Worker Cluster (Karton workers)
h. Cache/Rate Limiter (Redis)
i. Metrics & Monitoring (Prometheus, optionally Grafana)
j. Plugin system / Webhook receivers (external analysis backends)
Relationships and interactions
• React SPA ↔ HTTP/HTTPS → API Server (REST endpoints)
• API Server ↔ SQL (PostgreSQL) → persistent metadata
• API Server ↔ Lucene library or service → indexing & search queries
• API Server ↔ Redis → rate limiting and counters
• API Server ↔ Blob storage → store/retrieve malware binaries
• API Server ↔ OIDC/IdP → OAuth flows for login & token validation
• API Server → publish tasks to Karton → Worker Cluster processes analysis, plugins, webhooks
• Workers ↔ Blob storage & Database (read/write)
• API Server → expose Prometheus metrics → Monitoring system
• Optional: API Server ↔ External webhooks / third-party sharing endpoints
Architectural patterns and design principles
• Monolithic API service with clearly separated layers: routing (resources/), business logic (core/), data models (model/)
• Async/background processing via Karton (task queue pattern)
• Plugin/extension points (mwdb/core/plugins and web’s commons/plugins)
• Configuration and environment–driven deployment (docker-compose, deploy/ folders)
• OAuth/OIDC integration for single sign-on