How georgeprichici/metadefender-menlo-integration works
Project Type and Purpose
• This project is a middleware integration tool designed to bridge Menlo Security’s file sanitization requests to MetaDefender Core or Cloud. It exposes REST API endpoints for file submissions, analysis, health checks, and file retrieval.
• The middleware supports asynchronous processing (leveraging Python’s async features introduced in Python 3.5) and can be deployed directly, containerized via Docker, or run in a Kubernetes cluster.
Analyzing the File Structure
• Top-level files and directories indicate several key components:
– The Dockerfile, Kubernetes folder, and deploy scripts show that containerization and orchestration are supported.
– The “api” directory contains API definitions (Postman collection and OpenAPI spec) for interfacing with Menlo Security.
– The “metadefender_menlo” package houses the core application code, structured into subcomponents:
• The “api” subfolder is split into:
◦ “adaptors” (e.g., base_adaptor.py, file_submit.py) that implement integrations with various MetaDefender APIs.
◦ “handlers” (e.g., health_check.py, file_submit.py, retrieve_sanitized.py) that define REST endpoints handling requests.
◦ “models” and “responses” that encapsulate data structures and HTTP responses.
• The presence of config.yml indicates a centralized configuration for API endpoints, HTTPS settings, logging, and MetaDefender parameters.
• Documentation (docs folder) and examples (README content describing integration with Menlo) help understand the external and internal behaviors.
Key Architectural Components
a. Main Components:
– Menlo Security (External Client): Represents the external entity invoking the Sanitization API, requiring HTTPS communication.
– Middleware (This Project):
• API Gateway / Endpoint Layer — the REST endpoints (handlers) receiving calls from Menlo.
• Business Logic Layer — incorporates adaptors that translate incoming requests to MetaDefender API calls and handle responses.
• Data Models & Response Objects — to validate and construct payloads.
• Configuration & Logging Modules — ensure proper settings for HTTPS, logging, and runtime parameters.
– MetaDefender Services:
• MetaDefender Core API.
• MetaDefender Cloud API (optional based on the configuration in config.yml).
– Deployment & Orchestration Components:
• Docker Container — built using the Dockerfile.
• Kubernetes Deployment — described in the kubernetes folder (deployment.yaml, service.yaml, ingress configurations).
b. Relationships and Data Flows:
– Menlo Security initiates a HTTPS request (e.g., file submission, health check) to the Middleware’s exposed endpoint.
– The Middleware’s API handlers receive the request and, based on the action, delegate processing to specific adaptors (e.g., file_submit adaptor).
– The adaptor calls out to the appropriate MetaDefender API (Core or Cloud) using defined endpoints and configuration parameters.
– Responses from MetaDefender are processed and transformed into a standardized format before being sent back as responses to Menlo.
– Logging and configuration components monitor and adjust these flows.
– The system’s containerization layers (Docker and Kubernetes) are responsible for packaging and deploying the middleware.
c. Architectural Patterns and Design Principles:
– Modular Design: Separation of concerns is evident in the segregation between API handlers, adaptors, models, and utility functions.
– API Gateway Pattern: The middleware centrally manages calls from Menlo and routes them to relevant processing modules.
– Asynchronous Processing: Leveraging Python’s async features to handle network calls to external APIs effectively.
– Containerization and Orchestration: Support for Docker and Kubernetes for streamlined deployment and scaling.
d. Relevant Technologies, Frameworks, and Libraries:
– Python (as the chosen programming language with async support).
– REST APIs (typically using frameworks like Flask, FastAPI, or similar – inferred from the structure).
– Docker for containerization.
– Kubernetes for orchestration.
– HTTPS configuration using locally stored certificates (with custom CA support as described in the README).
• External Clients: Menlo Security
• Middleware API Endpoints: Include handlers (e.g., file_submit, retrieve_sanitized, health_check)
• Business Logic/Adaptors: The adaptor layer interfacing with MetaDefender (show separate paths for “metadefender_core_api.py” and “metadefender_cloud_api.py”)
• Data Models & Response Objects (indicate their flow within middleware)
• External MetaDefender Services: Illustrate MetaDefender Core and MetaDefender Cloud as external systems.
• From Menlo Security to Middleware’s HTTPS endpoint.
• Bidirectional communication where applicable.
• Use another for internal components (API Gateway, Adaptors, Models).
– Make sure every component is clearly labeled with its role and, if possible, reference the file structure (e.g., “metadefender_menlo/api/handlers/health_check.py” for clarity).