How rspamd/rspamd works
This project is Rspamd – an advanced, high-performance spam filtering system that processes and analyses email messages in real time. Its purpose is to examine emails using a variety of built‐in filters (such as regular expressions, Bayesian and fuzzy tests, DKIM/DMARC verification, etc.) and to interface with mail transfer agents (MTAs) using protocols like Milter, with additional support for an extensive Lua API allowing custom rule and plugin development.
Main Components
• Core Message Processing Engine: At the center of the system, this component handles core functionalities including scanning, scoring, and determining verdicts based on the applied rules.
• Worker Processes: Various workers (e.g., normal, fuzzy, proxy, etc.) are distributed to handle different aspects of email processing. They reside largely within sources in the “src” directory (e.g., src/worker.c, src/libserver, and src/libmime for message parsing).
• Lua API and Plugin Engine: The system includes extensive Lua libraries (in lualib and src/lua) which enable custom rule implementation and enhance flexibility through asynchronous network operations (DNS, HTTP and TCP/UDP).
• Configuration and Modules: The “conf” folder contains configuration files and module definitions. Dynamic configuration is supported; changes can be made to conf/modules.d and conf/settings.conf that affect various modules (e.g., DKIM, fuzzy, neural, etc.).
• Interfaces for External Integration:
– Milter Interface: Enables integration with MTAs like Postfix or Sendmail.
– Web Interface: Provided by the “interface” folder for administrative and monitoring purposes.
– External Services: Integration with caching systems and storage (e.g., Redis in fuzzy modules, SQL and other backends in libstat).
Interactions and Data Flow
• Incoming email messages are received via the Milter interface (or other network endpoints) – represented as an external input block.
• The main processing engine then distributes each message (via asynchronous events) to appropriate worker processes.
• Within each worker, the message is parsed (using libmime) and passed through multiple filters and modules for spam evaluation. This is where modules (from “src/plugins”, “rules”, and Lua-based plugins) apply filtering rules.
• The Lua API provides hooks for custom processing and interactions with external services (for DNS lookups, HTTP requests, Redis caching, etc.). Represent this as a distinct “Lua/Plugin Engine” layer interfacing with the core engine.
• After processing, a final verdict (along with evidence – headers, modifications, etc.) is returned to the MTA, or further actions (reject, tag, quarantine) are taken.
Architectural Patterns and Design Principles
• Loose Coupling: Note how configuration (via files in “conf”) and custom plugins (lualib, lua/scanners, etc.) can be modified without changing the core engine, promoting decoupling.
• High-Performance Scalability: Emphasize that the system is designed to handle hundreds of messages per second by leveraging multi-threaded or multi-process worker setups along with efficiently optimized C/C++ routines.
Relevant Technologies and Frameworks
• C/C++: Core processing is implemented in C/C++ (e.g., in src, src/libserver, src/libmime).
• Lua: The extensive Lua API (located primarily in lualib and src/lua) is used for rule definition and extended processing.
• Build and Configuration Tools: The project uses CMake (files like CMakeLists.txt, cmake/…), and GitHub workflows for CI.
• External Libraries: Many third-party libraries are embedded (in the “contrib” folder), such as hiredis, Hyperscan, and various string/utility libraries.
• Configuration files and build scripts as supplementary infrastructure (e.g., CMakeLists.txt, various scripts in utils/ and test/ directories).
• For a complete picture, include testing frameworks (represented by test/ directories) to indicate maintenance of functional and unit tests.
• Integration with external systems, such as the Redis pool (found in src/libserver/redis_pool.cxx) for caching and state management.
Happy diagramming!