How dogoncouch/logesp works
Main Components
• Web Frontend & Django Project:
• The primary Django project is located in the LogESP directory, which includes the standard Django files (settings.py, urls.py, views.py, wsgi.py).
• The three major Django apps – SIEM, Risk, and Asset Management (hwam) – provide the core functionalities: SIEM for log event management, risk for risk management, and hwam for asset management.
• Background Daemons:
• The “daemons” directory includes submodules for cleaner, parser, and sentry.
• The parser and sentry daemons are responsible for processing incoming log events and applying rules (including limit and list rules) to generate alerts or events.
• The cleaner daemon likely handles routine maintenance tasks such as purging or archiving logs.
• Configuration & Deployment:
• The “config” folder contains various configuration files for the database (db.conf.example), web servers (nginx configuration, uwsgi parameters), syslog (rsyslog server and client configurations) and parsing (parser.conf.example).
• The use of Makefile, scripts (install.sh, service files) and documentation in the “docs” directory shows that deployment automation and configuration management are key parts of the system.
• External Interfaces and Infrastructure:
• NGINX, uWSGI, and rsyslog configurations indicate that LogESP is designed to sit behind a web server that uses uWSGI to serve the Django application.
• Log sources send syslog messages to LogESP—handled externally by configured syslog daemons—and these logs are processed by the background daemons.
Relationships & Interactions Between Components
• Client – Web Frontend:
• Users interact via browsers that communicate with the Django web interface. The interface (templates, forms) is provided by both the core LogESP project and the individual apps.
• Django Server – Application Logic:
• The Django project (LogESP along with apps: hwam, risk, siem) implements the controller logic and the MVC pattern.
• These apps interact with the underlying database (configured externally) to store and retrieve log events, risk data, and asset information.
• Background Processing:
• The background daemons (cleaner, parser, sentry) run as separate processes. They continuously handle incoming and stored log data, process them (via core logic in their respective packages), and then update the database/application state.
• External Services & Infrastructure:
• NGINX acts as the reverse proxy and load balancer that passes requests to uWSGI, which in turn hands them off to Django.
• rsyslog components (configured in the “config/rsyslog” directory) are responsible for forwarding log data from network sources into LogESP’s processing pipeline.
Architectural Patterns & Technologies
• MVC / MVT Pattern:
• Modular Separation:
• Separation between the core web application (Django project and apps) and the independent background daemons.
• Unix Philosophy & Simplicity:
• The overall design emphasizes minimalism and independent components that do one thing (e.g., parsing, cleaning, rule evaluation).
• Deployment & Integration:
• Role of configuration (NGINX, uWSGI, rsyslog) in tying the components together.