How leonrhapsody/dnslogfilter works
Main Components
• CLI & Entry Points: The tool’s entry point is in main.go, with additional CLI command-specific code in the “cmd” directory (e.g., initConfigFile.go, root.go, version.go). These files handle startup, configuration initialization, version reporting, and parameter parsing.
• Configuration Management: Multiple configuration files (config.yaml, config.html, config.readme) and modules like readConfig.go and cmd/initConfigFile.go indicate a dedicated mechanism to read and parse user configurations.
• Log Processing Module:
• Parsing & Filtering: Source files such as analyze.go, collectFile.go, tree.go, and ip.txt, domain.txt, etc., suggest components that parse DNS log inputs and then filter them based on IP, domain, or query type using custom rules.
• Task-Based Processing: The YAML configuration specifies tasks (e.g., “apt” and “aliyun”) each with separate settings, routes, and file management options (backup, output directory, file size/time limits). Separate functionalities may be implemented in the manager folder (e.g., manager/main.go, manager/readConfig.go) and specific modules (SFTP.go for potential file transfers, zip.go for potential compression).
• Output Handling: The system supports multiple output formats and related processing (e.g., gzipping options, output formatting), as indicated by the readme configurations and file names like analyze.go and status.go. The processing pipeline converts parsed log data into the desired output formats (plain text, JSON, CSV, etc.).
Relationships and Data Flow
• The “Configuration Management” component reads from config.yaml and other related config files to set parameters for how logs should be processed.
• From configuration, the flow continues into the “Log Parsing & Filtering Engine” where files (e.g., DNS log files from an “input” directory as specified) are read, parsed, and filtered based on criteria such as domain names, IP addresses, etc.
• After parsing, the filtered information is handed off to the “Output Handling Module” which formats the data for export (plain text, JSON, CSV). This module also manages file backup (moving processed logs to backup directories) and applies file compression when required (with components like zip.go).
• Any optional or additional components (like SFTP.go for remote file handling or manager tools) can be shown as auxiliary modules that interact with the core log filtering engine.
Architectural Patterns and Design Principles
• Modular Design: The tool is organized into distinct modules (CLI, configuration, parsing, filtering, output, and management) that interact based on defined interfaces.
• Config-Driven Behavior: The use of YAML configuration permits flexibility in log formats, filters, output types, and operational modes (online/offline).
• Separation of Concerns: Each directory and file (e.g., “cmd” for CLI logic, “manager” for specific task management) supports a clear separation between user interaction, configuration, and business logic.
• Scalability Considerations: The configuration allows tuning (e.g., “analyze_threads”) and handling large volumes of DNS log data.
Elements
• Component Labels: Clearly label components such as "CLI Interface (main.go & cmd/)", "Configuration Loader (config.yaml, readConfig modules)", "DNS Log Parsing & Filtering Engine (analyze.go, tree.go, etc.)", and "Output Formatter (format conversion, backup, compression)".