# leonrhapsody/dnslogfilter architecture

> Architecture diagram of the GitHub repository leonrhapsody/dnslogfilter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/leonrhapsody/dnslogfilter
- Repository: https://github.com/leonrhapsody/dnslogfilter

## Overview

##### Main Components
- CLI & Entry Points: The tool’s entry point is in main.go, with additional CLI command-specific code in the “cmd” directory (e.g., initConfigFile.go, root.go, version.go). These files handle startup, configuration initialization, version reporting, and parameter parsing.
- Configuration Management: Multiple configuration files (config.yaml, config.html, config.readme) and modules like readConfig.go and cmd/initConfigFile.go indicate a dedicated mechanism to read and parse user configurations.
- Log Processing Module:
- Parsing & Filtering: Source files such as analyze.go, collectFile.go, tree.go, and ip.txt, domain.txt, etc., suggest components that parse DNS log inputs and then filter them based on IP, domain, or query type using custom rules.
- Task-Based Processing: The YAML configuration specifies tasks (e.g., “apt” and “aliyun”) each with separate settings, routes, and file management options (backup, output directory, file size/time limits). Separate functionalities may be implemented in the manager folder (e.g., manager/main.go, manager/readConfig.go) and specific modules (SFTP.go for potential file transfers, zip.go for potential compression).
- Output Handling: The system supports multiple output formats and related processing (e.g., gzipping options, output formatting), as indicated by the readme configurations and file names like analyze.go and status.go. The processing pipeline converts parsed log data into the desired output formats (plain text, JSON, CSV, etc.).

##### Relationships and Data Flow
- The “Configuration Management” component reads from config.yaml and other related config files to set parameters for how logs should be processed.
- From configuration, the flow continues into the “Log Parsing & Filtering Engine” where files (e.g., DNS log files from an “input” directory as specified) are read, parsed, and filtered based on criteria such as domain names, IP addresses, etc.
- After parsing, the filtered information is handed off to the “Output Handling Module” which formats the data for export (plain text, JSON, CSV). This module also manages file backup (moving processed logs to backup directories) and applies file compression when required (with components like zip.go).
- Any optional or additional components (like SFTP.go for remote file handling or manager tools) can be shown as auxiliary modules that interact with the core log filtering engine.

##### Architectural Patterns and Design Principles
- Modular Design: The tool is organized into distinct modules (CLI, configuration, parsing, filtering, output, and management) that interact based on defined interfaces.
- Config-Driven Behavior: The use of YAML configuration permits flexibility in log formats, filters, output types, and operational modes (online/offline).
- Separation of Concerns: Each directory and file (e.g., “cmd” for CLI logic, “manager” for specific task management) supports a clear separation between user interaction, configuration, and business logic.
- Scalability Considerations: The configuration allows tuning (e.g., “analyze_threads”) and handling large volumes of DNS log data.

##### Elements
- Component Labels: Clearly label components such as "CLI Interface (main.go & cmd/)", "Configuration Loader (config.yaml, readConfig modules)", "DNS Log Parsing & Filtering Engine (analyze.go, tree.go, etc.)", and "Output Formatter (format conversion, backup, compression)".

## Diagram (Mermaid)

```mermaid
graph TD
    %% CLI & Entry Points
    subgraph "CLI & Entry Points"
        CLI_Main["main.go"]:::cli
        CLI_Cmd["cmd/ CLI Commands"]:::cli
    end

    %% Configuration Management
    subgraph "Configuration Management"
        Config_Files["Config Files (config.yaml, config.html, config.readme)"]:::config
        Config_Loader["Config Loader (readConfig.go & cmd/initConfigFile.go)"]:::config
        Manager_Config["Manager Config (manager/readConfig.go)"]:::config
        Effective_Config["Effective Config"]:::config
    end

    %% DNS Log Parsing & Filtering Engine
    subgraph "DNS Log Parsing & Filtering Engine"
        Core_Parsing["Core Parsing (analyze.go, tree.go, v6tree.go)"]:::parsing
        Log_Collection["Log Collection (collectFile.go)"]:::parsing
        Lookup_Data["Lookup Data (ip.txt, domain.txt, v6IP.txt)"]:::parsing
    end

    %% Task-Based Processing
    subgraph "Task-Based Processing"
        Manager_Tools["Manager Tools (manager/)"]:::task
        Task_Lists["Task Lists (apt.list, aliyun.list, aliyun.list2)"]:::task
    end

    %% Output Handling
    subgraph "Output Handling"
        Output_Management["Output Management (status.go, run.go, reload.go)"]:::output
        Compression["Compression (zip.go)"]:::output
    end

    %% SFTP File Transfer
    SFTP_Module["SFTP Module (SFTP.go)"]:::sftp

    %% Connections
    CLI_Main -->|"init_config"| Config_Files
    CLI_Cmd  -->|"parse_commands"| Config_Files

    Config_Files -->|"load_configs"| Config_Loader
    Config_Files -->|"load_configs"| Manager_Config
    Config_Loader -->|"merge"| Effective_Config
    Manager_Config -->|"merge"| Effective_Config

    Effective_Config -->|"configure_parser"| Core_Parsing

    Log_Collection -->|"collect_logs"| Core_Parsing
    Lookup_Data   -->|"provide_lookup"| Core_Parsing

    Core_Parsing -->|"process_tasks"| Manager_Tools

    Manager_Tools -->|"manage_tasks"| Task_Lists

    Task_Lists -->|"prepare_output"| Output_Management

    Output_Management -->|"compress_data"| Compression

    Manager_Tools -->|"transfer_files"| SFTP_Module

    %% Click Events
    click CLI_Main "https://github.com/leonrhapsody/dnslogfilter/blob/main/main.go"
    click CLI_Cmd "https://github.com/leonrhapsody/dnslogfilter/tree/main/cmd/"
    click Config_Files "https://github.com/leonrhapsody/dnslogfilter/blob/main/config.yaml"
    click Config_Loader "https://github.com/leonrhapsody/dnslogfilter/blob/main/readConfig.go"
    click Manager_Config "https://github.com/leonrhapsody/dnslogfilter/blob/main/manager/readConfig.go"
    click Core_Parsing "https://github.com/leonrhapsody/dnslogfilter/blob/main/analyze.go"
    click Log_Collection "https://github.com/leonrhapsody/dnslogfilter/blob/main/collectFile.go"
    click Lookup_Data "https://github.com/leonrhapsody/dnslogfilter/blob/main/ip.txt"
    click Manager_Tools "https://github.com/leonrhapsody/dnslogfilter/tree/main/manager/"
    click Task_Lists "https://github.com/leonrhapsody/dnslogfilter/blob/main/apt.list"
    click Output_Management "https://github.com/leonrhapsody/dnslogfilter/blob/main/status.go"
    click Compression "https://github.com/leonrhapsody/dnslogfilter/blob/main/zip.go"
    click SFTP_Module "https://github.com/leonrhapsody/dnslogfilter/blob/main/SFTP.go"

    %% Styles
    classDef cli fill:#e0f7fa,stroke:#006064,stroke-width:2px;
    classDef config fill:#e8eaf6,stroke:#303f9f,stroke-width:2px;
    classDef parsing fill:#f3e5f5,stroke:#8e24aa,stroke-width:2px;
    classDef task fill:#fff3e0,stroke:#ef6c00,stroke-width:2px;
    classDef output fill:#ffebee,stroke:#c62828,stroke-width:2px;
    classDef sftp fill:#e0f2f1,stroke:#00796b,stroke-width:2px;
```
