# nikhilvijayakumar/log-guardians architecture

> Architecture diagram of the GitHub repository nikhilvijayakumar/log-guardians, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nikhilvijayakumar/log-guardians
- Repository: https://github.com/nikhilvijayakumar/log-guardians

## Overview

1. **Repository purpose**
   - This is a Python log-analysis pipeline called **Log Guardians**. It ingests raw log files, splits them into chunks, converts them into structured JSON, runs AI-based anomaly detection, and produces a consolidated markdown report.

2. **Primary execution model**
   - The system is organized as a **batch CLI pipeline**, not a web service.
   - The main entrypoint is `src/log/guardians/app/main/main.py`, which orchestrates the full end-to-end flow.
   - Individual stages can also be run independently from the agent scripts.

3. **Core pipeline stages**
   - The documented flow is: **Raw Logs → Chunking → JSON Conversion → Anomaly Detection → Final Report**.
   - Each stage consumes the artifact from the previous stage and writes a new artifact to disk, creating a file-based pipeline boundary between components.

4. **Input boundary**
   - Raw source logs live under `data/logs/` and include multiple platform/profile examples such as `Android_2k.log`, `Apache_2k.log`, `Hadoop_2k.log`, `Linux_2k.log`, `OpenSSH_2k.log`, and others.
   - The active input file and profile are configured in `src/log/guardians/app/main/config/chunker_config.yaml`.

5. **Chunking subsystem**
   - `src/log/guardians/app/features/chunking/chunker.py` is the preprocessing layer.
   - It splits large log files into smaller, manageable chunks according to `chunk_size` and profile-specific parsing rules defined in the YAML config.
   - This subsystem is the first transformation boundary: it turns one raw log stream into many chunk files.

6. **Agent-based processing layer**
   - The repository uses separate agent modules under `src/log/guardians/app/agent/`:
     - `json_converter_agent.py` converts chunked raw logs into structured JSON.
     - `anomaly_detection_agent.py` inspects the structured JSON and identifies anomalies.
     - `report_generator_agent.py` aggregates anomaly findings into the final report.
   - This suggests a loosely coupled, stage-by-stage architecture where each agent can be run as a standalone tool or as part of the orchestrated pipeline.

7. **Shared agent support**
   - `tools.py` likely contains common helper functions used across agents, acting as the shared service layer for repeated operations.
   - `json_cleaner.py` and `log_utils.py` in `src/log/guardians/app/utils/` provide utility boundaries for sanitizing structured output and handling log/file operations.

8. **Output artifacts and storage layout**
   - Pipeline outputs are written under a hidden project-local workspace `.LogGuardians/`.
   - Generated artifacts are separated by stage:
     - `output/logs/{profile}/{log_name}/chunk_*.log` for chunked raw logs
     - `output_json_structured_logs/chunk_*.json` for structured JSON
     - `output_anomalies/chunk_*_anomaly.json` for detected issues
     - `FINAL_ANOMALY_REPORT.md` for the consolidated report
   - This makes the architecture strongly artifact-driven and easy to diagram as a linear data pipeline.

9. **Configuration and profiles**
   - The config file `chunker_config.yaml` drives profile selection and chunking behavior.
   - The repository is designed for multiple log formats via profiles, which is why examples span Linux, Apache, HPC, Hadoop, SSH, Zookeeper, and more.
   - That profile mechanism is the main extensibility point for adding new log types.

10. **AI/LLM dependency boundary**
   - The README states that a Google API key must be configured in `.env`, which indicates that at least the conversion/detection/reporting stages may rely on a Google-hosted AI service.
   - Architecturally, this is an external dependency boundary: the pipeline is local file-based, but the analysis logic appears to call out to an external model/API.

11. **Reporting layer**
   - `report_generator_agent.py` consolidates anomaly outputs into a single markdown deliverable.
   - The report is severity-oriented, with sections like executive summary, critical threats, pattern analysis, detailed findings, and recommendations.
   - This is the final aggregation boundary: many per-chunk findings become one human-readable security report.

12. **Operational shape**
   - The system is meant for offline or scheduled analysis of log files rather than streaming ingestion.
   - Its main architectural characteristics are: **batch processing, disk-backed intermediate artifacts, profile-based parsing, and AI-assisted reasoning**.

13. **What to emphasize in an architecture diagram**
   - Show three major zones:
     - **Inputs**: raw logs and YAML config
     - **Processing pipeline**: chunker → JSON conversion agent → anomaly detection agent → report generator
     - **Outputs**: chunked logs, structured JSON, anomaly JSON, final markdown report
   - Include the external Google API/LLM dependency as a side integration to the agent layer, not as the primary data source.

Main entry points: [`src/log/guardians/app/agent/json_converter_agent.py`](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/json_converter_agent.py) (JSON agent), [`src/log/guardians/app/agent/anomaly_detection_agent.py`](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/anomaly_detection_agent.py) (Anomaly agent), [`src/log/guardians/app/agent/report_generator_agent.py`](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/report_generator_agent.py) (Report agent), [`src/log/guardians/app/agent/tools.py`](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/tools.py) (Shared tools)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Inputs"]
  node_raw_logs[("Raw logs<br/>log corpus")]
  node_chunk_config["Chunk config<br/>yaml config"]
end

subgraph group_pipeline["Processing Pipeline"]
  node_main_cli["Main CLI<br/>orchestrator<br/>[main.py]"]
  node_chunker["Chunker<br/>preprocessor<br/>[chunker.py]"]
  node_json_agent["JSON agent<br/>converter agent"]
  node_anomaly_agent["Anomaly agent<br/>detection agent"]
  node_report_agent["Report agent<br/>reporter agent"]
  node_tools["Shared tools<br/>support library<br/>[tools.py]"]
  node_json_cleaner["JSON cleaner<br/>[json_cleaner.py]"]
  node_log_utils["Log utils<br/>[log_utils.py]"]
end

subgraph group_outputs["Outputs"]
  node_chunked_logs["Chunk files<br/>artifact"]
  node_structured_json["Structured JSON<br/>artifact"]
  node_anomaly_json["Anomaly JSON<br/>artifact"]
  node_final_report["Final report<br/>artifact"]
end

subgraph group_external["External Dependency"]
  node_google_api(("Google API<br/>external AI service"))
end

node_raw_logs -->|"input logs"| node_chunker
node_chunk_config -->|"profile rules"| node_chunker
node_main_cli -->|"orchestrates"| node_chunker
node_chunker -->|"writes"| node_chunked_logs
node_chunked_logs -->|"input chunks"| node_json_agent
node_json_agent -->|"writes"| node_structured_json
node_structured_json -->|"input json"| node_anomaly_agent
node_anomaly_agent -->|"writes"| node_anomaly_json
node_anomaly_json -->|"inputs findings"| node_report_agent
node_report_agent -->|"writes"| node_final_report
node_main_cli -->|"can run"| node_json_agent
node_main_cli -->|"can run"| node_anomaly_agent
node_main_cli -->|"can run"| node_report_agent
node_json_agent -->|"uses"| node_tools
node_anomaly_agent -->|"uses"| node_tools
node_report_agent -->|"uses"| node_tools
node_tools -->|"supports"| node_json_cleaner
node_tools -->|"supports"| node_log_utils
node_json_agent -->|"calls"| node_google_api
node_anomaly_agent -->|"calls"| node_google_api
node_report_agent -->|"calls"| node_google_api

click node_raw_logs "https://github.com/nikhilvijayakumar/log-guardians/tree/main/data/logs"
click node_chunk_config "https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/main/config/chunker_config.yaml"
click node_main_cli "https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/main/main.py"
click node_chunker "https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/features/chunking/chunker.py"
click node_json_agent "https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/json_converter_agent.py"
click node_anomaly_agent "https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/anomaly_detection_agent.py"
click node_report_agent "https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/report_generator_agent.py"
click node_tools "https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/tools.py"
click node_json_cleaner "https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/utils/json_cleaner.py"
click node_log_utils "https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/utils/log_utils.py"
click node_final_report "https://github.com/nikhilvijayakumar/log-guardians/blob/main/FINAL_ANOMALY_REPORT.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_logs,node_chunk_config toneBlue
class node_main_cli,node_chunker,node_json_agent,node_anomaly_agent,node_report_agent,node_tools,node_json_cleaner,node_log_utils toneAmber
class node_chunked_logs,node_structured_json,node_anomaly_json,node_final_report toneMint
class node_google_api toneRose
```

## Components

### Inputs

Raw logs and configuration that seed the batch pipeline.

- [Raw logs](https://github.com/nikhilvijayakumar/log-guardians/tree/main/data/logs): log corpus · Example source logs across multiple platform profiles. · `data/logs`
- [Chunk config](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/main/config/chunker_config.yaml): yaml config · Profile selection and chunking rules for the active run. · `src/log/guardians/app/main/config/chunker_config.yaml`

### Processing Pipeline

Core staged transformation from logs to report.

- [Main CLI](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/main/main.py): orchestrator · Batch entrypoint that coordinates the end-to-end workflow. · `src/log/guardians/app/main/main.py`
- [Chunker](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/features/chunking/chunker.py): preprocessor · Splits raw logs into smaller file-backed chunks. · `src/log/guardians/app/features/chunking/chunker.py`
- [JSON agent](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/json_converter_agent.py): converter agent · Turns chunked logs into structured JSON. · `src/log/guardians/app/agent/json_converter_agent.py`
- [Anomaly agent](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/anomaly_detection_agent.py): detection agent · Inspects structured JSON and flags anomalies. · `src/log/guardians/app/agent/anomaly_detection_agent.py`
- [Report agent](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/report_generator_agent.py): reporter agent · Aggregates anomaly findings into the final markdown report. · `src/log/guardians/app/agent/report_generator_agent.py`
- [Shared tools](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/agent/tools.py): support library · Common helper functions reused across agents. · `src/log/guardians/app/agent/tools.py`
- [JSON cleaner](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/utils/json_cleaner.py): utility · Sanitizes structured output before downstream use. · `src/log/guardians/app/utils/json_cleaner.py`
- [Log utils](https://github.com/nikhilvijayakumar/log-guardians/blob/main/src/log/guardians/app/utils/log_utils.py): utility · File and log handling helpers for pipeline stages. · `src/log/guardians/app/utils/log_utils.py`

### Outputs

Artifacts produced by the pipeline.

- **Chunk files**: artifact · Per-profile chunked log outputs written to disk.
- **Structured JSON**: artifact · Chunk-level structured JSON emitted by conversion.
- **Anomaly JSON**: artifact · Chunk-level anomaly findings written by detection.
- [Final report](https://github.com/nikhilvijayakumar/log-guardians/blob/main/FINAL_ANOMALY_REPORT.md): artifact · Consolidated markdown deliverable for human review. · `FINAL_ANOMALY_REPORT.md`

### External Dependency

Hosted AI/API service used by the agent layer.

- **Google API**: external AI service · External hosted model/API boundary used by agent stages.

## Connections

- Raw logs → Chunker: input logs · The chunker reads raw log files from the repository data set.
- Chunk config → Chunker: profile rules · Chunk size and profile-specific parsing behavior come from YAML config.
- Main CLI → Chunker: orchestrates · The main CLI starts and coordinates chunking.
- Chunker → Chunk files: writes · Chunking materializes raw logs into chunk files on disk.
- Chunk files → JSON agent: input chunks · Structured conversion consumes chunked log artifacts.
- JSON agent → Structured JSON: writes · The conversion agent emits structured JSON per chunk.
- Structured JSON → Anomaly agent: input json · Detection runs on structured JSON artifacts.
- Anomaly agent → Anomaly JSON: writes · Anomaly findings are persisted as JSON artifacts.
- Anomaly JSON → Report agent: inputs findings · The report generator aggregates anomaly outputs.
- Report agent → Final report: writes · The final stage produces a consolidated markdown report.
- Main CLI → JSON agent: can run · Agents can also be executed independently of the orchestrator.
- Main CLI → Anomaly agent: can run · Detection is available as a standalone stage.
- Main CLI → Report agent: can run · Reporting can be invoked independently as well.
- JSON agent → Shared tools: uses · Conversion reuses shared helper functions.
- Anomaly agent → Shared tools: uses · Detection reuses shared helper functions.
- Report agent → Shared tools: uses · Reporting reuses shared helper functions.
- Shared tools → JSON cleaner: supports · Shared helpers coordinate sanitization utilities.
- Shared tools → Log utils: supports · Shared helpers coordinate log and file utilities.
- JSON agent → Google API: calls · Conversion may rely on an external AI/model API.
- Anomaly agent → Google API: calls · Anomaly reasoning may rely on an external AI/model API.
- Report agent → Google API: calls · Report synthesis may use the hosted model boundary.
