How nikhilvijayakumar/log-guardians works
1. Repository purpose
- This is a Python log-analysis pipeline called Log Guardians. It ingests raw log files, splits them into chunks, converts them into structured JSON, runs AI-based anomaly detection, and produces a consolidated markdown report.
2. Primary execution model
- The system is organized as a batch CLI pipeline, not a web service.
- The main entrypoint is src/log/guardians/app/main/main.py, which orchestrates the full end-to-end flow.
- Individual stages can also be run independently from the agent scripts.
3. Core pipeline stages
- The documented flow is: Raw Logs → Chunking → JSON Conversion → Anomaly Detection → Final Report.
- Each stage consumes the artifact from the previous stage and writes a new artifact to disk, creating a file-based pipeline boundary between components.
4. Input boundary
- Raw source logs live under data/logs/ and include multiple platform/profile examples such as Android_2k.log, Apache_2k.log, Hadoop_2k.log, Linux_2k.log, OpenSSH_2k.log, and others.
- The active input file and profile are configured in src/log/guardians/app/main/config/chunker_config.yaml.
5. Chunking subsystem
- src/log/guardians/app/features/chunking/chunker.py is the preprocessing layer.
- It splits large log files into smaller, manageable chunks according to chunk_size and profile-specific parsing rules defined in the YAML config.
- This subsystem is the first transformation boundary: it turns one raw log stream into many chunk files.
6. Agent-based processing layer
- The repository uses separate agent modules under src/log/guardians/app/agent/:
- json_converter_agent.py converts chunked raw logs into structured JSON.
- anomaly_detection_agent.py inspects the structured JSON and identifies anomalies.
- report_generator_agent.py aggregates anomaly findings into the final report.
- This suggests a loosely coupled, stage-by-stage architecture where each agent can be run as a standalone tool or as part of the orchestrated pipeline.
7. Shared agent support
- tools.py likely contains common helper functions used across agents, acting as the shared service layer for repeated operations.
- json_cleaner.py and log_utils.py in src/log/guardians/app/utils/ provide utility boundaries for sanitizing structured output and handling log/file operations.
8. Output artifacts and storage layout
- Pipeline outputs are written under a hidden project-local workspace .LogGuardians/.
- Generated artifacts are separated by stage:
- output/logs/{profile}/{log_name}/chunk_*.log for chunked raw logs
- output_json_structured_logs/chunk_*.json for structured JSON
- output_anomalies/chunk_*_anomaly.json for detected issues
- FINAL_ANOMALY_REPORT.md for the consolidated report
- This makes the architecture strongly artifact-driven and easy to diagram as a linear data pipeline.
9. Configuration and profiles
- The config file chunker_config.yaml drives profile selection and chunking behavior.
- The repository is designed for multiple log formats via profiles, which is why examples span Linux, Apache, HPC, Hadoop, SSH, Zookeeper, and more.
- That profile mechanism is the main extensibility point for adding new log types.
10. AI/LLM dependency boundary
- The README states that a Google API key must be configured in .env, which indicates that at least the conversion/detection/reporting stages may rely on a Google-hosted AI service.
- Architecturally, this is an external dependency boundary: the pipeline is local file-based, but the analysis logic appears to call out to an external model/API.
11. Reporting layer
- report_generator_agent.py consolidates anomaly outputs into a single markdown deliverable.
- The report is severity-oriented, with sections like executive summary, critical threats, pattern analysis, detailed findings, and recommendations.
- This is the final aggregation boundary: many per-chunk findings become one human-readable security report.
12. Operational shape
- The system is meant for offline or scheduled analysis of log files rather than streaming ingestion.
- Its main architectural characteristics are: batch processing, disk-backed intermediate artifacts, profile-based parsing, and AI-assisted reasoning.
13. What to emphasize in an architecture diagram
- Show three major zones:
- Inputs: raw logs and YAML config
- Processing pipeline: chunker → JSON conversion agent → anomaly detection agent → report generator
- Outputs: chunked logs, structured JSON, anomaly JSON, final markdown report
- Include the external Google API/LLM dependency as a side integration to the agent layer, not as the primary data source.
Main entry points: json_converter_agent.py JSON agent · anomaly_detection_agent.py Anomaly agent · report_generator_agent.py Report agent · tools.py Shared tools
Components 15
Inputs
Raw logs and configuration that seed the batch pipeline.
- Raw logs · Example source logs across multiple platform profiles.
data/logs - Chunk config · Profile selection and chunking rules for the active run.
src/log/guardians/app/main/config/chunker_config.yaml
Processing Pipeline
Core staged transformation from logs to report.
- Main CLI · Batch entrypoint that coordinates the end-to-end workflow.
src/log/guardians/app/main/main.py - Chunker · Splits raw logs into smaller file-backed chunks.
src/log/guardians/app/features/chunking/chunker.py - JSON agent · Turns chunked logs into structured JSON.
src/log/guardians/app/agent/json_converter_agent.py - Anomaly agent · Inspects structured JSON and flags anomalies.
src/log/guardians/app/agent/anomaly_detection_agent.py - Report agent · Aggregates anomaly findings into the final markdown report.
src/log/guardians/app/agent/report_generator_agent.py - Shared tools · Common helper functions reused across agents.
src/log/guardians/app/agent/tools.py - JSON cleaner · Sanitizes structured output before downstream use.
src/log/guardians/app/utils/json_cleaner.py - Log utils · File and log handling helpers for pipeline stages.
src/log/guardians/app/utils/log_utils.py
Outputs
Artifacts produced by the pipeline.
- Chunk files · Per-profile chunked log outputs written to disk.
- Structured JSON · Chunk-level structured JSON emitted by conversion.
- Anomaly JSON · Chunk-level anomaly findings written by detection.
- Final report · Consolidated markdown deliverable for human review.
FINAL_ANOMALY_REPORT.md
External Dependency
Hosted AI/API service used by the agent layer.
- Google API · External hosted model/API boundary used by agent stages.
Connections 21
- Raw logs → Chunker: input logs · The chunker reads raw log files from the repository data set.
- Chunk config → Chunker: profile rules · Chunk size and profile-specific parsing behavior come from YAML config.
- Main CLI → Chunker: orchestrates · The main CLI starts and coordinates chunking.
- Chunker → Chunk files: writes · Chunking materializes raw logs into chunk files on disk.
- Chunk files → JSON agent: input chunks · Structured conversion consumes chunked log artifacts.
- JSON agent → Structured JSON: writes · The conversion agent emits structured JSON per chunk.
- Structured JSON → Anomaly agent: input json · Detection runs on structured JSON artifacts.
- Anomaly agent → Anomaly JSON: writes · Anomaly findings are persisted as JSON artifacts.
- Anomaly JSON → Report agent: inputs findings · The report generator aggregates anomaly outputs.
- Report agent → Final report: writes · The final stage produces a consolidated markdown report.
- Main CLI → JSON agent: can run · Agents can also be executed independently of the orchestrator.
- Main CLI → Anomaly agent: can run · Detection is available as a standalone stage.
- Main CLI → Report agent: can run · Reporting can be invoked independently as well.
- JSON agent → Shared tools: uses · Conversion reuses shared helper functions.
- Anomaly agent → Shared tools: uses · Detection reuses shared helper functions.
- Report agent → Shared tools: uses · Reporting reuses shared helper functions.
- Shared tools → JSON cleaner: supports · Shared helpers coordinate sanitization utilities.
- Shared tools → Log utils: supports · Shared helpers coordinate log and file utilities.
- JSON agent → Google API: calls · Conversion may rely on an external AI/model API.
- Anomaly agent → Google API: calls · Anomaly reasoning may rely on an external AI/model API.
- Report agent → Google API: calls · Report synthesis may use the hosted model boundary.