# nikhilsanjay25/log_anomly_detection architecture

> Architecture diagram of the GitHub repository nikhilsanjay25/log_anomly_detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nikhilsanjay25/log_anomly_detection
- Repository: https://github.com/nikhilsanjay25/log_anomly_detection

## Overview

This is an HDFS-only log anomaly analysis application. Users upload or paste raw logs, event-trace CSVs, or event sequences; the Streamlit UI or CLI passes a batch to the Coordinator, which identifies the format, preprocesses it, stores medallion layers, detects anomalous sessions, derives root-cause evidence, and produces insights and a report. Follow-up chat and run memory extend the result workflow. The README documents the ML and language-model capabilities; some internal wiring is not shown in the sampled excerpts, so those links are omitted or marked unverified.

Main entry points: [`log_agents/agents/coordinator.py`](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/coordinator.py) (Coordinator), [`streamlit_app.py`](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/streamlit_app.py) (Streamlit UI), [`log_agents/agents/rca.py`](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/rca.py) (Root cause analysis), [`log_agents/agents/anomaly.py`](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/anomaly.py) (Anomaly detection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User interfaces"]
  node_streamlit["Streamlit UI<br/>[streamlit_app.py]"]
  node_cli["Command line<br/>[cli.py]"]
  node_chat["Analysis chat<br/>[chat.py]"]
end

subgraph group_workflow["Analysis workflow"]
  node_ingestion["Log ingestion<br/>[ingestion.py]"]
  node_coordinator["Coordinator<br/>[coordinator.py]"]
  node_identification["Format identification<br/>[identification.py]"]
  node_preprocessing["Log preprocessing<br/>[preprocessing.py]"]
  node_hdfs["HDFS event semantics<br/>[hdfs.py]"]
  node_medallion["Medallion processing<br/>[medallion.py]"]
end

subgraph group_intelligence["Detection and insights"]
  node_detection["Anomaly detection<br/>[anomaly.py]"]
  node_ml["Transformer FAISS RAG RF<br/>[ml.py]"]
  node_lora["LoRA second opinion<br/>[slm.py]"]
  node_rca["Root cause analysis<br/>[rca.py]"]
  node_knowledge["RCA knowledge base<br/>[knowledge.py]"]
  node_insight["Insight generation<br/>[insight.py]"]
  node_insight_slm["Insight language model<br/>[slm.py]"]
  node_resources["Model resources<br/>[resources.py]"]
end

subgraph group_state["Run state and output"]
  node_context["Run context<br/>[context.py]"]
  node_memory[("Run memory<br/>[context.py]")]
  node_report["Report builder<br/>[report.py]"]
end

node_user(("Log analyst"))

node_user -->|"submits logs"| node_streamlit
node_user -->|"submits logs"| node_cli
node_streamlit -->|"creates batch"| node_ingestion
node_streamlit -->|"runs analysis"| node_coordinator
node_cli -->|"creates batch"| node_ingestion
node_cli -->|"runs analysis"| node_coordinator
node_coordinator -->|"executes"| node_identification
node_coordinator -->|"executes"| node_preprocessing
node_coordinator -->|"executes"| node_medallion
node_coordinator -->|"executes"| node_detection
node_coordinator -->|"executes"| node_rca
node_coordinator -->|"executes"| node_insight
node_coordinator -->|"creates run context"| node_context
node_identification -->|"uses event format"| node_hdfs
node_preprocessing -->|"parses events"| node_hdfs
node_medallion -->|"uses event semantics"| node_hdfs
node_detection -.->|"runs detection pipeline"| node_ml
node_detection -.->|"requests advisory vote"| node_lora
node_rca -->|"uses RCA rules"| node_knowledge
node_detection -->|"provides session evidence"| node_rca
node_rca -->|"provides causes and evidence"| node_insight
node_insight -.->|"generates explanations"| node_insight_slm
node_resources -->|"loads pipeline"| node_ml
node_resources -->|"loads language model"| node_insight_slm
node_coordinator -->|"builds report"| node_report
node_coordinator -->|"reads and saves runs"| node_memory
node_context -->|"stores run summaries"| node_memory
node_streamlit -->|"starts follow-up chat"| node_chat
node_chat -->|"asks follow-up questions"| node_insight_slm
node_chat -->|"retrieves block evidence"| node_rca
node_streamlit -->|"displays results"| node_report

click node_streamlit "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/streamlit_app.py"
click node_cli "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/cli.py"
click node_ingestion "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/ingestion.py"
click node_coordinator "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/coordinator.py"
click node_identification "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/identification.py"
click node_preprocessing "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/preprocessing.py"
click node_hdfs "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/hdfs.py"
click node_medallion "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/medallion.py"
click node_detection "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/anomaly.py"
click node_ml "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/ml.py"
click node_lora "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/slm.py"
click node_rca "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/rca.py"
click node_knowledge "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/knowledge.py"
click node_insight "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/insight.py"
click node_insight_slm "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/slm.py"
click node_resources "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/resources.py"
click node_context "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/context.py"
click node_memory "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/context.py"
click node_report "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/report.py"
click node_chat "https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/chat.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_streamlit,node_cli,node_chat toneBlue
class node_ingestion,node_coordinator,node_identification,node_preprocessing,node_hdfs,node_medallion toneAmber
class node_detection,node_ml,node_lora,node_rca,node_knowledge,node_insight,node_insight_slm,node_resources toneMint
class node_context,node_memory,node_report toneRose
class node_user toneIndigo
```

## Components

### User interfaces

- [Streamlit UI](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/streamlit_app.py): `streamlit_app.py`
- [Command line](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/cli.py): `log_agents/cli.py`
- [Analysis chat](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/chat.py): `log_agents/chat.py`

### Analysis workflow

- [Log ingestion](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/ingestion.py): `log_agents/ingestion.py`
- [Coordinator](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/coordinator.py): `log_agents/agents/coordinator.py`
- [Format identification](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/identification.py): `log_agents/agents/identification.py`
- [Log preprocessing](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/preprocessing.py): `log_agents/agents/preprocessing.py`
- [HDFS event semantics](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/hdfs.py): `log_agents/hdfs.py`
- [Medallion processing](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/medallion.py): `log_agents/medallion.py`

### Detection and insights

- [Anomaly detection](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/anomaly.py): `log_agents/agents/anomaly.py`
- [Transformer FAISS RAG RF](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/ml.py): `log_agents/ml.py`
- [LoRA second opinion](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/slm.py): `log_agents/slm.py`
- [Root cause analysis](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/rca.py): `log_agents/agents/rca.py`
- [RCA knowledge base](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/knowledge.py): `log_agents/knowledge.py`
- [Insight generation](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/agents/insight.py): `log_agents/agents/insight.py`
- [Insight language model](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/slm.py): `log_agents/slm.py`
- [Model resources](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/resources.py): `log_agents/resources.py`

### Run state and output

- [Run context](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/context.py): `log_agents/context.py`
- [Run memory](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/context.py): `log_agents/context.py`
- [Report builder](https://github.com/nikhilsanjay25/log_anomly_detection/blob/main/log_agents/report.py): `log_agents/report.py`

### Other components

- **Log analyst**

## Connections

- Log analyst → Streamlit UI: submits logs (evidence: `README.md`)
- Log analyst → Command line: submits logs (evidence: `README.md`)
- Streamlit UI → Log ingestion: creates batch (evidence: `streamlit_app.py`)
- Streamlit UI → Coordinator: runs analysis (evidence: `streamlit_app.py`)
- Command line → Log ingestion: creates batch (evidence: `log_agents/cli.py`)
- Command line → Coordinator: runs analysis (evidence: `log_agents/cli.py`)
- Coordinator → Format identification: executes (evidence: `log_agents/agents/coordinator.py`)
- Coordinator → Log preprocessing: executes (evidence: `log_agents/agents/coordinator.py`)
- Coordinator → Medallion processing: executes (evidence: `log_agents/agents/coordinator.py`)
- Coordinator → Anomaly detection: executes (evidence: `log_agents/agents/coordinator.py`)
- Coordinator → Root cause analysis: executes (evidence: `log_agents/agents/coordinator.py`)
- Coordinator → Insight generation: executes (evidence: `log_agents/agents/coordinator.py`)
- Coordinator → Run context: creates run context (evidence: `log_agents/agents/coordinator.py`)
- Format identification → HDFS event semantics: uses event format (evidence: `log_agents/agents/identification.py`)
- Log preprocessing → HDFS event semantics: parses events (evidence: `log_agents/agents/preprocessing.py`)
- Medallion processing → HDFS event semantics: uses event semantics (evidence: `log_agents/medallion.py`)
- Anomaly detection → Transformer FAISS RAG RF: runs detection pipeline
- Anomaly detection → LoRA second opinion: requests advisory vote (evidence: `README.md`)
- Root cause analysis → RCA knowledge base: uses RCA rules (evidence: `log_agents/agents/rca.py`)
- Anomaly detection → Root cause analysis: provides session evidence (evidence: `README.md`)
- Root cause analysis → Insight generation: provides causes and evidence (evidence: `README.md`)
- Insight generation → Insight language model: generates explanations (evidence: `README.md`)
- Model resources → Transformer FAISS RAG RF: loads pipeline (evidence: `log_agents/resources.py`)
- Model resources → Insight language model: loads language model (evidence: `log_agents/resources.py`)
- Coordinator → Report builder: builds report (evidence: `log_agents/agents/coordinator.py`)
- Coordinator → Run memory: reads and saves runs (evidence: `log_agents/agents/coordinator.py`)
- Run context → Run memory: stores run summaries (evidence: `log_agents/context.py`)
- Streamlit UI → Analysis chat: starts follow-up chat (evidence: `streamlit_app.py`)
- Analysis chat → Insight language model: asks follow-up questions (evidence: `log_agents/chat.py`)
- Analysis chat → Root cause analysis: retrieves block evidence (evidence: `log_agents/chat.py`)
- Streamlit UI → Report builder: displays results (evidence: `README.md`)
