How nikhilsanjay25/log_anomly_detection works
This is an HDFS-only log anomaly analysis application. Users upload or paste raw logs, event-trace CSVs, or event sequences; the Streamlit UI or CLI passes a batch to the Coordinator, which identifies the format, preprocesses it, stores medallion layers, detects anomalous sessions, derives root-cause evidence, and produces insights and a report. Follow-up chat and run memory extend the result workflow. The README documents the ML and language-model capabilities; some internal wiring is not shown in the sampled excerpts, so those links are omitted or marked unverified.
Main entry points: coordinator.py Coordinator · streamlit_app.py Streamlit UI · rca.py Root cause analysis · anomaly.py Anomaly detection
Components 21
User interfaces
- Streamlit UI
streamlit_app.py - Command line
log_agents/cli.py - Analysis chat
log_agents/chat.py
Analysis workflow
- Log ingestion
log_agents/ingestion.py - Coordinator
log_agents/agents/coordinator.py - Format identification
log_agents/agents/identification.py - Log preprocessing
log_agents/agents/preprocessing.py - HDFS event semantics
log_agents/hdfs.py - Medallion processing
log_agents/medallion.py
Detection and insights
- Anomaly detection
log_agents/agents/anomaly.py - Transformer FAISS RAG RF
log_agents/ml.py - LoRA second opinion
log_agents/slm.py - Root cause analysis
log_agents/agents/rca.py - RCA knowledge base
log_agents/knowledge.py - Insight generation
log_agents/agents/insight.py - Insight language model
log_agents/slm.py - Model resources
log_agents/resources.py
Run state and output
- Run context
log_agents/context.py - Run memory
log_agents/context.py - Report builder
log_agents/report.py
Other components
- Log analyst
Connections 31
- Log analyst → Streamlit UI: submits logs
README.md - Log analyst → Command line: submits logs
README.md - Streamlit UI → Log ingestion: creates batch
streamlit_app.py - Streamlit UI → Coordinator: runs analysis
streamlit_app.py - Command line → Log ingestion: creates batch
log_agents/cli.py - Command line → Coordinator: runs analysis
log_agents/cli.py - Coordinator → Format identification: executes
log_agents/agents/coordinator.py - Coordinator → Log preprocessing: executes
log_agents/agents/coordinator.py - Coordinator → Medallion processing: executes
log_agents/agents/coordinator.py - Coordinator → Anomaly detection: executes
log_agents/agents/coordinator.py - Coordinator → Root cause analysis: executes
log_agents/agents/coordinator.py - Coordinator → Insight generation: executes
log_agents/agents/coordinator.py - Coordinator → Run context: creates run context
log_agents/agents/coordinator.py - Format identification → HDFS event semantics: uses event format
log_agents/agents/identification.py - Log preprocessing → HDFS event semantics: parses events
log_agents/agents/preprocessing.py - Medallion processing → HDFS event semantics: uses event semantics
log_agents/medallion.py - Anomaly detection → Transformer FAISS RAG RF: runs detection pipeline
- Anomaly detection → LoRA second opinion: requests advisory vote
README.md - Root cause analysis → RCA knowledge base: uses RCA rules
log_agents/agents/rca.py - Anomaly detection → Root cause analysis: provides session evidence
README.md - Root cause analysis → Insight generation: provides causes and evidence
README.md - Insight generation → Insight language model: generates explanations
README.md - Model resources → Transformer FAISS RAG RF: loads pipeline
log_agents/resources.py - Model resources → Insight language model: loads language model
log_agents/resources.py - Coordinator → Report builder: builds report
log_agents/agents/coordinator.py - Coordinator → Run memory: reads and saves runs
log_agents/agents/coordinator.py - Run context → Run memory: stores run summaries
log_agents/context.py - Streamlit UI → Analysis chat: starts follow-up chat
streamlit_app.py - Analysis chat → Insight language model: asks follow-up questions
log_agents/chat.py - Analysis chat → Root cause analysis: retrieves block evidence
log_agents/chat.py - Streamlit UI → Report builder: displays results
README.md