How oxapavan/loganalysis-llm works
Project type and purpose
• GenAI_LogAnalyzer is a full-stack AI-powered log analysis application built in Python.
• Purpose: Ingest raw IT log files, perform root cause analysis via large language models (Cohere, AWS Bedrock), generate summaries, and integrate with ITSM tools (e.g., ServiceNow).
File structure
• Top-level scripts:
• app.py, main.py – orchestration/entry points
• streamlitUI.py / streamlitUI_showcase.py – Streamlit-based frontend
• OneTimeExecution.py – AWS setup (credentials, infrastructure)
• LLM modules: LLM.py, “LLM Chatbot…” .py/.ipynb – exploratory notebooks / CLI demos
• Summary modules: SmallSummary.py, LargeSummary.py, Chunking.py
• VectorDB.py – wrapper around vector database for embedding storage & retrieval
• utils/bedrock.py – AWS Bedrock integration
• utils/tools_agents.py – connector to external services (e.g., ServiceNow API)
• input/ & output/ – sample log files and report outputs
• requirements.txt / requirement_whole.txt – dependencies
README for additional insights
• Describes AWS prerequisites (OneTimeExecution.py).
• Launch via streamlit run streamlitUI.py.
• Mentions integration plugin for ServiceNow.
Main components and their interactions
a. Streamlit Frontend
– Components: streamlitUI.py (primary), optional showcase.
– Presents upload widget for log files, progress indicators, and displays analysis reports.
b. Orchestration Layer (API & Controllers)
– app.py / main.py handle request routing from UI to backend logic.
– OneTimeExecution.py provisions AWS resources before runtime.
c. Preprocessing & Chunking
– Chunking.py breaks raw logs into token-limited segments.
d. Embedding & Vector Database
– VectorDB.py creates embeddings (via LLM module) and performs similarity search.
e. LLM Processing
– LLM.py abstracts calls to Cohere or AWS Bedrock (via utils/bedrock.py).
– “LLM Chatbot” scripts demonstrate conversational analysis mode.
f. Summarization & Analysis
– SmallSummary.py for quick summaries, LargeSummary.py for in-depth reports, LargeSummary writes final_report.txt.
g. External Integrations
– utils/tools_agents.py handles ServiceNow API calls to push tickets or retrieve context.
h. Data Stores & Outputs
– Local file system (output/*.txt) for interim and final reports.
– (Optionally) vector DB can be hosted in cloud (e.g., AWS memory store, not in file tree but configurable).
Architectural patterns and design principles
• Layered architecture: UI → API/Orchestrator → Domain Services (preprocessing, vector store, LLM) → External Integrations.
• Modular design: each functional area in its own module; easy to extend or swap LLM providers.
• Separation of concerns: chunking, embedding, summarization separated into discrete scripts/classes.
• Unidirectional data flow: user uploads → preprocessing → embedding → analysis → summarization → output/integration.
Actionable next steps
• Lay out the layers horizontally: UI on the left, processing pipeline in the center, external services on the right.
• Every major .py or module in your file tree is represented or grouped logically.