How shaz-cy/hybridlogiq-smart-log-anomaly-explorer works
Project Type & Purpose
• Full-stack, end-to-end anomaly detection platform.
• Backend: Python/Flask API performing log ingestion, NLP embedding (RoBERTa), autoencoder scoring, SOM clustering.
• Frontend: React.js + Tailwind CSS dashboard for uploading logs and visualizing results in real time.
File Structure
• Top level:
• AnomalyBackend/ – Flask app, data pipelines, model artifacts, utilities
• anomaly-dashboard/ – React dashboard code, components, chart libraries
• README.md, LICENSE
• AnomalyBackend subdirectories:
• models/ – trained PyTorch autoencoders, scalers, SOM pickles, metadata
• training_data/, uploads/, results/ – raw logs, generated training logs, anomaly output JSON
• Frontend:
• src/components/ – uploader, charts, tables, status & insights views
• src/components/charts/ – reusable chart components
• public/index.html, tailwind.config.js, package.json
Key Architecture & Patterns
• Monolithic Flask service exposing REST endpoints (e.g., POST /upload, GET /results).
• Pipeline pattern in backend: FileHandler → Embedder (RoBERTa) → Autoencoder → SOM → Results writer.
• React component hierarchy for UI: top-level App → pages (FileUploader, TrainingUploader) → visualization components.
Main Components
a. Frontend (React Dashboard)
• Browser UI
• Components: FileUploader, TrainingUploader, Visualizations (charts, tables)
• Data flow: HTTP(S) requests to Flask API
b. API Gateway / Flask Backend
• Endpoints:
• /upload/logs → FileHandler parses, writes to AnomalyBackend/uploads
• /upload/training → triggers clean_training_data & model re-training (optional)
• Modules:
• FileHandler (file_handler.py)
• NLP Embedder (HuggingFace RoBERTa)
• Autoencoder (PyTorch)
• Scaler (scikit-learn)
• SOM (MiniSom)
• Visualizer (generates JSON results in AnomalyBackend/results)
c. Data Stores & Artifacts
• File system directories: uploads/, training_data/, results/
• models/: persisted .pt, .save, .pkl files + metadata
d. External Dependencies
• HuggingFace Transformers API
• PyTorch compute environment (CPU/GPU)
• scikit-learn library
• Node/NPM for frontend dev server
Relationships & Data Flow
1. User uploads log file via React → HTTP POST to Flask
2. Flask FileHandler saves raw logs → triggers pipeline
3. Embedder module transforms logs into vector embeddings
4. Scaler normalizes features → passes to autoencoder
5. Autoencoder computes reconstruction error = anomaly score
6. SOM clusters anomalies → assigns BMU and cluster IDs
7. Visualizer converts outputs to JSON (anomalies.json, result_*.json)
8. Frontend polls or requests GET /results → renders charts & tables
Architectural Patterns & Principles
• Separation of concerns: distinct modules for I/O, embedding, scoring, clustering, visualization
• Pipeline/chain-of-responsibility for data processing
• Stateless API calls; state persisted as files
• Component-based UI with reusable chart-building blocks