How shinie19/hybrid-log-classification-system works
Project Identification
• The project is a hybrid log classification system that processes logs using three distinct methods: a regex-based classifier, a Sentence Transformer with Logistic Regression classifier, and an LLM-based classifier.
• It is structured as a full-stack API service (built with FastAPI) that accepts CSV inputs, classifies each log entry, and outputs a CSV file with labels.
Analysis of the File Structure
• Top-Level Files:
• README.md: Provides an overview, usage instructions, and an architecture image reference.
• classify.py: Likely contains the core logic that integrates and directs the classification process.
• processor_bert.py, processor_llm.py, processor_regex.py: These modules encapsulate the different classification approaches.
• server.py: Contains the FastAPI server code which exposes the classification API endpoints.
• requirements.txt: Lists the project’s dependencies.
• Directories:
• models/: Stores the pre-trained models (e.g., the joblib file for the classifier using Sentence Transformers and Logistic Regression).
• resources/: Contains CSV files and possibly additional assets (like the architecture image if the file name were similar).
• training/: Holds training scripts and notebooks (including a dataset and Jupyter notebook) that were used to train the models.
Main Components
a. API Layer:
• FastAPI Server (server.py) that receives requests.
• API endpoints (documented in the README – accessible via /, /docs, /redoc) that allow CSV uploads.
b. Classification Engine:
• Core classification logic (likely integrated via classify.py) that determines which classification methodology to apply per log entry.
• Three processors:
• Regex Processor (processor_regex.py) for simple, predetermined patterns.
• Sentence Transformer + Logistic Regression Processor (processor_bert.py) that uses embeddings and a joblib model from the models/ directory.
• LLM Processor (processor_llm.py) that activates when data is complex or insufficiently labeled.
c. Data Handling:
• Input CSV (expected columns: source, log_message) is parsed on ingestion.
• Output CSV (resources/output.csv) which aggregates classification results.
d. Training & Model Management:
• Training Module (training/ directory with dataset and the Jupyter notebook) shows how models are developed and refined.
• The models directory contains the saved model artifacts deployed by the API.
Relationships and Data Flow
• The client (or user) initiates a request by uploading a CSV file to the API endpoint.
• The FastAPI server (server.py) then invokes the classification module (classify.py) which directs the log entries to one of the three processors based on the log pattern complexity.
• Each processor (whether regex, Sentence Transformer-based, or LLM) processes the log message and returns a classification label.
• The processed results are compiled and an output CSV file is generated and sent as a response.
• There is a clear dependency on the pre-trained model stored in the models directory by the Sentence Transformer processor.
• Additionally, training artifacts from the training folder indicate that model refinement is a separate process from live classification.
Architectural Patterns and Technologies
• The architecture separates concerns between the API layer and the classification logic, following a form of service-oriented design.
• Modular design is apparent in the separation of processor components.
• The use of FastAPI defines a RESTful interface.
• Integration with external libraries (for example, joblib for model serialization, Sentence Transformers, and an LLM interface) is crucial.
• From the API server, represent a “Classification Dispatcher” (the core logic that takes in logs) that routes to three separate modules:
◦ “Regex Processor” – simple pattern match.
◦ “Sentence Transformer + Logistic Regression Processor” – complex data handling using models from the models/ folder.
◦ “LLM Processor” – fallback for logs with insufficient structured training data.
• After processing, indicate a reconvergence where output (with the appended classification labels) is generated as a CSV file.
• Off to one side, show a “Training Environment” block (from the training/ directory) that develops and refines the models, which then feed into the “Persisted Models” in the models/ directory.