# shinie19/hybrid-log-classification-system architecture

> Architecture diagram of the GitHub repository shinie19/hybrid-log-classification-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shinie19/hybrid-log-classification-system
- Repository: https://github.com/shinie19/hybrid-log-classification-system

## Overview

##### Project Identification
- The project is a hybrid log classification system that processes logs using three distinct methods: a regex-based classifier, a Sentence Transformer with Logistic Regression classifier, and an LLM-based classifier.
- It is structured as a full-stack API service (built with FastAPI) that accepts CSV inputs, classifies each log entry, and outputs a CSV file with labels.

##### Analysis of the File Structure
- Top-Level Files:
- README.md: Provides an overview, usage instructions, and an architecture image reference.
- classify.py: Likely contains the core logic that integrates and directs the classification process.
- processor_bert.py, processor_llm.py, processor_regex.py: These modules encapsulate the different classification approaches.
- server.py: Contains the FastAPI server code which exposes the classification API endpoints.
- requirements.txt: Lists the project’s dependencies.
- Directories:
- models/: Stores the pre-trained models (e.g., the joblib file for the classifier using Sentence Transformers and Logistic Regression).
- resources/: Contains CSV files and possibly additional assets (like the architecture image if the file name were similar).
- training/: Holds training scripts and notebooks (including a dataset and Jupyter notebook) that were used to train the models.

##### Main Components
a. API Layer:
- FastAPI Server (server.py) that receives requests.
- API endpoints (documented in the README – accessible via /, /docs, /redoc) that allow CSV uploads.
b. Classification Engine:
- Core classification logic (likely integrated via classify.py) that determines which classification methodology to apply per log entry.
- Three processors:
- Regex Processor (processor_regex.py) for simple, predetermined patterns.
- Sentence Transformer + Logistic Regression Processor (processor_bert.py) that uses embeddings and a joblib model from the models/ directory.
- LLM Processor (processor_llm.py) that activates when data is complex or insufficiently labeled.
c. Data Handling:
- Input CSV (expected columns: source, log_message) is parsed on ingestion.
- Output CSV (resources/output.csv) which aggregates classification results.
d. Training & Model Management:
- Training Module (training/ directory with dataset and the Jupyter notebook) shows how models are developed and refined.
- The models directory contains the saved model artifacts deployed by the API.

##### Relationships and Data Flow
- The client (or user) initiates a request by uploading a CSV file to the API endpoint.
- The FastAPI server (server.py) then invokes the classification module (classify.py) which directs the log entries to one of the three processors based on the log pattern complexity.
- Each processor (whether regex, Sentence Transformer-based, or LLM) processes the log message and returns a classification label.
- The processed results are compiled and an output CSV file is generated and sent as a response.
- There is a clear dependency on the pre-trained model stored in the models directory by the Sentence Transformer processor.
- Additionally, training artifacts from the training folder indicate that model refinement is a separate process from live classification.

##### Architectural Patterns and Technologies
- The architecture separates concerns between the API layer and the classification logic, following a form of service-oriented design.
- Modular design is apparent in the separation of processor components.
- The use of FastAPI defines a RESTful interface.
- Integration with external libraries (for example, joblib for model serialization, Sentence Transformers, and an LLM interface) is crucial.

- From the API server, represent a “Classification Dispatcher” (the core logic that takes in logs) that routes to three separate modules:
◦ “Regex Processor” – simple pattern match.
◦ “Sentence Transformer + Logistic Regression Processor” – complex data handling using models from the models/ folder.
◦ “LLM Processor” – fallback for logs with insufficient structured training data.
- After processing, indicate a reconvergence where output (with the appended classification labels) is generated as a CSV file.
- Off to one side, show a “Training Environment” block (from the training/ directory) that develops and refines the models, which then feed into the “Persisted Models” in the models/ directory.

## Diagram (Mermaid)

```mermaid
flowchart TD
    Client["Client (CSV Upload)"]:::external
    Client -->|"uploads"| FastAPIServer["FastAPI Server"]:::api

    FastAPIServer -->|"forwards CSV"| Dispatcher["Classification Dispatcher"]:::dispatcher

    subgraph "Classification Processing"
        Dispatcher -->|"routes"| RegexProcessor["Regex Processor"]:::processor
        Dispatcher -->|"routes"| TransformerProcessor["Sentence Transformer+Logistic Regression Processor"]:::processor
        Dispatcher -->|"routes"| LLMProcessor["LLM Processor"]:::processor
    end

    RegexProcessor -->|"returns"| Merge["Merge Results"]:::process
    TransformerProcessor -->|"returns"| Merge
    LLMProcessor -->|"returns"| Merge

    Merge -->|"generates"| OutputCSV["Output CSV File"]:::database

    subgraph "Model Development"
        Training["Training Module"]:::training
        Training -->|"develops"| Models["Persisted Models"]:::database
        Models -->|"supplies model"| TransformerProcessor
    end

    %% Click Events
    click FastAPIServer "https://github.com/shinie19/hybrid-log-classification-system/blob/main/server.py"
    click Dispatcher "https://github.com/shinie19/hybrid-log-classification-system/blob/main/classify.py"
    click RegexProcessor "https://github.com/shinie19/hybrid-log-classification-system/blob/main/processor_regex.py"
    click TransformerProcessor "https://github.com/shinie19/hybrid-log-classification-system/blob/main/processor_bert.py"
    click LLMProcessor "https://github.com/shinie19/hybrid-log-classification-system/blob/main/processor_llm.py"
    click Models "https://github.com/shinie19/hybrid-log-classification-system/tree/main/models/"
    click OutputCSV "https://github.com/shinie19/hybrid-log-classification-system/tree/main/resources/"
    click Training "https://github.com/shinie19/hybrid-log-classification-system/tree/main/training/"

    %% Styles
    classDef external fill:#cff,stroke:#036,stroke-width:2px;
    classDef api fill:#fcf,stroke:#900,stroke-width:2px;
    classDef dispatcher fill:#efe,stroke:#393,stroke-width:2px;
    classDef processor fill:#ffd,stroke:#993,stroke-width:2px;
    classDef process fill:#eef,stroke:#557,stroke-width:2px;
    classDef database fill:#dfd,stroke:#393,stroke-width:2px,stroke-dasharray: 5 5;
    classDef training fill:#fee,stroke:#933,stroke-width:2px;
```
