# tapusaggarwal/loglingo architecture

> Architecture diagram of the GitHub repository tapusaggarwal/loglingo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tapusaggarwal/loglingo
- Repository: https://github.com/tapusaggarwal/loglingo

## Overview

##### Project Type and Purpose
- This is a command-line log data analysis tool that uses multiple AI “agents” (scripts) to:
- Ingest and parse raw log files
- Store/extract data in an SQLite database
- Perform NLP/text-similarity analysis (TF-IDF, cosine similarity)
- Expose a user interface via a CLI entry point (`app.py`)

##### File Structure Analysis
- Top-level directories/files:
- `loganalysis-main-data/data`: raw `.out` log files
- `src`: all Python modules + `app.py` + database file + `requirements.txt`
- `.idea`, `LICENSE`, `README.md`, and demo media for documentation/demo purposes
- Key modules under `src`:
- Data ingestion: `DataExtraction.py`, `wordnetDownload.py`
- Data mapping & helper functions: `mappingFunctions.py`, `DefinedFunctions.py`
- AI agent orchestration: `TwoAgents.py`
- Text processing/similarity: `FunctionCalling.py`
- CLI/orchestration: `app.py`

##### Core Components
a. Input Layer
- Source: Raw log files (`*.out` in `loganalysis-main-data/data` or user-supplied)
b. Data Ingestion Layer
- Module: `DataExtraction.py`
- Actions: parse logs, extract unique tokens, sanitize entries
- Outputs: populates SQLite DB (`log_data.db`)
c. Storage Layer
- Component: SQLite database (`log_data.db`)
d. Data Mapping & Utilities
- Modules: `mappingFunctions.py`, `DefinedFunctions.py`, `wordnetDownload.py`
- Responsibilities: map extracted fields to DB schemas, download WordNet resources, define common helper routines
e. NLP Processing Layer
- Module: `FunctionCalling.py`
- Responsibilities: compute TF-IDF vectors, cosine similarity, advanced text-analysis
- Dependencies: scikit-learn, NumPy
f. Agent Orchestration
- Module: `TwoAgents.py`
- Responsibilities: coordinate AI-driven scripts that may write custom queries or functions back into the DB, dynamically call other modules
g. Presentation / CLI Layer
- Entry point: `app.py`
- Responsibilities: parse user commands, route requests to DataExtraction, FunctionCalling, TwoAgents, then display results

##### Architectural Patterns & Principles
- Layered pipeline: clear separation between ingestion, storage, processing, and presentation
- Modular agents: `TwoAgents.py` implements a plugin-style orchestration of AI-driven tasks
- Single responsibility: each module has a well-defined role (parsing vs. NLP vs. mapping)
- Dependency inversion: high-level `app.py` depends on abstractions in lower modules

##### Final Recommendations
- Be as granular as needed: if a module has sub-steps (regex parsing vs. DB schema creation), show them as sub-components.
- Error paths (e.g., DB connection failures) if relevant for operational clarity.
- Iterate: validate the draft with the team and refine naming/flows.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Input Layer
    subgraph "Input Layer"
        Raw["Raw Log Files (*.out)"]:::external
    end

    %% Ingestion & Storage
    subgraph "Ingestion & Storage"
        DataExt["DataExtraction.py"]:::ingestion
        DB[(SQLite Database)]:::storage
    end

    %% Mapping & Utilities
    subgraph "Data Mapping & Utilities"
        MapFunc["mappingFunctions.py"]:::util
        DefFunc["DefinedFunctions.py"]:::util
        WordNetDL["wordnetDownload.py"]:::util
    end

    %% Processing & Agents
    subgraph "NLP Processing & Agent Orchestration"
        NLP["FunctionCalling.py"]:::processing
        Agents["TwoAgents.py"]:::agents
        scikit["scikit-learn"]:::external
        numpy["NumPy"]:::external
        WordNetLib["WordNet"]:::external
    end

    %% CLI / Presentation
    subgraph "CLI / Presentation"
        CLI["app.py"]:::cli
    end

    %% Connections
    Raw -->|"ingest logs"| DataExt
    DataExt -->|"writes / reads"| DB
    DB -->|"schema mapping"| MapFunc
    MapFunc -->|"reads / writes"| DB
    DB -->|"helper routines"| DefFunc
    DefFunc -->|"uses schema"| DB
    WordNetLib -->|"downloads resources"| WordNetDL
    WordNetDL -->|"prepares WordNet data"| DB
    DB -->|"provides data for NLP"| NLP
    NLP -->|"stores results"| DB
    scikit -->|"provides algorithms"| NLP
    numpy -->|"provides math"| NLP
    NLP -->|"called by"| Agents
    DataExt -->|"called by"| Agents
    Agents -->|"coordinates modules"| DB
    CLI -->|"invokes ingestion"| DataExt
    CLI -->|"invokes NLP"| NLP
    CLI -->|"invokes agents"| Agents
    CLI -->|"queries"| DB

    %% Click Events
    click Raw "https://github.com/tapusaggarwal/loglingo/blob/main/loganalysis-main-data/data/*.out"
    click DataExt "https://github.com/tapusaggarwal/loglingo/blob/main/src/DataExtraction.py"
    click DB "https://github.com/tapusaggarwal/loglingo/blob/main/src/log_data.db"
    click MapFunc "https://github.com/tapusaggarwal/loglingo/blob/main/src/mappingFunctions.py"
    click DefFunc "https://github.com/tapusaggarwal/loglingo/blob/main/src/DefinedFunctions.py"
    click WordNetDL "https://github.com/tapusaggarwal/loglingo/blob/main/src/wordnetDownload.py"
    click NLP "https://github.com/tapusaggarwal/loglingo/blob/main/src/FunctionCalling.py"
    click Agents "https://github.com/tapusaggarwal/loglingo/blob/main/src/TwoAgents.py"
    click CLI "https://github.com/tapusaggarwal/loglingo/blob/main/src/app.py"

    %% Styles
    classDef ingestion fill:#cce5ff,stroke:#004085,color:#004085;
    classDef storage fill:#d4edda,stroke:#155724,color:#155724;
    classDef util fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef processing fill:#fff3cd,stroke:#856404,color:#856404;
    classDef agents fill:#f5c6cb,stroke:#721c24,color:#721c24;
    classDef cli fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef external fill:#d1ecf1,stroke:#0c5460,color:#0c5460;
```
