# danpung2/logpilot-lite architecture

> Architecture diagram of the GitHub repository danpung2/logpilot-lite, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/danpung2/logpilot-lite
- Repository: https://github.com/danpung2/logpilot-lite

## Overview

##### Project type and purpose
- This is a mono-repo for “LogPilot-Lite,” a lightweight log collection system offering both a REST API server, a gRPC server, and a standalone client library.
- Purpose: collect, store, and stream application logs via HTTP or gRPC, storing them either in plain files or a SQLite database.

##### File structure
- Top-level packages:
- logpilot-rest-server/ – Express-based REST API
- logpilot-grpc-server/ – gRPC API
- logpilot-lite-client/ – npm-publishable client library
- shared/ – shared constants, schemas, middleware, and storage services (file & SQLite) consumed by both servers and bundled into the client.
- Root files: Dockerfile, start-server.ts (switches mode by LOGPILOT_MODE), .env for configuration.

##### Key README insights
- Single Docker image can run in “rest” or “grpc” mode via an environment variable.
- Two storage backends: file write/read and SQLite.
- Client supports SendLog and offset-based ConsumeLog flows.

- Client Application
– logpilot-lite-client modules: Producer, Consumer, gRPCClient (for gRPC mode), HTTPClient (for REST—e.g., Axios/Fetch)
– OffsetStore (tracks read offsets in SQLite or in-memory)
- API Server
– REST Server (Express routes in logpilot-rest-server/src/routes/logs.ts)
– gRPC Server (grpcServer.ts in logpilot-grpc-server/src)
- Shared Library (imported by both servers & client)
– Validation middleware (shared/middleware)
– Schemas (shared/schemas/logEntry)
– Services
- FileReader/FileWriter
- SQLiteReader/SQLiteWriter
- Storage Backends
– Log files on disk (./data/*.log)
– SQLite database (./data/logs.db)
- Orchestration
– start-server.ts selects mode, spins up chosen server
– Docker container with env var LOGPILOT_MODE

1. Client → API Server (Client.PerformSendLog → REST POST /logs or gRPC SendLog RPC)
2. API Server applies Validation → Serializes request into shared/logEntry schema
3. Based on “storage” field → Shared Service layer writes to file or SQLite
4. Consumer client → API Server (REST GET /logs?offset=… or gRPC StreamLogs RPC) → Shared Service reads from file/SQLite → returns log entries

c. Architectural patterns & principles
- Layered architecture:
- Presentation (Client / API endpoints)
- Application logic (Validation middleware → Controllers/Handlers)
- Domain services (shared/services for persistence)
- Mode switch as strategy pattern: one process, two interchangeable server implementations
- Code reuse via a shared package

d. Relevant technologies
- Node.js + TypeScript
- Express (REST)
- @grpc/grpc-js + proto definitions (gRPC)
- SQLite (via shared/services)
- Docker

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestration
    subgraph "Orchestration"
        Decision{LOGPILOT_MODE?}:::orchestration
        StartServer["start-server.ts"]:::orchestration
    end

    Decision -->|rest| RestServer
    Decision -->|grpc| GRPCEntry
    StartServer --> Decision

    %% Client Library
    subgraph "Client Library" 
        Producer["Producer"]:::client
        Consumer["Consumer"]:::client
        HTTPClient["HTTPClient"]:::client
        GRPCClient["gRPCClient"]:::client
        OffsetStore["OffsetStore"]:::client
        Index["Public API (index.ts)"]:::client
    end

    %% API Servers
    subgraph "API Servers"
        RestServer["REST Server"]:::server
        LogsController["Logs Controller"]:::server
        GRPCEntry["gRPC Server Entry"]:::server
        GRPCBootstrap["gRPC Server Bootstrap"]:::server
    end

    %% Shared Library
    subgraph "Shared Library"
        ValidationHTTP["HTTP Validation Middleware"]:::shared
        ValidationGRPC["gRPC Validation Middleware"]:::shared
        Schema["LogEntry Schema"]:::shared
        Constants["Validation Constants"]:::shared
        FileWriter["FileWriter Service"]:::shared
        FileReader["FileReader Service"]:::shared
        SQLiteWriter["SQLiteWriter Service"]:::shared
        SQLiteReader["SQLiteReader Service"]:::shared
        DBSetup["SQLite DB Setup"]:::shared
    end

    %% Storage Backends
    subgraph "Storage Backends"
        FileStorage["Log Files (.log)"]:::storage
        SQLiteDB["SQLite DB (logs.db)"]:::storage
    end

    %% Client to Server send flow
    Producer -->|uses HTTPClient| HTTPClient
    Producer -->|uses gRPCClient| GRPCClient

    HTTPClient -->|POST /logs| RestServer
    GRPCClient -->|SendLog RPC| GRPCEntry

    RestServer -->|calls LogsController| LogsController
    LogsController -->|validate| ValidationHTTP
    GRPCEntry -->|registers service| GRPCBootstrap
    GRPCBootstrap -->|validate| ValidationGRPC

    ValidationHTTP -->|schema| Schema
    ValidationGRPC -->|schema| Schema
    Schema --> Constants

    %% Write flows
    ValidationHTTP -->|writes| FileWriter
    ValidationHTTP -->|writes| SQLiteWriter
    ValidationGRPC -->|writes| FileWriter
    ValidationGRPC -->|writes| SQLiteWriter

    FileWriter -->|writes to| FileStorage
    SQLiteWriter -->|writes to| SQLiteDB
    DBSetup --> SQLiteDB

    %% Client to Server consume flow
    Consumer -->|uses HTTPClient| HTTPClient
    Consumer -->|uses gRPCClient| GRPCClient

    HTTPClient -->|GET /logs| RestServer
    GRPCClient -->|StreamLogs RPC| GRPCEntry

    RestServer -->|reads| FileReader
    RestServer -->|reads| SQLiteReader
    GRPCEntry -->|reads| FileReader
    GRPCEntry -->|reads| SQLiteReader

    FileReader -->|reads from| FileStorage
    SQLiteReader -->|reads from| SQLiteDB
    FileReader --> Consumer
    SQLiteReader --> Consumer

    %% Click Events
    click RestServer "https://github.com/danpung2/logpilot-lite/blob/main/logpilot-rest-server/src/server.ts"
    click LogsController "https://github.com/danpung2/logpilot-lite/blob/main/logpilot-rest-server/src/routes/logs.ts"
    click GRPCEntry "https://github.com/danpung2/logpilot-lite/blob/main/logpilot-grpc-server/src/grpcServer.ts"
    click GRPCBootstrap "https://github.com/danpung2/logpilot-lite/blob/main/logpilot-grpc-server/src/server.ts"
    click Producer "https://github.com/danpung2/logpilot-lite/blob/main/logpilot-lite-client/src/producer.ts"
    click Consumer "https://github.com/danpung2/logpilot-lite/blob/main/logpilot-lite-client/src/consumer.ts"
    click GRPCClient "https://github.com/danpung2/logpilot-lite/blob/main/logpilot-lite-client/src/grpcClient.ts"
    click OffsetStore "https://github.com/danpung2/logpilot-lite/blob/main/logpilot-lite-client/src/offsetStore.ts"
    click Index "https://github.com/danpung2/logpilot-lite/blob/main/logpilot-lite-client/src/index.ts"
    click ValidationHTTP "https://github.com/danpung2/logpilot-lite/blob/main/shared/middleware/validation.ts"
    click ValidationGRPC "https://github.com/danpung2/logpilot-lite/blob/main/shared/middleware/grpcValidation.ts"
    click Schema "https://github.com/danpung2/logpilot-lite/blob/main/shared/schemas/logEntry.ts"
    click Constants "https://github.com/danpung2/logpilot-lite/blob/main/shared/constants/validation.ts"
    click FileWriter "https://github.com/danpung2/logpilot-lite/blob/main/shared/services/fileWriter.ts"
    click FileReader "https://github.com/danpung2/logpilot-lite/blob/main/shared/services/fileReader.ts"
    click SQLiteWriter "https://github.com/danpung2/logpilot-lite/blob/main/shared/services/sqliteWriter.ts"
    click SQLiteReader "https://github.com/danpung2/logpilot-lite/blob/main/shared/services/sqliteReader.ts"
    click DBSetup "https://github.com/danpung2/logpilot-lite/blob/main/shared/sqlite/db.ts"
    click StartServer "https://github.com/danpung2/logpilot-lite/blob/main/start-server.ts"

    %% Styles
    classDef client fill:#a2f5a0,stroke:#333,stroke-width:1px
    classDef server fill:#a0c4ff,stroke:#333,stroke-width:1px
    classDef shared fill:#ffd6a5,stroke:#333,stroke-width:1px
    classDef storage fill:#e0e0e0,stroke:#333,stroke-width:1px
    classDef orchestration fill:#d0d0d0,stroke:#333,stroke-width:1px
```
