How danpung2/logpilot-lite works
Project type and purpose
• This is a mono-repo for “LogPilot-Lite,” a lightweight log collection system offering both a REST API server, a gRPC server, and a standalone client library.
• Purpose: collect, store, and stream application logs via HTTP or gRPC, storing them either in plain files or a SQLite database.
File structure
• Top-level packages:
• logpilot-rest-server/ – Express-based REST API
• logpilot-grpc-server/ – gRPC API
• logpilot-lite-client/ – npm-publishable client library
• shared/ – shared constants, schemas, middleware, and storage services (file & SQLite) consumed by both servers and bundled into the client.
• Root files: Dockerfile, start-server.ts (switches mode by LOGPILOT_MODE), .env for configuration.
Key README insights
• Single Docker image can run in “rest” or “grpc” mode via an environment variable.
• Two storage backends: file write/read and SQLite.
• Client supports SendLog and offset-based ConsumeLog flows.
• Client Application
– logpilot-lite-client modules: Producer, Consumer, gRPCClient (for gRPC mode), HTTPClient (for REST—e.g., Axios/Fetch)
– OffsetStore (tracks read offsets in SQLite or in-memory)
• API Server
– REST Server (Express routes in logpilot-rest-server/src/routes/logs.ts)
– gRPC Server (grpcServer.ts in logpilot-grpc-server/src)
• Shared Library (imported by both servers & client)
– Validation middleware (shared/middleware)
– Schemas (shared/schemas/logEntry)
– Services
• FileReader/FileWriter
• SQLiteReader/SQLiteWriter
• Storage Backends
– Log files on disk (./data/*.log)
– SQLite database (./data/logs.db)
• Orchestration
– start-server.ts selects mode, spins up chosen server
– Docker container with env var LOGPILOT_MODE
1. Client → API Server (Client.PerformSendLog → REST POST /logs or gRPC SendLog RPC)
2. API Server applies Validation → Serializes request into shared/logEntry schema
3. Based on “storage” field → Shared Service layer writes to file or SQLite
4. Consumer client → API Server (REST GET /logs?offset=… or gRPC StreamLogs RPC) → Shared Service reads from file/SQLite → returns log entries
c. Architectural patterns & principles
• Layered architecture:
• Presentation (Client / API endpoints)
• Application logic (Validation middleware → Controllers/Handlers)
• Domain services (shared/services for persistence)
• Mode switch as strategy pattern: one process, two interchangeable server implementations
• Code reuse via a shared package
d. Relevant technologies
• Node.js + TypeScript
• Express (REST)
• @grpc/grpc-js + proto definitions (gRPC)
• SQLite (via shared/services)
• Docker