How ngz91/eunomia works
Project Type and Purpose
• Eunomia is a Python‐based CLI tool for locally analyzing code via a GPT4All LLM.
• It leverages Langchain for chunking and embeddings, Chroma as a local vector store, and a GPT4All or LlamaCpp model for query answering.
• Key use cases: ingest a target codebase into a vector database, then interactively query it without any internet.
High‐Level Architectural Components
a. CLI Entry Point (Eunomia.py)
• Parses commands (ingest, start).
• Loads environment variables from .env.
b. Configuration Module (src/constants.py)
• Reads and validates settings: PERSIST_DIRECTORY, LLM, BACKEND, embedding model name, chunk sizes, ignore list.
c. Ingestion Pipeline (src/ingest.py)
• File System Scanner: reads files in working directory, honoring IGNORE_FOLDERS.
• Text Splitter: splits each document into chunks (via Langchain code splitter).
• Embedding Generator: uses SentenceTransformers to embed each chunk.
• Vector Store Writer: writes embeddings + metadata into a local Chroma store.
d. Query/Interaction Pipeline (handled in Eunomia.py under start)
• Vector Store Loader: re‐loads Chroma store from PERSIST_DIRECTORY.
• Retriever: Langchain Retriever that takes user query, finds top‐N relevant chunks.
• LLM Model Loader: loads GPT4All / LlamaCpp model via the specified backend and model file.
• Q&A Chain: feeds retrieved chunks + user question into the LLM, streams or returns the answer.
e. Persistent Storage
• Local file system for: vector store files (Chroma), downloaded embeddings models (SentenceTransformers), downloaded GPT4All model binaries.
f. External Dependencies
• Langchain, Chroma (vector DB), SentenceTransformers, GPT4All / LlamaCpp C++ backend.
Relationships & Data Flows
• User runs eunomia ingest → CLI → Config → Ingestion Pipeline → produces Vector Store in FS.
• User runs eunomia start → CLI → Config → loads Vector Store + LLM Model → on each query: Retriever → Q&A Chain → LLM → reply.
• CLI → Config
• CLI → Ingest Module → File System → Text Splitter → Embeddings → Vector Store
• CLI → Query Module → Vector Store → Retriever → LLM Model → Output
Architectural Patterns
• Pipeline Pattern for the ingest workflow (sequential stages: load → split → embed → store).
• Layered/Modular separation: configuration, ingestion, retrieval, model inference.
• CLI-driven orchestration: a single‐process, script-based architecture.