How charli117/easy-dataset works
Main System Components
• Frontend (UI):
• Next.js pages inside the “app” directory (e.g., pages for home, projects, text splitting, questions, settings, etc.).
• React components housed under “components/…” (such as home, projects, playground, text-split, and more).
• Internationalization support via the “locales” folder.
• Backend (API server):
• API endpoints under “app/api/” that handle operations such as document uploads, text splitting, question generation, dataset export, and LLM interactions.
• Project-specific APIs under “app/api/projects/…” structured with dynamic URLs ([projectId] and subroutes).
• Core Libraries & Utilities:
• “lib/” contains core functions including database interactions (“lib/db/”), LLM integrations (“lib/llm/”), prompt templates (“lib/llm/prompts/”), and text splitting utilities (“lib/split-mardown” and “lib/text-splitter.js”).
• Shared constants and utilities (“constant/”, “hooks/”, “lib/util/”).
• Database / Persistence:
• The “local-db” directory, which is used as a file-based or local persistence layer to store project data.
• External Integrations & Deployment:
• External LLM APIs integration (Ollama, OpenAI, etc.) via modules in the “app/api/llm/” and “lib/llm/core”.
• Docker containerization (Dockerfile provided) for deployment.
• Electron integration (contained in the “electron” folder) for building a desktop application.
Relationships and Interactions
• API endpoints then invoke services defined in the “lib/” directory (e.g., for database access, text-splitting, and LLM integrations) to process data.
• The LLM integration layer (within “lib/llm/”) communicates with external services (Ollama and possibly OpenAI endpoints).
• The local file-based database (“local-db”) is updated/retrieved through modules in “lib/db/”.
• Electron’s “main.js”, “preload.js”, etc., serve as the bridge for desktop-specific concerns, interfacing with the Next.js APIs as needed.
Architectural Patterns and Design Principles
• Separation of Concerns: Notice how the project distinctly separates the presentation layer (frontend pages/components), business logic (API routes and libraries), and data persistence (local-db accessed via lib/db).
• RESTful / API-driven Design: The “app/api” directory shows clearly defined API endpoints that follow REST conventions with nested dynamic routing (e.g., by projectId, chunkId, etc.).
• Modularity: Clear modular boundaries exist with dedicated libraries for LLM processing, text splitting, and internationalization.
• Extensibility: Through use of Electron for desktop builds and Docker for containerization, the project is designed to be deployed in multiple environments.
Relevant Technology and Framework Details
• Frontend technologies: Next.js (React 18, MUI 5), highlighting server-side rendering capabilities.
• Backend: Node.js environment with API routes using Next.js API handlers.
• Database: Local file-based database handling under “local-db”, accessed via custom modules.
• External APIs: LLM APIs (e.g., Ollama, OpenAI) integrated via dedicated modules.
• Build and deployment tools: Docker for containerization, Electron for desktop packaging.