# charli117/easy-dataset architecture

> Architecture diagram of the GitHub repository charli117/easy-dataset, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/charli117/easy-dataset
- Repository: https://github.com/charli117/easy-dataset

## Overview

##### Main System Components
- Frontend (UI):
- Next.js pages inside the “app” directory (e.g., pages for home, projects, text splitting, questions, settings, etc.).
- React components housed under “components/…” (such as home, projects, playground, text-split, and more).
- Internationalization support via the “locales” folder.
- Backend (API server):
- API endpoints under “app/api/” that handle operations such as document uploads, text splitting, question generation, dataset export, and LLM interactions.
- Project-specific APIs under “app/api/projects/…” structured with dynamic URLs ([projectId] and subroutes).
- Core Libraries & Utilities:
- “lib/” contains core functions including database interactions (“lib/db/”), LLM integrations (“lib/llm/”), prompt templates (“lib/llm/prompts/”), and text splitting utilities (“lib/split-mardown” and “lib/text-splitter.js”).
- Shared constants and utilities (“constant/”, “hooks/”, “lib/util/”).
- Database / Persistence:
- The “local-db” directory, which is used as a file-based or local persistence layer to store project data.
- External Integrations & Deployment:
- External LLM APIs integration (Ollama, OpenAI, etc.) via modules in the “app/api/llm/” and “lib/llm/core”.
- Docker containerization (Dockerfile provided) for deployment.
- Electron integration (contained in the “electron” folder) for building a desktop application.

##### Relationships and Interactions
- API endpoints then invoke services defined in the “lib/” directory (e.g., for database access, text-splitting, and LLM integrations) to process data.
- The LLM integration layer (within “lib/llm/”) communicates with external services (Ollama and possibly OpenAI endpoints).
- The local file-based database (“local-db”) is updated/retrieved through modules in “lib/db/”.
- Electron’s “main.js”, “preload.js”, etc., serve as the bridge for desktop-specific concerns, interfacing with the Next.js APIs as needed.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Notice how the project distinctly separates the presentation layer (frontend pages/components), business logic (API routes and libraries), and data persistence (local-db accessed via lib/db).
- RESTful / API-driven Design: The “app/api” directory shows clearly defined API endpoints that follow REST conventions with nested dynamic routing (e.g., by projectId, chunkId, etc.).
- Modularity: Clear modular boundaries exist with dedicated libraries for LLM processing, text splitting, and internationalization.
- Extensibility: Through use of Electron for desktop builds and Docker for containerization, the project is designed to be deployed in multiple environments.

##### Relevant Technology and Framework Details
- Frontend technologies: Next.js (React 18, MUI 5), highlighting server-side rendering capabilities.
- Backend: Node.js environment with API routes using Next.js API handlers.
- Database: Local file-based database handling under “local-db”, accessed via custom modules.
- External APIs: LLM APIs (e.g., Ollama, OpenAI) integrated via dedicated modules.
- Build and deployment tools: Docker for containerization, Electron for desktop packaging.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Docker Container Deployment
    DC("Docker Container"):::docker

    %% Frontend Layer
    subgraph "Frontend"
        FP("Next.js Pages"):::frontend
        RC("React Components"):::frontend
        INT("Internationalization"):::frontend
    end

    %% Backend API Layer
    subgraph "Backend"
        API("RESTful API Endpoints"):::backend
    end

    %% Core Libraries & Utilities
    subgraph "Core Libraries & Utilities"
        DBM("Database Module"):::lib
        TS("Text Splitting Utilities"):::lib
        LLM("LLM Integration"):::lib
    end

    %% Persistence and External Services
    LDB("Local Database"):::database
    EXLLM("External LLM Services"):::external

    %% Electron Desktop Integration
    EL("Electron Desktop"):::electron

    %% Connections: Deployment
    DC -->|"hosts"| FP
    DC -->|"hosts"| API
    DC -->|"hosts"| EL

    %% Connections: Frontend to Backend
    FP -->|"calls"| API
    RC -->|"calls"| API
    INT -->|"localizes"| FP

    %% Connections: Backend invoking Core Libraries
    API -->|"invokes"| DBM
    API -->|"invokes"| TS
    API -->|"invokes"| LLM

    %% Connections: Core Libraries to Persistence and External APIs
    DBM -->|"read/write"| LDB
    LLM -->|"integrates"| EXLLM

    %% Connection: Electron communicating with API Server
    EL -->|"communicates"| API

    %% Click Events
    click FP "https://github.com/charli117/easy-dataset/tree/main/app/"
    click RC "https://github.com/charli117/easy-dataset/tree/main/components/"
    click INT "https://github.com/charli117/easy-dataset/tree/main/locales/"
    click API "https://github.com/charli117/easy-dataset/tree/main/app/api/"
    click DBM "https://github.com/charli117/easy-dataset/tree/main/lib/db/"
    click TS "https://github.com/charli117/easy-dataset/tree/main/lib/split-mardown/"
    click LLM "https://github.com/charli117/easy-dataset/tree/main/lib/llm/core/"
    click EXLLM "https://github.com/charli117/easy-dataset/tree/main/app/api/llm/"
    click EL "https://github.com/charli117/easy-dataset/tree/main/electron/"

    %% Styles
    classDef frontend fill:#a6cee3,stroke:#000,stroke-width:1px;
    classDef backend fill:#1f78b4,stroke:#000,stroke-width:1px;
    classDef lib fill:#b2df8a,stroke:#000,stroke-width:1px;
    classDef database fill:#33a02c,stroke:#000,stroke-width:1px;
    classDef electron fill:#fb9a99,stroke:#000,stroke-width:1px;
    classDef external fill:#fdbf6f,stroke:#000,stroke-width:1px;
    classDef docker fill:#cab2d6,stroke:#000,stroke-width:1px;
```
