# devbm7/synth-gen architecture

> Architecture diagram of the GitHub repository devbm7/synth-gen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devbm7/synth-gen
- Repository: https://github.com/devbm7/synth-gen

## Overview

##### Project Type and Purpose
- This is a full-stack data-generation application.
- Frontend: Streamlit app (`streamlit_app.py`) that runs on port 8501.
- Backend: FastAPI service (`app.py`) that exposes a REST-ful API on port 8000.
- External LLM service: Ollama (local or remote) used for generative tasks.
- Storage: Local filesystem directories (`data/`, `uploads/`, `logs/`) for persistence.

##### Main Components
a. User Interface
- Streamlit Frontend
- Runs in the user’s browser or desktop session
- Responsible for schema definition, file upload, parameter selection, and download links
b. API Layer
- FastAPI Backend
- Exposes endpoints: `/generate`, `/task/{task_id}`, `/data/{task_id}`, `/convert_to_csv`, `/upload_csv`, `/append_data`
- Validates requests using Pydantic
- Orchestrates generation workflow
c. Background Worker (Async Task Processor)
- Could be FastAPI BackgroundTasks or a separate queue/worker
- Manages long-running LLM calls and status tracking
d. LLM Engine
- Ollama LLM service (e.g., `gemma3:latest`)
- Accepts prompts, returns JSON data
e. Data Storage
- `data/` directory for completed datasets
- `uploads/` directory for user-provided CSVs
f. Logging
- `logs/api.log` for request and error logs
g. Helper Modules
- `csv_converter.py` for JSON→CSV conversion
- Archive folder for historical versions (not part of runtime)

##### Interactions & Data Flow
1. User defines schema or uploads CSV in the Streamlit UI.
2. UI sends HTTP POST to FastAPI (`/generate` or `/upload_csv`).
3. FastAPI enqueues a background task for LLM generation.
4. Background worker formats a prompt and calls Ollama via its API.
5. Ollama returns synthetic records in JSON.
6. Worker validates (Pydantic) and persists results to `data/` (JSON or via `csv_converter.py` for CSV).
7. FastAPI updates task status; UI polls `/task/{task_id}` and fetches data at `/data/{task_id}`.
8. Logs are written to `logs/api.log` at each step.

##### Architectural Patterns & Principles
- API-First Design: Clean separation of concerns, clearly defined endpoints.
- Modular Monolith: Backend modules (validation, prompt builder, storage) can be broken into microservices later.
- Asynchronous Processing: Background tasks decouple request/response from long-running LLM calls.
- File-based Persistence: Simple, no external database dependency; easy to replace with RDBMS/NoSQL if needed.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layer 1: Client
    subgraph "Layer 1: Client"
        UI["Streamlit Frontend<br/>(port 8501)"]:::ui
    end

    %% Layer 2: Application
    subgraph "Layer 2: Application"
        subgraph "Backend Modules"
            API["FastAPI Backend<br/>(port 8000)"]:::backend
            PromptBuilder["Prompt Builder Module"]:::backend
            CsvConverter["CSV Converter Module"]:::backend
        end
        Worker["Background Worker"]:::worker
    end

    %% Layer 3: Services
    subgraph "Layer 3: Services"
        Ollama["Ollama LLM Service<br/>gemma3:latest"]:::service
    end

    %% Layer 4: Storage
    subgraph "Layer 4: Storage"
        Data["data/"]:::storage
        Uploads["uploads/"]:::storage
        Logs["logs/api.log"]:::storage
    end

    %% Project Files
    subgraph "Project Files"
        Requirements["requirements.txt"]:::docs
        Readme["README.md"]:::docs
        Gitignore[".gitignore"]:::docs
        Archive["archive/"]:::docs
    end

    %% Connections
    UI -->|"POST /generate, /upload_csv"| API
    UI -->|"GET /task/{task_id}, /data/{task_id}"| API

    API -->|"enqueue task"| Worker
    Worker -->|"LLM prompt"| Ollama
    Ollama -->|"JSON response"| Worker

    Worker -->|"write JSON"| Data
    Worker -->|"store uploads"| Uploads
    Worker -->|"log events"| Logs

    Worker -->|"convert JSON→CSV"| CsvConverter
    CsvConverter -->|"write CSV"| Data

    API -->|uses| PromptBuilder

    %% Click Events
    click UI "https://github.com/devbm7/synth-gen/blob/main/streamlit_app.py"
    click API "https://github.com/devbm7/synth-gen/blob/main/app.py"
    click CsvConverter "https://github.com/devbm7/synth-gen/blob/main/csv_converter.py"
    click Archive "https://github.com/devbm7/synth-gen/tree/main/archive/"
    click Requirements "https://github.com/devbm7/synth-gen/blob/main/requirements.txt"
    click Readme "https://github.com/devbm7/synth-gen/blob/main/README.md"
    click Gitignore "https://github.com/devbm7/synth-gen/blob/main/.gitignore"

    %% Styles
    classDef ui fill:#E0F7FA,stroke:#006064,color:#000;
    classDef backend fill:#E8F5E9,stroke:#2E7D32,color:#000;
    classDef worker fill:#FFF9C4,stroke:#F9A825,color:#000;
    classDef service fill:#CFD8DC,stroke:#455A64,color:#000;
    classDef storage fill:#ECEFF1,stroke:#90A4AE,color:#000;
    classDef docs fill:#F3E5F5,stroke:#6A1B9A,color:#000;
```
