How devbm7/synth-gen works
Project Type and Purpose
• This is a full-stack data-generation application.
• Frontend: Streamlit app (streamlit_app.py) that runs on port 8501.
• Backend: FastAPI service (app.py) that exposes a REST-ful API on port 8000.
• External LLM service: Ollama (local or remote) used for generative tasks.
• Storage: Local filesystem directories (data/, uploads/, logs/) for persistence.
Main Components
a. User Interface
• Streamlit Frontend
• Runs in the user’s browser or desktop session
• Responsible for schema definition, file upload, parameter selection, and download links
b. API Layer
• FastAPI Backend
• Exposes endpoints: /generate, /task/{task_id}, /data/{task_id}, /convert_to_csv, /upload_csv, /append_data
• Validates requests using Pydantic
• Orchestrates generation workflow
c. Background Worker (Async Task Processor)
• Could be FastAPI BackgroundTasks or a separate queue/worker
• Manages long-running LLM calls and status tracking
d. LLM Engine
• Ollama LLM service (e.g., gemma3:latest)
• Accepts prompts, returns JSON data
e. Data Storage
• data/ directory for completed datasets
• uploads/ directory for user-provided CSVs
f. Logging
• logs/api.log for request and error logs
g. Helper Modules
• csv_converter.py for JSON→CSV conversion
• Archive folder for historical versions (not part of runtime)
Interactions & Data Flow
1. User defines schema or uploads CSV in the Streamlit UI.
2. UI sends HTTP POST to FastAPI (/generate or /upload_csv).
3. FastAPI enqueues a background task for LLM generation.
4. Background worker formats a prompt and calls Ollama via its API.
5. Ollama returns synthetic records in JSON.
6. Worker validates (Pydantic) and persists results to data/ (JSON or via csv_converter.py for CSV).
7. FastAPI updates task status; UI polls /task/{task_id} and fetches data at /data/{task_id}.
8. Logs are written to logs/api.log at each step.
Architectural Patterns & Principles
• API-First Design: Clean separation of concerns, clearly defined endpoints.
• Modular Monolith: Backend modules (validation, prompt builder, storage) can be broken into microservices later.
• Asynchronous Processing: Background tasks decouple request/response from long-running LLM calls.
• File-based Persistence: Simple, no external database dependency; easy to replace with RDBMS/NoSQL if needed.