How aviroop07/scribbledb works
Purpose and Entry Points
NL2DATA converts natural-language database requirements into ER designs, normalized relational schemas, SQLite-valid DDL, query requirements, constraints, and synthetic-data generation plans. Programmatic workflow entry points are NL2DATA/orchestration/graphs/master.py and NL2DATA/orchestration/state.py; an example caller exists at NL2DATA/orchestration/example_usage.py. The deployable application starts in backend/main.py; the browser client starts in front end/src/main.tsx.
Client/API and Job Boundary
The React/TypeScript client submits descriptions, renders checkpoint editors and schema diagrams, and receives live progress through REST plus WebSocket connections. Its application state and transport boundaries are front end/src/stores/useAppStore.ts, front end/src/services/apiService.ts, and front end/src/hooks/useWebSocket.ts. FastAPI routes separate processing, schema/checkpoint persistence, and suggestions in backend/api/routes/processing.py, backend/api/routes/checkpoints.py, and backend/api/routes/schema.py. Long-running work is represented as jobs and broadcast through backend/utils/job_manager.py and backend/utils/websocket_manager.py.
Pipeline Orchestration and State
LangGraph composes a master workflow from per-phase graphs; phase transitions, conditional paths, loops, and shared graph behavior are centered in NL2DATA/orchestration/graphs/master.py, NL2DATA/orchestration/graphs/common.py, and NL2DATA/orchestration/graphs/phase1.py. A cumulative IRGenerationState carries the NL input and all derived artifacts across phases in NL2DATA/orchestration/state.py. Phase gates validate state at boundaries, while the step registry provides execution metadata and messaging: NL2DATA/orchestration/phase_gates/gates.py, NL2DATA/orchestration/phase_gates/validators.py, and NL2DATA/orchestration/step_registry/registry.py.
Discovery and Schema Construction
Phases 1–2 transform prose into a domain, entities, relations, cardinalities, attributes, keys, and early constraints. Representative extraction/consolidation stages are NL2DATA/phases/phase1/step_1_4_key_entity_extraction.py, NL2DATA/phases/phase1/step_1_7_entity_consolidation.py, and NL2DATA/phases/phase2/step_2_2_intrinsic_attributes.py. Phases 3–4 compile those findings first into ER design and then into the canonical relational schema at NL2DATA/phases/phase3/step_3_1_er_design_compilation.py and NL2DATA/phases/phase4/step_4_1_relational_schema_compilation.py. IR models define this internal boundary in NL2DATA/ir/models/dataset_ir.py and NL2DATA/ir/models/er_relational.py.
Type, DDL, and Query Stages
Phase 5 creates attribute dependency/type decisions, including deterministic foreign-key typing: NL2DATA/phases/phase5/step_5_1_attribute_dependency_graph.py and NL2DATA/phases/phase5/step_5_3_deterministic_fk_data_types.py. Phase 6 deterministically compiles, validates, and creates the schema in SQLite through NL2DATA/phases/phase6/step_6_1_ddl_compilation.py, NL2DATA/phases/phase6/step_6_2_ddl_validation.py, and NL2DATA/phases/phase6/step_6_3_schema_creation.py. Phase 7 derives information needs and read-only SQL validation in NL2DATA/phases/phase7/step_7_1_information_need_identification.py and NL2DATA/phases/phase7/step_7_2_sql_generation_and_validation.py.
Constraints and Data-Generation Planning
Phase 8 adds functional dependencies, categorical values, enforcement decisions, and conflict-checked constraint specifications; its defining stages are NL2DATA/phases/phase8/step_8_1_functional_dependency_analysis.py, NL2DATA/phases/phase8/step_8_4_constraint_detection.py, and NL2DATA/phases/phase8/step_8_8_constraint_compilation.py. Phase 9 produces attribute distributions, volumes, partitioning, and strategy selections via NL2DATA/phases/phase9/step_9_6_distribution_compilation.py, NL2DATA/phases/phase9/tools/strategy_selection.py, and NL2DATA/phases/phase9/strategies/base.py.
External Services and Cross-Cutting Runtime
Python uses LangChain/LangGraph for asynchronous LLM workflow execution, OpenAI models via routing/configuration, SQLite for DDL validation/schema creation, and optional LangSmith tracing. LLM call, retry, tool, and structured-output boundaries reside in NL2DATA/utils/llm/base_router.py, NL2DATA/utils/llm/agent_executor.py, and NL2DATA/utils/llm/structured_output.py; runtime policy is configured in NL2DATA/config/config.yaml. The custom derived-attribute DSL is parsed and validated through NL2DATA/utils/dsl/parser.py and NL2DATA/utils/dsl/validator.py.
Main entry points: master.py Master LangGraph workflow · job_manager.py Jobs & WebSocket broadcast · useAppStore.ts App store & transport · processing.py Processing & persistence routes
Components 18
Browser Client
- React client · Submits requirements and renders editors, diagrams, and progress.
front end/src/main.tsx - App store & transport
front end/src/stores/useAppStore.ts
Backend Runtime
- FastAPI application
backend/main.py - Processing & persistence routes · Processing, checkpoints, schemas, and suggestions endpoints.
backend/api/routes/processing.py - Jobs & WebSocket broadcast · Represents long-running work and streams progress.
backend/utils/job_manager.py
NL2DATA Workflow
- Master LangGraph workflow · Composes phases, transitions, conditional paths, and loops.
NL2DATA/orchestration/graphs/master.py - IRGenerationState · Carries input and cumulative artifacts across all phases.
NL2DATA/orchestration/state.py - Phase gates & step registry · Validates phase boundaries and supplies execution metadata.
NL2DATA/orchestration/phase_gates/gates.py - Discovery & schema findings · Extracts and consolidates domain entities, relationships, attributes, and keys.
NL2DATA/phases/phase1/step_1_4_key_entity_extraction.py - ER and relational models · Internal boundary for ER design and canonical relational schema.
NL2DATA/ir/models/er_relational.py - ER & relational compilation · Compiles findings into ER design then relational schema.
NL2DATA/phases/phase4/step_4_1_relational_schema_compilation.py - Type and DDL pipeline · Determines types, compiles and validates SQLite DDL, then creates schema.
NL2DATA/phases/phase6/step_6_1_ddl_compilation.py - Query requirements · Derives information needs and validates read-only SQL.
NL2DATA/phases/phase7/step_7_2_sql_generation_and_validation.py - Constraint compilation · Builds conflict-checked constraint specifications.
NL2DATA/phases/phase8/step_8_8_constraint_compilation.py - Synthetic-data plan · Compiles distributions, volumes, partitions, and generation strategies.
NL2DATA/phases/phase9/step_9_6_distribution_compilation.py - Derived-attribute DSL
NL2DATA/utils/dsl/parser.py
External Runtime
- LLM execution layer · Routes calls with retries, tools, and structured outputs.
NL2DATA/utils/llm/base_router.py - SQLite · Validates DDL and hosts created schemas.
Connections 21
- React client → App store & transport: uses
- App store & transport → Processing & persistence routes: REST requests
- App store & transport → Jobs & WebSocket broadcast: WebSocket progress
- FastAPI application → Processing & persistence routes: serves
- Processing & persistence routes → Jobs & WebSocket broadcast: creates jobs
- Jobs & WebSocket broadcast → Master LangGraph workflow: runs workflow
- Master LangGraph workflow → IRGenerationState: reads and updates
- Master LangGraph workflow → Phase gates & step registry: gates phases
- Master LangGraph workflow → Discovery & schema findings: phases 1–2
- Discovery & schema findings → ER and relational models: schema findings
- Master LangGraph workflow → ER & relational compilation: phases 3–4
- ER & relational compilation → ER and relational models: compiles
- Master LangGraph workflow → Type and DDL pipeline: phases 5–6
- ER and relational models → Type and DDL pipeline: canonical schema
- Type and DDL pipeline → SQLite: validate and create
- Master LangGraph workflow → Query requirements: phase 7
- Master LangGraph workflow → Constraint compilation: phase 8
- Master LangGraph workflow → Synthetic-data plan: phase 9
- Master LangGraph workflow → LLM execution layer: LLM stages
- Constraint compilation → Derived-attribute DSL: validates expressions
- Jobs & WebSocket broadcast → React client: broadcasts progress