How hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation works
Project type and purpose
• This is a desktop-based, full-stack “QueryFlow” application written in Python.
• It exposes a SQL-like DSL through a GUI, compiles and transpiles queries into Python code, runs ETL jobs against local or remote data sources, and displays results with visualization.
File structure
• Top-level:
• main.py – entry point that bootstraps the GUI and orchestrates the engine
• requirements.txt, LICENSE, README.md
• app/compiler – the Lexer, Parser (PLY), and AST node definitions for SQL-DSL
• app/core – low-level utilities (error definitions, result_monad) used across modules
• app/etl – core engine that interprets/translates AST into executable Python ETL logic
• controllers.py, core.py, helpers.py
• data/… – strategy pattern for local (CSV, JSON, DB via SQLAlchemy) and remote (Google Earth Engine) sources
• app/gui – CustomTkinter-based UI
• ui_compiler.py – ties GUI events to compiler & ETL logic
• vertical_tab_view, results_frame, error_frame – reusable widgets for tabs, results table, error popups
• app/cv – an auxiliary Computer Vision sub-module (bird detection, video reader)
README
• Confirms architecture:
• DSL engine uses CFG with LEX & YACC (PLY)
• GUI built on CustomTkinter + TkSheet
• ETL back-end using Pandas, SQLAlchemy, GEE SDK
• Visualization via Matplotlib / Seaborn
a. Main components
• User Interface Layer (GUI)
• DSL Compiler Layer (Lexer, Parser, AST)
• ETL Core Layer (Controllers, Core, Helpers)
• Data Source Layer (Local & Remote Data Factories)
• Visualization Layer (Tables, Charts)
• Error Handling Module (core/errors + GUI error_frame)
• (Optional) Computer Vision Module as a separate workflow
b. Relationships & interactions
1. User enters SQL in the GUI → ui_compiler captures event
2. Compiler.lex/yacc modules tokenize & parse → produce AST
3. ETL Core reads AST → controllers.core generate Python DSL code → executes via utility.py
4. ETL Core invokes Data Factories:
• LocalDataFactory (flat_data.py, database.py, media.py)
• RemoteDataFactory (GEE API collector, data_processor)
5. Data flows back as Pandas DataFrame → Visualization modules render charts/tables → GUI displays
6. Errors bubble up via result_monad → core/errors → error_frame popups
7. (Isolated) CV Module reads media files → detector, details_extractor → outputs analysis
c. Architectural patterns & design principles
• Layered architecture: UI → Compilation → Execution → Data → Visualization
• Strategy/Factory pattern for pluggable data sources (local vs. remote)
• Monad pattern for error/result propagation (result_monad)
• Plugin‐style core: compiler rules (CFG) loosely coupled for runtime extension
d. Key technologies/frameworks
• CustomTkinter, TkSheet (GUI)
• PLY (lex.py, yacc.py) for DSL parsing
• Pandas, SQLAlchemy, GEE SDK (ETL execution)
• Matplotlib, Seaborn (visualization)
Detail
• Underlying libraries (e.g., “Compiler (PLY lex/yacc)”)