# hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation architecture

> Architecture diagram of the GitHub repository hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation
- Repository: https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation

## Overview

##### Project type and purpose
- This is a desktop-based, full-stack “QueryFlow” application written in Python.
- It exposes a SQL-like DSL through a GUI, compiles and transpiles queries into Python code, runs ETL jobs against local or remote data sources, and displays results with visualization.

##### File structure
- Top-level:
- main.py – entry point that bootstraps the GUI and orchestrates the engine
- requirements.txt, LICENSE, README.md
- app/compiler – the Lexer, Parser (PLY), and AST node definitions for SQL-DSL
- app/core – low-level utilities (error definitions, result_monad) used across modules
- app/etl – core engine that interprets/translates AST into executable Python ETL logic
- controllers.py, core.py, helpers.py
- data/… – strategy pattern for local (CSV, JSON, DB via SQLAlchemy) and remote (Google Earth Engine) sources
- app/gui – CustomTkinter-based UI
- ui_compiler.py – ties GUI events to compiler & ETL logic
- vertical_tab_view, results_frame, error_frame – reusable widgets for tabs, results table, error popups
- app/cv – an auxiliary Computer Vision sub-module (bird detection, video reader)

##### README
- Confirms architecture:
- DSL engine uses CFG with LEX & YACC (PLY)
- GUI built on CustomTkinter + TkSheet
- ETL back-end using Pandas, SQLAlchemy, GEE SDK
- Visualization via Matplotlib / Seaborn

a. Main components
- User Interface Layer (GUI)
- DSL Compiler Layer (Lexer, Parser, AST)
- ETL Core Layer (Controllers, Core, Helpers)
- Data Source Layer (Local & Remote Data Factories)
- Visualization Layer (Tables, Charts)
- Error Handling Module (core/errors + GUI error_frame)
- (Optional) Computer Vision Module as a separate workflow
b. Relationships & interactions
1. User enters SQL in the GUI → ui_compiler captures event
2. Compiler.lex/yacc modules tokenize & parse → produce AST
3. ETL Core reads AST → controllers.core generate Python DSL code → executes via utility.py
4. ETL Core invokes Data Factories:
- LocalDataFactory (flat_data.py, database.py, media.py)
- RemoteDataFactory (GEE API collector, data_processor)
5. Data flows back as Pandas DataFrame → Visualization modules render charts/tables → GUI displays
6. Errors bubble up via result_monad → core/errors → error_frame popups
7. (Isolated) CV Module reads media files → detector, details_extractor → outputs analysis
c. Architectural patterns & design principles
- Layered architecture: UI → Compilation → Execution → Data → Visualization
- Strategy/Factory pattern for pluggable data sources (local vs. remote)
- Monad pattern for error/result propagation (result_monad)
- Plugin‐style core: compiler rules (CFG) loosely coupled for runtime extension
d. Key technologies/frameworks
- CustomTkinter, TkSheet (GUI)
- PLY (lex.py, yacc.py) for DSL parsing
- Pandas, SQLAlchemy, GEE SDK (ETL execution)
- Matplotlib, Seaborn (visualization)

##### Detail
- Underlying libraries (e.g., “Compiler (PLY lex/yacc)”)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend (User Interface Layer)"
        direction TB
        UI_COMPILER["ui_compiler.py"]:::green
        subgraph "vertical_tab_view"
            direction TB
            VTV_STC["sql_textbox_colorizer.py"]:::green
            VTV_CF["content_frame.py"]:::green
            VTV_TB["tab_button.py"]:::green
        end
        VTV["vertical_tab_view.py"]:::green
        subgraph "results_frame"
            direction TB
            RF_PF["python_frame.py"]:::green
            subgraph "table_result_frame"
                direction TB
                TRV["visualization.py"]:::green
                TF["table_frame.py"]:::green
                TW["table_widget.py"]:::green
                MSD["multi_selection_dropdown.py"]:::green
            end
        end
        subgraph "error_frame"
            direction TB
            EF_MAIN["error_frame.py"]:::red
            EF_DP["error_details_popup.py"]:::red
        end
    end

    %% Compiler Layer
    subgraph "DSL Compiler Layer"
        direction TB
        LEX["lex.py"]:::blue
        YACC["yacc.py"]:::blue
        AST["ast_nodes.py"]:::blue
    end

    %% ETL Core Layer
    subgraph "ETL Core Layer"
        direction TB
        CONTROLLERS["controllers.py"]:::blue
        CORE_ETL["core.py"]:::blue
        HELPERS["helpers.py"]:::blue
    end

    %% Data Source Layer
    subgraph "Data Source Layer"
        direction TB
        subgraph "LocalDataFactory"
            direction TB
            FLAT["flat_data.py"]:::orange
            DB["database.py"]:::orange
            MEDIA["media.py"]:::orange
        end
        subgraph "RemoteDataFactory"
            direction TB
            REMOTE["remote_data.py"]:::orange
            GEE_COLLECT["google_earth_api_data_collector.py"]:::orange
            GEE_PROC["data_processor.py"]:::orange
        end
    end

    %% Error Handling Core
    subgraph "Error Handling Module"
        direction TB
        ERR_CORE["errors.py"]:::red
        MONAD["result_monad.py"]:::red
    end

    %% CV Module
    subgraph "Computer Vision Module" 
        direction TB
        DET["detector.py"]:::gray
        DE["details_extractor.py"]:::gray
        VR["video_reader.py"]:::gray
        FR["folder_reader.py"]:::gray
        BIRD["bird.py"]:::gray
        B_MOVE["bird_move.py"]:::gray
        OP_MAIN["operation_main.py"]:::gray
        TH_MAIN["threading_main.py"]:::gray
    end

    %% Utility and Entry
    UTIL["utility.py"]:::blue
    MAIN["main.py"]:::green

    %% Relationships
    MAIN -->|"bootstrap GUI"| UI_COMPILER
    UI_COMPILER -->|"tokenize/parse"| LEX
    LEX --> YACC
    YACC --> AST
    AST -->|"generate Python code"| CONTROLLERS
    CONTROLLERS --> CORE_ETL
    CORE_ETL --> HELPERS
    HELPERS -->|"execute query"| FLAT
    HELPERS -->|"execute query"| DB
    HELPERS -->|"execute query"| MEDIA
    HELPERS -->|"execute query"| REMOTE
    REMOTE --> GEE_COLLECT
    GEE_COLLECT --> GEE_PROC
    GEE_PROC --> CORE_ETL
    CORE_ETL -->|"fetch DataFrame"| RF_PF
    RF_PF --> TRV
    RF_PF --> TF
    TF --> TW
    TW --> MSD
    CORE_ETL --> ERR_CORE
    ERR_CORE --> MONAD
    MONAD --> EF_MAIN
    OP_MAIN --> DET
    DET --> DE
    DE --> OP_MAIN
    TH_MAIN --> VR
    VR --> FR
    FR --> OP_MAIN
    OP_MAIN --> MAIN

    %% Click Events
    click UI_COMPILER "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/ui_compiler.py"
    click VTV "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/vertical_tab_view/vertical_tab_view.py"
    click VTV_STC "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/vertical_tab_view/sql_textbox_colorizer.py"
    click VTV_CF "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/vertical_tab_view/content_frame.py"
    click VTV_TB "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/vertical_tab_view/tab_button.py"
    click RF_PF "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/results_frame/python_frame.py"
    click TRV "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/results_frame/table_result_frame/visualization.py"
    click TF "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/results_frame/table_result_frame/table_frame.py"
    click TW "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/results_frame/table_result_frame/table_widget.py"
    click MSD "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/results_frame/table_result_frame/multi_selection_dropdown.py"
    click EF_MAIN "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/error_frame/error_frame.py"
    click EF_DP "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/gui/error_frame/error_details_popup.py"
    click LEX "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/compiler/lex.py"
    click YACC "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/compiler/yacc.py"
    click AST "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/compiler/ast_nodes.py"
    click CONTROLLERS "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/etl/controllers.py"
    click CORE_ETL "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/etl/core.py"
    click HELPERS "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/etl/helpers.py"
    click FLAT "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/etl/data/local/flat_data.py"
    click DB "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/etl/data/local/database.py"
    click MEDIA "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/etl/data/local/media.py"
    click REMOTE "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/etl/data/remote/remote_data.py"
    click GEE_COLLECT "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/etl/data/remote/gee/google_earth_api_data_collector.py"
    click GEE_PROC "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/etl/data/remote/gee/data_processor.py"
    click ERR_CORE "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/core/errors.py"
    click MONAD "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/core/result_monad.py"
    click DET "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/cv/detector.py"
    click DE "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/cv/details_extractor.py"
    click VR "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/cv/video_reader.py"
    click FR "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/cv/folder_reader.py"
    click BIRD "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/cv/bird.py"
    click B_MOVE "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/cv/bird_move.py"
    click OP_MAIN "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/cv/operation_main.py"
    click TH_MAIN "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/cv/threading_main.py"
    click UTIL "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/app/utility.py"
    click MAIN "https://github.com/hussiensharaf/a-cfg-dynamic-engine-for-dynamic-dsl-code-generation/blob/main/main.py"

    %% Styles
    classDef green fill:#e0ffe0,stroke:#0a0,stroke-width:1px
    classDef blue fill:#e0f0ff,stroke:#00f,stroke-width:1px
    classDef orange fill:#fff0e0,stroke:#f80,stroke-width:1px
    classDef red fill:#ffe0e0,stroke:#f00,stroke-width:1px
    classDef gray fill:#f0f0f0,stroke:#888,stroke-width:1px
```
