# sfu-db/dataprep architecture

> Architecture diagram of the GitHub repository sfu-db/dataprep, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sfu-db/dataprep
- Repository: https://github.com/sfu-db/dataprep

## Overview

##### Main Components
- Data Preparation Core (the dataprep package)
- The Clean Module (dataprep.clean)
– Contains a large number of cleaning functions for various data types (country, date, email, currency, etc.)
– Includes sub-components for imputation, encoding, scaling, and a GUI component (dataprep/clean/gui) developed with a separate front-end (Vue.js based, as indicated by the files in build_frontend) for no-code cleaning.
- The EDA Module (dataprep.eda)
– Supports interactive exploratory data analysis, including profile reports, correlation, distribution, missing value analysis, etc.
– Uses task-centric APIs (e.g., create_report) and is optimized for performance (leveraging Dask).
– Contains several sub-modules (correlation, diff, distribution, missing, outlier) each forming part of the analysis pipeline.
- The Connector Module (dataprep.connector)
– Provides a unified API interface to external data sources (web APIs and databases).
– Contains components such as config management, auto-pagination, throttling, error handling, and a generator for dynamic queries.
- The Lineage Module (dataprep.lineage)
– Produces an interactive column-level lineage graph for SQL.
– Manages dependency analysis among SQL files and integrates with external tools like dbt.
- Supporting Components
– Datasets (dataprep.datasets) for example files and test cases.
– Testing (dataprep/tests) and documentation (docs, examples) directories.
– Build/CI configurations and external configuration files for deployment and code quality (e.g., .github folder, pyproject.toml).

##### Relationships and Interactions
- For the Lineage module, show its role in reading SQL files, processing dependencies, and generating the interactive lineage graph. Present a connection to both SQL files and external systems (e.g., dbt).

##### Architectural Patterns and Design Principles
- Modularity: Each major component (Clean, EDA, Connector, Lineage) is a self-contained module with further sub-modules.
- Separation of Concerns: Data cleaning, exploratory analysis, data connection, and lineage analysis are clearly separated.
- Extensibility: Especially within the Connector module, the design allows for adding new configuration files to support additional APIs.
- UI Integration: The Clean module includes a dedicated GUI front-end that interacts with back-end cleaning functionalities.

##### Relevant Technologies and Frameworks
- Python as the primary language.
- Dask for parallelized computing within EDA and cleaning.
- Vue.js (and related front-end build tools like webpack) for the Clean module’s GUI.
- ConnectorX (for database connections via the Connector module).
- Standard libraries, testing frameworks, and CI/CD configuration files from the repository (e.g., GitHub Workflows, pyright, mypy).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction
    user["User (Data Scientist/Developer)"]:::user

    %% Core Node
    core["DataPrep Core"]:::core

    %% Main Modules
    clean["DataPrep Clean"]:::module
    eda["DataPrep EDA"]:::module
    connector["DataPrep Connector"]:::module
    lineage["DataPrep Lineage"]:::module

    %% Supporting Components
    datasets["Datasets (Examples/Tests)"]:::support
    testing["Testing Components"]:::support
    docs["Documentation & Examples"]:::support
    ci["Build/CI & Config"]:::support

    %% Clean Module Subcomponents
    cleanFunctions["Cleaning Functions"]:::process
    gui["GUI Front-End"]:::ui
    guiBuild["GUI Build Tools"]:::ui
    cleanComp["Imputation/Encoding/Scaling"]:::process

    %% EDA Module Submodules
    edaCorr["Correlation Analysis"]:::analysis
    edaDiff["Diff Comparisons"]:::analysis
    edaDist["Distribution Analysis"]:::analysis
    edaMiss["Missing Data Analysis"]:::analysis
    edaOutlier["Outlier Detection"]:::analysis
    viz["Interactive Visualizations"]:::visual

    %% Connector Module Subcomponents
    connConfig["Config & API Wrappers"]:::integration
    connGen["Query Generator & Schema"]:::integration
    extData["External Data Sources (APIs/DBs)"]:::external

    %% Lineage Module Subcomponent
    lineageSQL["SQL Dependency Analyzer"]:::analysis
    extDbt["External Tool (dbt)"]:::external

    %% User Workflow Connections
    user -->|"ingests via"| connector
    user -->|"triggers via UI"| gui
    user -->|"analyzes"| eda
    user -->|"inspects SQL"| lineage

    %% Core to Modules
    core --> clean
    core --> eda
    core --> connector
    core --> lineage

    %% Supporting Components connected to Core
    core --> datasets
    core --> testing
    core --> docs
    core --> ci

    %% Clean Module Internal Flow
    clean --> gui
    clean --> cleanFunctions
    clean --> cleanComp
    guiBuild -->|"builds"| gui
    gui -->|"triggers"| cleanFunctions

    %% EDA Module Submodule Connections
    eda --> edaCorr
    eda --> edaDiff
    eda --> edaDist
    eda --> edaMiss
    eda --> edaOutlier
    edaCorr --> viz
    edaDiff --> viz
    edaDist --> viz
    edaMiss --> viz
    edaOutlier --> viz

    %% Connector Module Internal Flow
    connector --> connConfig
    connector --> connGen
    connConfig -->|"calls"| extData

    %% Lineage Module Internal Flow
    lineage --> lineageSQL
    lineageSQL -->|"integrates"| extDbt

    %% Click Events
    click core "https://github.com/sfu-db/dataprep/tree/develop/dataprep/"
    click clean "https://github.com/sfu-db/dataprep/tree/develop/dataprep/clean/"
    click gui "https://github.com/sfu-db/dataprep/blob/develop/dataprep/clean/gui/clean_gui.py"
    click guiBuild "https://github.com/sfu-db/dataprep/tree/develop/dataprep/clean/gui/clean_frontend/"
    click cleanComp "https://github.com/sfu-db/dataprep/tree/develop/dataprep/clean/components/"
    click eda "https://github.com/sfu-db/dataprep/tree/develop/dataprep/eda/"
    click edaCorr "https://github.com/sfu-db/dataprep/tree/develop/dataprep/eda/correlation/"
    click edaDiff "https://github.com/sfu-db/dataprep/tree/develop/dataprep/eda/diff/"
    click edaDist "https://github.com/sfu-db/dataprep/tree/develop/dataprep/eda/distribution/"
    click edaMiss "https://github.com/sfu-db/dataprep/tree/develop/dataprep/eda/missing/"
    click edaOutlier "https://github.com/sfu-db/dataprep/tree/develop/dataprep/eda/outlier/"
    click connector "https://github.com/sfu-db/dataprep/tree/develop/dataprep/connector/"
    click lineage "https://github.com/sfu-db/dataprep/tree/develop/dataprep/lineage/"
    click datasets "https://github.com/sfu-db/dataprep/tree/develop/dataprep/datasets/"
    click testing "https://github.com/sfu-db/dataprep/tree/develop/dataprep/tests/"
    click docs "https://github.com/sfu-db/dataprep/tree/develop/docs/"
    click ci "https://github.com/sfu-db/dataprep/tree/develop/.github/"

    %% Styles
    classDef core fill:#bbdefb,stroke:#0d47a1,stroke-width:2px,color:#0d47a1
    classDef module fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px,color:#1b5e20
    classDef process fill:#fff9c4,stroke:#f57f17,stroke-width:2px,color:#f57f17
    classDef analysis fill:#ffe0b2,stroke:#fb8c00,stroke-width:2px,color:#fb8c00
    classDef ui fill:#d1c4e9,stroke:#673ab7,stroke-width:2px,color:#673ab7
    classDef integration fill:#b2dfdb,stroke:#004d40,stroke-width:2px,color:#004d40
    classDef external fill:#ffcdd2,stroke:#c62828,stroke-width:2px,color:#c62828
    classDef support fill:#eeeeee,stroke:#757575,stroke-width:2px,color:#424242
    classDef user fill:#e1f5fe,stroke:#0277bd,stroke-width:2px,color:#0277bd
    classDef visual fill:#dcedc8,stroke:#558b2f,stroke-width:2px,color:#558b2f
```
