How sfu-db/dataprep works
Main Components
• Data Preparation Core (the dataprep package)
• The Clean Module (dataprep.clean)
– Contains a large number of cleaning functions for various data types (country, date, email, currency, etc.)
– Includes sub-components for imputation, encoding, scaling, and a GUI component (dataprep/clean/gui) developed with a separate front-end (Vue.js based, as indicated by the files in build_frontend) for no-code cleaning.
• The EDA Module (dataprep.eda)
– Supports interactive exploratory data analysis, including profile reports, correlation, distribution, missing value analysis, etc.
– Uses task-centric APIs (e.g., create_report) and is optimized for performance (leveraging Dask).
– Contains several sub-modules (correlation, diff, distribution, missing, outlier) each forming part of the analysis pipeline.
• The Connector Module (dataprep.connector)
– Provides a unified API interface to external data sources (web APIs and databases).
– Contains components such as config management, auto-pagination, throttling, error handling, and a generator for dynamic queries.
• The Lineage Module (dataprep.lineage)
– Produces an interactive column-level lineage graph for SQL.
– Manages dependency analysis among SQL files and integrates with external tools like dbt.
• Supporting Components
– Datasets (dataprep.datasets) for example files and test cases.
– Testing (dataprep/tests) and documentation (docs, examples) directories.
– Build/CI configurations and external configuration files for deployment and code quality (e.g., .github folder, pyproject.toml).
Relationships and Interactions
• For the Lineage module, show its role in reading SQL files, processing dependencies, and generating the interactive lineage graph. Present a connection to both SQL files and external systems (e.g., dbt).
Architectural Patterns and Design Principles
• Modularity: Each major component (Clean, EDA, Connector, Lineage) is a self-contained module with further sub-modules.
• Separation of Concerns: Data cleaning, exploratory analysis, data connection, and lineage analysis are clearly separated.
• Extensibility: Especially within the Connector module, the design allows for adding new configuration files to support additional APIs.
• UI Integration: The Clean module includes a dedicated GUI front-end that interacts with back-end cleaning functionalities.
Relevant Technologies and Frameworks
• Python as the primary language.
• Dask for parallelized computing within EDA and cleaning.
• Vue.js (and related front-end build tools like webpack) for the Clean module’s GUI.
• ConnectorX (for database connections via the Connector module).
• Standard libraries, testing frameworks, and CI/CD configuration files from the repository (e.g., GitHub Workflows, pyright, mypy).