How skrub-data/skrub works
Main Components
• Core Library – The main “skrub” package contains submodules and functions such as joiners, encoders (MinHashEncoder, ToCategorical, etc.), deduplication methods, and other transformation utilities.
• Datasets Module – Located under skrub/datasets, which provides functionality to fetch or generate example data (e.g., fetch_employee_salaries).
• Reporting & Visualization – Under skrub/_reporting, this component generates interactive table reports and includes its own assets (HTML templates, CSS, and JavaScript files).
• Benchmarks – The “benchmarks” directory holds scripts and utilities to measure the performance of different functions (like fuzzy join and gap encoder benchmarks).
• Documentation – The “doc” folder, along with Sphinx-related configurations and templates, is used to generate the project’s documentation.
• Examples – The “examples” directory provides usage scripts that demonstrate how to apply the library’s functionality.
• Testing & CI/CD – The project contains a comprehensive set of tests (located in skrub/tests and related subdirectories) as well as CI/CD configuration files (.circleci, .github/workflows, etc.) that ensure code quality and continuous integration.
• Build Tools & Configuration – Additional directories like “build_tools” and configuration files (pyproject.toml, .pre-commit-config.yaml) indicate toolchains for auto-formatting, dependency management, and deployment.
Relationships and Interactions
• The Core Library is the heart of the system. Its various modules (joiners, encoders, table vectorizer, tabular learner, etc.) provide APIs that are called by both the user-facing examples and the benchmarks.
• The Datasets Module interacts with the Core Library by supplying sample data for testing and demonstration purposes.
• The Reporting module integrates with the rest of the library to produce visual outputs, feeding off transformed data.
• Examples and Benchmarks consume the Core Library’s functionalities to illustrate usage and evaluate performance.
• Testing frameworks and CI/CD workflows (CircleCI, GitHub Actions) actively run tests located in skrub/tests and ensure that changes in the Core Library and reporting tools maintain quality.
• Documentation pulls source information (docstrings, examples, and API references from the code) to generate developer and user guides using Sphinx.
Architectural Patterns and Design Principles
• Modularity – The project is organized into self-contained, well-defined modules (e.g., separate modules for joiners, encoders, reporting) that promote maintainability and reusability.
• Separation of Concerns – There is a clear separation between data preparation (encoding/joining), data visualization (reporting), examples, and testing/CI processes.
• Extensibility – Many modules (e.g., the reporting and encoding utilities) are designed to be extendable, permitting easy integration of additional functionality.
• Integration with external systems – Emphasis on CI/CD pipelines, automated documentation generation, and sample benchmarks shows an ecosystem-wide approach toward reliable software development.
Relevant Technologies, Frameworks, or Libraries
• Python language features (with emphasis on modular packages).
• Sphinx – Used for documentation generation as indicated in the “doc” folder.
• Testing frameworks – Likely pytest or similar for tests found in skrub/tests.
• CI/CD tools – CircleCI, GitHub Actions, and configuration files for scheduling builds and code coverage (codecov, etc.).
• Other external integrations (e.g., dependencies managed via pyproject.toml and pre-commit hooks).