How alteryx/featuretools works
1. What this repository is
Featuretools is a Python library for automated feature engineering. Its core job is to turn relational or time-based data stored in an EntitySet into a machine-learning-ready feature matrix using Deep Feature Synthesis (DFS).
2. Primary user-facing entry points
The top-level featuretools package exposes the main API, including dfs, EntitySet, demo data loaders, and feature/primitive APIs. The package can also be run as a module via featuretools/__main__.py, but the main architecture is library-first rather than service-based.
3. Core data model: EntitySet and relationships
The entityset subsystem defines the project’s central data abstraction: tables, relationships, timestamps, and last-time-index logic. It also includes serialization/deserialization so entity sets can be saved and reloaded across sessions or environments.
4. Feature definition layer
The feature_base subsystem represents features as composable objects rather than simple column names. It handles feature construction, caching, descriptions, visualization, and serialization, which makes feature definitions portable and reusable.
5. Primitive system
The primitives package provides the building blocks for feature generation. It contains base classes for aggregation and transform primitives, plus a large catalog of standard primitives grouped by type: numeric, datetime, email, URL, natural language, lat/long, time series, cumulative, and binary operations. This is the main extensibility boundary for custom feature logic.
6. Feature synthesis pipeline
The synthesis subsystem implements DFS and related orchestration. It selects valid primitives, combines features according to entity relationships, and produces the final feature definitions. This is the layer that turns entity graph structure plus primitive catalog into candidate features.
7. Feature matrix computation
The computational_backends subsystem executes the synthesized features against actual data. It builds feature sets, calculates the feature matrix, and manages backend-specific utilities. This is the separation point between symbolic feature definitions and numeric execution over tables.
8. Feature selection and reduction
The selection package provides post-synthesis filtering and selection logic. It acts after feature generation to reduce feature sets based on criteria such as redundancy or usefulness, helping keep outputs tractable for downstream ML pipelines.
9. Feature discovery utilities
The feature_discovery package contains lightweight discovery and conversion tools, including FeatureCollection and LiteFeature. This appears to support importing, comparing, and managing feature definitions outside the full DFS pipeline.
10. Demo and example datasets
The demo package supplies ready-made sample datasets such as retail, flight, weather, and mock customer data. These are used to illustrate the library’s flow: load an EntitySet, define a target dataframe, run DFS, and inspect the resulting feature matrix.
11. Utilities and supporting services
The utils package contains cross-cutting helpers for configuration, time handling, schema/description utilities, plugin entry points, S3 helpers, plotting, and primitive recommendation. These are shared support functions rather than a separate domain layer.
12. Documentation and notebooks as a major repo component
The docs tree is substantial and includes Sphinx docs plus many Jupyter notebooks showing getting started, guides, and resource material. For this repo, documentation is not ancillary; it is part of the primary product experience and explains the library’s multi-step workflow.
13. Testing and compatibility boundaries
The tests tree mirrors the library architecture: entity sets, primitives, synthesis, selection, discovery, utilities, entry points, and compatibility with add-ons. There are also requirement files for minimum/latest dependency matrices, indicating the repo carefully manages behavior across dependency versions and optional extras like Dask.
14. Operational/runtime characteristics
This is a pure Python package distributed via pip/conda, with optional dependency-based features such as Dask-backed parallelism and add-on primitive families. CI workflows cover linting, docs, release automation, dependency testing, and performance checks, but there is no standalone server or persistent runtime in the repository.
15. End-to-end data flow
The typical flow is: load or build an EntitySet → define target dataframe and relationships → DFS synthesizes candidate features from primitives → computational backend evaluates them into a feature matrix → optional selection/reduction and serialization/export for downstream ML use.
Main entry points: tests Core tests · featuretools/__init__.py featuretools · deep_feature_synthesis.py DFS synthesis · standard/__init__.py Standard primitives
Components 21
Core library
Main runtime packages for entity modeling, feature definitions, synthesis, and execution.
- featuretools · Public package surface exposing DFS, EntitySet, primitives, and demos.
featuretools/__init__.py - Module entry · Allows running the library as a module.
featuretools/__main__.py - EntitySet · Tables, relationships, timestamps, and serialization for relational/time-based data.
featuretools/entityset/entityset.py - Feature base · Composable feature objects with caching, descriptions, visualization, and serialization.
featuretools/feature_base/feature_base.py - DFS synthesis · Deep Feature Synthesis orchestration and primitive selection.
featuretools/synthesis/deep_feature_synthesis.py - Execution backend · Builds feature sets and calculates feature matrices over data.
featuretools/computational_backends/calculate_feature_matrix.py - Selection · Filters and reduces generated feature sets.
featuretools/selection/selection.py - Discovery · Lightweight feature collection and conversion utilities.
featuretools/feature_discovery/feature_discovery.py
Primitives
Base classes and standard feature primitives used by DFS.
- Primitive base · Base classes for aggregation and transform primitives.
featuretools/primitives/base/primitive_base.py - Standard primitives · Built-in aggregation, transform, and time-series primitive families.
featuretools/primitives/standard/__init__.py - Datetime · Calendar, holiday, and time-difference transforms.
featuretools/primitives/standard/transform/datetime/__init__.py - Numeric · Numeric transforms such as absolute value, logs, percent change, and lag-like operations.
featuretools/primitives/standard/transform/numeric/__init__.py - Text & IDs · Email, URL, natural language, postal, name, and file-derived transforms.
featuretools/primitives/standard/transform/natural_language/__init__.py - Time series · Rolling, expanding, cumulative, and lag primitives for ordered data.
featuretools/primitives/standard/transform/time_series/__init__.py
Support
Shared utilities, demo data, and package entry points.
Docs
Sphinx docs, guides, and notebooks used as part of the product experience.
- Documentation · Sphinx docs and notebooks demonstrating the library workflow.
docs/source/index.ipynb
Tests
Architecture-mirroring test suites and compatibility checks.
- Core tests · Tests covering entity sets, primitives, synthesis, selection, discovery, and utilities.
featuretools/tests - Add-on tests · Entry-point and plugin tests for external primitive and plugin packages.
featuretools/tests/entry_point_tests - Req matrices · Minimum/latest/core/dask requirement files used to validate supported environments.
featuretools/tests/requirement_files
Tooling
Packaging, CI, release, and developer workflow files.
- CI & release · Workflow, packaging, and development automation files.
pyproject.toml
Connections 42
- featuretools → EntitySet: exports · Public API exposes EntitySet and related helpers.
- featuretools → Feature base: exports · Public API exposes feature objects and feature construction helpers.
- featuretools → DFS synthesis: exports · Public API exposes DFS entry points.
- featuretools → Execution backend: exports · Public API exposes feature matrix computation.
- featuretools → Selection: exports · Public API exposes feature selection helpers.
- featuretools → Demo data: exports · Public API exposes demo datasets.
- Module entry → featuretools: uses · Module entry delegates to library API.
- DFS synthesis → EntitySet: analyzes · DFS uses entity tables and relationships to build candidate features.
- DFS synthesis → Primitive base: consumes · Synthesis relies on primitive interfaces to enumerate valid features.
- DFS synthesis → Standard primitives: selects · DFS selects from the built-in primitive catalog.
- DFS synthesis → Feature base: builds · Synthesized outputs are feature objects.
- Execution backend → Feature base: executes · Backends evaluate feature definitions against data.
- Execution backend → EntitySet: reads · Matrix calculation reads from entity sets and relationships.
- Selection → Feature base: filters · Selection operates on generated feature objects.
- Discovery → Feature base: converts · Discovery utilities import and transform feature definitions.
- Standard primitives → Primitive base: implements · Standard primitives extend the base primitive abstractions.
- Standard primitives → Datetime: groups · Datetime family is part of the standard catalog.
- Standard primitives → Numeric: groups · Numeric family is part of the standard catalog.
- Standard primitives → Text & IDs: groups · Text and identifier family is part of the standard catalog.
- Standard primitives → Time series: groups · Time-series family is part of the standard catalog.
- Demo data → EntitySet: loads · Demo datasets are used to construct example entity sets.
- Demo data → DFS synthesis: illustrates · Demo flows demonstrate DFS usage.
- Utilities → EntitySet: supports · Shared utilities support entity set handling and metadata.
- Utilities → Feature base: supports · Shared utilities support feature descriptions and serialization.
- Utilities → Primitive base: supports · Utilities support primitive discovery and recommendation.
- Documentation → featuretools: documents · Docs explain the main public API and workflow.
- Documentation → EntitySet: teaches · Guides cover building and using entity sets.
- Documentation → DFS synthesis: teaches · Guides cover DFS tuning and primitive options.
- Documentation → Selection: teaches · Guides cover feature selection and reduction.
- Core tests → EntitySet: verifies · Tests cover entity set behavior and serialization.
- Core tests → Feature base: verifies · Tests cover feature modeling, descriptions, and serialization.
- Core tests → Standard primitives: verifies · Tests cover standard primitive behavior.
- Core tests → DFS synthesis: verifies · Tests cover DFS and primitive selection.
- Core tests → Execution backend: verifies · Tests cover feature matrix calculation.
- Core tests → Selection: verifies · Tests cover feature selection logic.
- Core tests → Discovery: verifies · Tests cover feature discovery utilities.
- Core tests → Utilities: verifies · Tests cover shared utility behavior.
- Add-on tests → Utilities: verifies · Addon tests validate entry-point and plugin loading paths.
- Req matrices → CI & release: supports · Dependency matrices support CI and compatibility testing.
- CI & release → Documentation: builds · Workflow and packaging tooling build and validate docs.
- CI & release → Core tests: runs · CI validates core behavior across dependency sets.
- CI & release → Req matrices: checks · CI uses requirement matrices for minimum/latest coverage.