How blue-yonder/tsfresh works
1. Repository purpose
- This is the tsfresh Python library: a time-series feature engineering and feature selection framework.
- The architectural center is a pipeline that turns long-form time-series/event-sequence data into a tabular feature matrix, then optionally filters it down to relevant features for ML tasks.
2. Core data flow
- Input starts as pandas-style tabular data representing multiple series, usually keyed by an id column and ordered by a time/index column.
- The main flow is: raw sequence data → feature extraction → wide feature matrix → statistical relevance testing/selection → model-ready dataset.
- A second flow wraps the same logic for sklearn-style usage, where extraction/selection can be embedded in pipelines.
3. Feature extraction subsystem
- tsfresh/feature_extraction is the main computation engine.
- extraction.py coordinates how raw time series are grouped, rolled through feature calculators, and assembled into a feature matrix.
- feature_calculators.py contains the actual feature definitions; this is the extensibility point for adding new numerical descriptors.
- settings.py defines extraction presets and feature selection of which calculators to run.
- data.py provides data-shaping helpers used by the extraction engine.
4. Feature selection subsystem
- tsfresh/feature_selection implements the statistical filtering layer.
- relevance.py is the central relevance-testing logic, deciding which extracted features are predictive.
- significance_tests.py contains the hypothesis tests used during filtering.
- selection.py applies the feature-relevance results to produce a reduced feature set.
- This subsystem is the main architectural boundary between “feature generation” and “model-ready subset.”
5. Convenience/API layer
- tsfresh/convenience exposes higher-level user entry points that combine extraction and selection.
- bindings.py provides simpler, opinionated function bindings for common workflows.
- relevant_extraction.py supports the common “extract only relevant features” path, hiding lower-level orchestration.
6. sklearn integration
- tsfresh/transformers contains estimator-style wrappers that make the library composable in scikit-learn pipelines.
- feature_augmenter.py and relevant_feature_augmenter.py add extracted features back to an input frame.
- feature_selector.py exposes selection as a transformer.
- per_column_imputer.py handles missing-value filling as part of pipeline preparation.
- This is the main boundary for using tsfresh inside standard ML workflows rather than as a standalone preprocessing step.
7. Utility layer
- tsfresh/utilities contains supporting infrastructure that is not domain-specific to feature extraction.
- dataframe_functions.py likely handles pandas reshaping/alignment and long-to-wide conversions.
- distribution.py and profiling.py support execution/runtime concerns.
- string_manipulation.py supports naming/identifier handling for generated features.
8. Example datasets and reusable domain demos
- tsfresh/examples packages small reference datasets and example loaders such as har_dataset.py, driftbif_simulation.py, and robot_execution_failures.py.
- These examples act as integration fixtures for documentation and tests, and demonstrate real data ingestion patterns.
9. Command-line / script entry points
- tsfresh/scripts provides runnable utilities around extraction timing and the main run_tsfresh.py entry point.
- measure_execution_time.py and test_timing.py indicate a performance-monitoring angle for feature calculator throughput.
- This suggests the library supports both library usage and command-line execution.
10. Testing strategy
- tests/units covers low-level behavior per subsystem: feature extraction, feature selection, transformers, utilities, and scripts.
- tests/integrations exercises end-to-end paths, including examples, notebooks, bindings, and full pipelines.
- The presence of dask-worker-space under integration tests indicates distributed execution is part of the validation surface.
11. Documentation and learning assets
- docs/ is extensive and acts as a major part of the project architecture, not just ancillary material.
- It documents the conceptual model, data formats, feature calculation, selection, sklearn transformers, cluster usage, and large-data workflows.
- notebooks/ provide executable examples of the main user journeys: feature extraction, sklearn pipelines, forecasting, multiclass selection, and advanced analysis.
12. Execution/runtime considerations
- The repo is Python-packaged and organized as a pure library rather than a service.
- Dependency management and quality gates are driven by setup.py, setup.cfg, Makefile, and pre-commit config.
- Dockerfiles, Binder config, and ReadTheDocs config indicate the project is designed to be runnable in containers, interactive notebooks, and hosted documentation environments.
13. CI/CD and release boundaries
- GitHub Actions workflows cover testing, style checks, benchmark runs, and deployment.
- ReadTheDocs builds the documentation site; Binder supports reproducible notebook demos.
- This means the repo’s operational boundary is mostly around packaging, validation, docs publishing, and benchmark/deploy automation rather than runtime infrastructure.
14. What to diagram as the main architecture
- The clearest diagram would show three layers: data ingestion/reshaping, feature extraction + statistical relevance filtering, and downstream ML integration via sklearn-compatible transformers.
- Around that core are example datasets, CLI scripts, utilities, tests, and documentation.
- The primary internal boundary is between calculator-based feature generation and hypothesis-test-based feature selection.
Main entry points: extraction.py Extraction engine · docs Docs · selection.py Selection · relevance.py Relevance
Components 27
Core pipeline
Time-series feature extraction and statistical feature selection.
- Defaults · Library-wide defaults and shared configuration constants.
tsfresh/defaults.py - Data helpers · Shaping helpers used by the extraction engine.
tsfresh/feature_extraction/data.py - Settings · Extraction presets and calculator selection.
tsfresh/feature_extraction/settings.py - Calculators · Feature calculator implementations and extensibility point.
tsfresh/feature_extraction/feature_calculators.py - Extraction engine · Coordinates grouping, rolling, and assembly of extracted features.
tsfresh/feature_extraction/extraction.py - Significance tests · Hypothesis tests used during relevance filtering.
tsfresh/feature_selection/significance_tests.py - Relevance · Determines which extracted features are predictive.
tsfresh/feature_selection/relevance.py - Selection · Applies relevance results to reduce the feature matrix.
tsfresh/feature_selection/selection.py
User-facing API
Convenience wrappers and sklearn-compatible transformers.
- Bindings · Opinionated high-level function bindings for common workflows.
tsfresh/convenience/bindings.py - Relevant extract · End-to-end helper for extracting only relevant features.
tsfresh/convenience/relevant_extraction.py - Feature augmenter · Adds extracted features back onto an input frame.
tsfresh/transformers/feature_augmenter.py - Relevant augmenter · Augmenter variant that keeps only relevant features.
tsfresh/transformers/relevant_feature_augmenter.py - Feature selector · Wraps feature selection for pipeline use.
tsfresh/transformers/feature_selector.py - Imputer · Per-column missing-value filling for sklearn workflows.
tsfresh/transformers/per_column_imputer.py
Support assets
Examples, scripts, utilities, and packaging around the core library.
- HAR data · Reference dataset loader for human activity recognition.
tsfresh/examples/har_dataset.py - Drift sim · Synthetic drift/bifurcation simulation example.
tsfresh/examples/driftbif_simulation.py - Robot failures · Real-world robot execution failure example loader.
tsfresh/examples/robot_execution_failures.py - Run CLI · Main command-line entry point for extraction workflows.
tsfresh/scripts/run_tsfresh.py - Timing tools · Measures feature calculator execution time.
tsfresh/scripts/measure_execution_time.py - DataFrame utils · Reshaping, alignment, and long-to-wide helpers.
tsfresh/utilities/dataframe_functions.py - Distribution · Execution/distribution support helpers.
tsfresh/utilities/distribution.py - Profiling · Profiling helpers for execution analysis.
tsfresh/utilities/profiling.py - String utils · Feature naming and identifier manipulation helpers.
tsfresh/utilities/string_manipulation.py
Quality and docs
Tests, documentation, notebooks, and release/CI scaffolding.
- Unit tests · Subsystem-level behavioral tests.
tests/units - Integration tests · End-to-end tests covering examples, bindings, notebooks, and pipelines.
tests/integrations - Docs · User guide, API docs, and conceptual walkthroughs.
docs - Notebooks · Executable walkthroughs for extraction, pipelines, forecasting, and analysis.
notebooks
Connections 32
- Defaults → Settings: configures · Shared defaults feed extraction presets and behavior.
- DataFrame utils → Data helpers: shapes · DataFrame helpers prepare and align time-series tables.
- Data helpers → Extraction engine: input prep · Extraction engine consumes reshaped series data.
- Settings → Extraction engine: selects · Settings determine which calculators run.
- Calculators → Extraction engine: computes · Calculator definitions are executed by the extraction engine.
- Extraction engine → Relevance: feature matrix · Extracted features are passed to relevance testing.
- Significance tests → Relevance: tests · Statistical tests underpin relevance decisions.
- Relevance → Selection: ranks · Relevance results drive final feature filtering.
- Extraction engine → Bindings: exposes · Convenience functions wrap extraction workflows.
- Selection → Relevant extract: filters · Relevant extraction combines generation and selection.
- Extraction engine → Feature augmenter: wraps · Transformer augments the input with extracted features.
- Extraction engine → Relevant augmenter: wraps · Relevant augmenter reuses extraction plus selection.
- Selection → Feature selector: wraps · Selection is exposed as an sklearn transformer.
- Imputer → Feature augmenter: prepares · Imputation is used before feature augmentation in pipelines.
- HAR data → Docs: illustrates · Example datasets support documentation and demos.
- Drift sim → Docs: illustrates · Synthetic examples are referenced in docs and learning materials.
- Robot failures → Docs: illustrates · Real-world examples support narrative documentation.
- Run CLI → Extraction engine: invokes · CLI entry point runs extraction workflows.
- Run CLI → Selection: invokes · CLI can apply relevance-based reduction.
- Timing tools → Calculators: benchmarks · Timing script measures calculator throughput.
- Profiling → Timing tools: supports · Profiling helpers support performance measurement.
- String utils → Calculators: names · String helpers support generated feature identifiers.
- Unit tests → Extraction engine: covers · Unit tests validate extraction internals.
- Unit tests → Selection: covers · Unit tests validate relevance and selection logic.
- Unit tests → Feature augmenter: covers · Unit tests validate sklearn wrappers and imputers.
- Integration tests → Bindings: covers · Integration tests exercise convenience APIs.
- Integration tests → Notebooks: covers · Integration tests validate notebook-based workflows.
- Integration tests → HAR data: covers · Integration tests validate example loaders.
- Docs → Extraction engine: documents · Documentation explains feature extraction concepts and settings.
- Docs → Relevance: documents · Documentation explains feature filtering and selection.
- Docs → Feature selector: documents · Documentation covers sklearn transformers.
- Docs → DataFrame utils: documents · Documentation covers data formats and reshaping helpers.