# blue-yonder/tsfresh architecture

> Architecture diagram of the GitHub repository blue-yonder/tsfresh, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/blue-yonder/tsfresh
- Repository: https://github.com/blue-yonder/tsfresh

## Overview

1. **Repository purpose**
   - This is the `tsfresh` Python library: a time-series feature engineering and feature selection framework.
   - The architectural center is a pipeline that turns long-form time-series/event-sequence data into a tabular feature matrix, then optionally filters it down to relevant features for ML tasks.

2. **Core data flow**
   - Input starts as pandas-style tabular data representing multiple series, usually keyed by an `id` column and ordered by a time/index column.
   - The main flow is: raw sequence data → feature extraction → wide feature matrix → statistical relevance testing/selection → model-ready dataset.
   - A second flow wraps the same logic for sklearn-style usage, where extraction/selection can be embedded in pipelines.

3. **Feature extraction subsystem**
   - `tsfresh/feature_extraction` is the main computation engine.
   - `extraction.py` coordinates how raw time series are grouped, rolled through feature calculators, and assembled into a feature matrix.
   - `feature_calculators.py` contains the actual feature definitions; this is the extensibility point for adding new numerical descriptors.
   - `settings.py` defines extraction presets and feature selection of which calculators to run.
   - `data.py` provides data-shaping helpers used by the extraction engine.

4. **Feature selection subsystem**
   - `tsfresh/feature_selection` implements the statistical filtering layer.
   - `relevance.py` is the central relevance-testing logic, deciding which extracted features are predictive.
   - `significance_tests.py` contains the hypothesis tests used during filtering.
   - `selection.py` applies the feature-relevance results to produce a reduced feature set.
   - This subsystem is the main architectural boundary between “feature generation” and “model-ready subset.”

5. **Convenience/API layer**
   - `tsfresh/convenience` exposes higher-level user entry points that combine extraction and selection.
   - `bindings.py` provides simpler, opinionated function bindings for common workflows.
   - `relevant_extraction.py` supports the common “extract only relevant features” path, hiding lower-level orchestration.

6. **sklearn integration**
   - `tsfresh/transformers` contains estimator-style wrappers that make the library composable in scikit-learn pipelines.
   - `feature_augmenter.py` and `relevant_feature_augmenter.py` add extracted features back to an input frame.
   - `feature_selector.py` exposes selection as a transformer.
   - `per_column_imputer.py` handles missing-value filling as part of pipeline preparation.
   - This is the main boundary for using tsfresh inside standard ML workflows rather than as a standalone preprocessing step.

7. **Utility layer**
   - `tsfresh/utilities` contains supporting infrastructure that is not domain-specific to feature extraction.
   - `dataframe_functions.py` likely handles pandas reshaping/alignment and long-to-wide conversions.
   - `distribution.py` and `profiling.py` support execution/runtime concerns.
   - `string_manipulation.py` supports naming/identifier handling for generated features.

8. **Example datasets and reusable domain demos**
   - `tsfresh/examples` packages small reference datasets and example loaders such as `har_dataset.py`, `driftbif_simulation.py`, and `robot_execution_failures.py`.
   - These examples act as integration fixtures for documentation and tests, and demonstrate real data ingestion patterns.

9. **Command-line / script entry points**
   - `tsfresh/scripts` provides runnable utilities around extraction timing and the main `run_tsfresh.py` entry point.
   - `measure_execution_time.py` and `test_timing.py` indicate a performance-monitoring angle for feature calculator throughput.
   - This suggests the library supports both library usage and command-line execution.

10. **Testing strategy**
    - `tests/units` covers low-level behavior per subsystem: feature extraction, feature selection, transformers, utilities, and scripts.
    - `tests/integrations` exercises end-to-end paths, including examples, notebooks, bindings, and full pipelines.
    - The presence of `dask-worker-space` under integration tests indicates distributed execution is part of the validation surface.

11. **Documentation and learning assets**
    - `docs/` is extensive and acts as a major part of the project architecture, not just ancillary material.
    - It documents the conceptual model, data formats, feature calculation, selection, sklearn transformers, cluster usage, and large-data workflows.
    - `notebooks/` provide executable examples of the main user journeys: feature extraction, sklearn pipelines, forecasting, multiclass selection, and advanced analysis.

12. **Execution/runtime considerations**
    - The repo is Python-packaged and organized as a pure library rather than a service.
    - Dependency management and quality gates are driven by `setup.py`, `setup.cfg`, `Makefile`, and pre-commit config.
    - Dockerfiles, Binder config, and ReadTheDocs config indicate the project is designed to be runnable in containers, interactive notebooks, and hosted documentation environments.

13. **CI/CD and release boundaries**
    - GitHub Actions workflows cover testing, style checks, benchmark runs, and deployment.
    - ReadTheDocs builds the documentation site; Binder supports reproducible notebook demos.
    - This means the repo’s operational boundary is mostly around packaging, validation, docs publishing, and benchmark/deploy automation rather than runtime infrastructure.

14. **What to diagram as the main architecture**
    - The clearest diagram would show three layers: data ingestion/reshaping, feature extraction + statistical relevance filtering, and downstream ML integration via sklearn-compatible transformers.
    - Around that core are example datasets, CLI scripts, utilities, tests, and documentation.
    - The primary internal boundary is between calculator-based feature generation and hypothesis-test-based feature selection.

Main entry points: [`tsfresh/feature_extraction/extraction.py`](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_extraction/extraction.py) (Extraction engine), [`docs`](https://github.com/blue-yonder/tsfresh/tree/main/docs) (Docs), [`tsfresh/feature_selection/selection.py`](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_selection/selection.py) (Selection), [`tsfresh/feature_selection/relevance.py`](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_selection/relevance.py) (Relevance)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core pipeline"]
  node_tsfresh_defaults["Defaults<br/>package config<br/>[defaults.py]"]
  node_fe_data["Data helpers<br/>feature prep<br/>[data.py]"]
  node_fe_settings["Settings<br/>feature config<br/>[settings.py]"]
  node_fe_calculators["Calculators<br/>feature defs"]
  node_fe_extraction["Extraction engine<br/>feature pipeline<br/>[extraction.py]"]
  node_fs_significance["Significance tests<br/>stats tests"]
  node_fs_relevance["Relevance<br/>selection logic<br/>[relevance.py]"]
  node_fs_selection["Selection<br/>feature filter<br/>[selection.py]"]
end

subgraph group_api["User-facing API"]
  node_conv_bindings["Bindings<br/>convenience API<br/>[bindings.py]"]
  node_conv_relevant["Relevant extract<br/>convenience API"]
  node_tr_feature_aug["Feature augmenter<br/>sklearn transformer"]
  node_tr_relevant_aug["Relevant augmenter<br/>sklearn transformer"]
  node_tr_selector["Feature selector<br/>sklearn transformer"]
  node_tr_imputer["Imputer<br/>pipeline prep"]
end

subgraph group_support["Support assets"]
  node_examples_har["HAR data<br/>example dataset<br/>[har_dataset.py]"]
  node_examples_drift["Drift sim<br/>example dataset"]
  node_examples_robot["Robot failures<br/>example dataset"]
  node_script_run["Run CLI<br/>script entrypoint<br/>[run_tsfresh.py]"]
  node_script_timing["Timing tools<br/>benchmark script"]
  node_util_df["DataFrame utils<br/>pandas utilities"]
  node_util_dist["Distribution<br/>runtime utils<br/>[distribution.py]"]
  node_util_profile["Profiling<br/>runtime utils<br/>[profiling.py]"]
  node_util_string["String utils<br/>naming utils"]
end

subgraph group_quality["Quality and docs"]
  node_tests_units["Unit tests<br/>test suite"]
  node_tests_integ["Integration tests<br/>test suite"]
  node_docs["Docs<br/>documentation"]
  node_notebooks["Notebooks<br/>examples"]
end

node_tsfresh_defaults -->|"configures"| node_fe_settings
node_util_df -->|"shapes"| node_fe_data
node_fe_data -->|"input prep"| node_fe_extraction
node_fe_settings -->|"selects"| node_fe_extraction
node_fe_calculators -->|"computes"| node_fe_extraction
node_fe_extraction -->|"feature matrix"| node_fs_relevance
node_fs_significance -->|"tests"| node_fs_relevance
node_fs_relevance -->|"ranks"| node_fs_selection
node_fe_extraction -->|"exposes"| node_conv_bindings
node_fs_selection -->|"filters"| node_conv_relevant
node_fe_extraction -->|"wraps"| node_tr_feature_aug
node_fe_extraction -->|"wraps"| node_tr_relevant_aug
node_fs_selection -->|"wraps"| node_tr_selector
node_tr_imputer -->|"prepares"| node_tr_feature_aug
node_examples_har -.->|"illustrates"| node_docs
node_examples_drift -.->|"illustrates"| node_docs
node_examples_robot -.->|"illustrates"| node_docs
node_script_run -->|"invokes"| node_fe_extraction
node_script_run -->|"invokes"| node_fs_selection
node_script_timing -->|"benchmarks"| node_fe_calculators
node_util_profile -->|"supports"| node_script_timing
node_util_string -->|"names"| node_fe_calculators
node_tests_units -.->|"covers"| node_fe_extraction
node_tests_units -.->|"covers"| node_fs_selection
node_tests_units -.->|"covers"| node_tr_feature_aug
node_tests_integ -.->|"covers"| node_conv_bindings
node_tests_integ -.->|"covers"| node_notebooks
node_tests_integ -.->|"covers"| node_examples_har
node_docs -.->|"documents"| node_fe_extraction
node_docs -.->|"documents"| node_fs_relevance
node_docs -.->|"documents"| node_tr_selector
node_docs -.->|"documents"| node_util_df

click node_tsfresh_defaults "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/defaults.py"
click node_fe_data "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_extraction/data.py"
click node_fe_settings "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_extraction/settings.py"
click node_fe_calculators "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_extraction/feature_calculators.py"
click node_fe_extraction "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_extraction/extraction.py"
click node_fs_significance "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_selection/significance_tests.py"
click node_fs_relevance "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_selection/relevance.py"
click node_fs_selection "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_selection/selection.py"
click node_conv_bindings "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/convenience/bindings.py"
click node_conv_relevant "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/convenience/relevant_extraction.py"
click node_tr_feature_aug "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/transformers/feature_augmenter.py"
click node_tr_relevant_aug "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/transformers/relevant_feature_augmenter.py"
click node_tr_selector "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/transformers/feature_selector.py"
click node_tr_imputer "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/transformers/per_column_imputer.py"
click node_examples_har "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/examples/har_dataset.py"
click node_examples_drift "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/examples/driftbif_simulation.py"
click node_examples_robot "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/examples/robot_execution_failures.py"
click node_script_run "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/scripts/run_tsfresh.py"
click node_script_timing "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/scripts/measure_execution_time.py"
click node_util_df "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/utilities/dataframe_functions.py"
click node_util_dist "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/utilities/distribution.py"
click node_util_profile "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/utilities/profiling.py"
click node_util_string "https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/utilities/string_manipulation.py"
click node_tests_units "https://github.com/blue-yonder/tsfresh/tree/main/tests/units"
click node_tests_integ "https://github.com/blue-yonder/tsfresh/tree/main/tests/integrations"
click node_docs "https://github.com/blue-yonder/tsfresh/tree/main/docs"
click node_notebooks "https://github.com/blue-yonder/tsfresh/tree/main/notebooks"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tsfresh_defaults,node_fe_data,node_fe_settings,node_fe_calculators,node_fe_extraction,node_fs_significance,node_fs_relevance,node_fs_selection toneBlue
class node_conv_bindings,node_conv_relevant,node_tr_feature_aug,node_tr_relevant_aug,node_tr_selector,node_tr_imputer toneAmber
class node_examples_har,node_examples_drift,node_examples_robot,node_script_run,node_script_timing,node_util_df,node_util_dist,node_util_profile,node_util_string toneMint
class node_tests_units,node_tests_integ,node_docs,node_notebooks toneRose
```

## Components

### Core pipeline

Time-series feature extraction and statistical feature selection.

- [Defaults](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/defaults.py): package config · Library-wide defaults and shared configuration constants. · `tsfresh/defaults.py`
- [Data helpers](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_extraction/data.py): feature prep · Shaping helpers used by the extraction engine. · `tsfresh/feature_extraction/data.py`
- [Settings](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_extraction/settings.py): feature config · Extraction presets and calculator selection. · `tsfresh/feature_extraction/settings.py`
- [Calculators](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_extraction/feature_calculators.py): feature defs · Feature calculator implementations and extensibility point. · `tsfresh/feature_extraction/feature_calculators.py`
- [Extraction engine](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_extraction/extraction.py): feature pipeline · Coordinates grouping, rolling, and assembly of extracted features. · `tsfresh/feature_extraction/extraction.py`
- [Significance tests](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_selection/significance_tests.py): stats tests · Hypothesis tests used during relevance filtering. · `tsfresh/feature_selection/significance_tests.py`
- [Relevance](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_selection/relevance.py): selection logic · Determines which extracted features are predictive. · `tsfresh/feature_selection/relevance.py`
- [Selection](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/feature_selection/selection.py): feature filter · Applies relevance results to reduce the feature matrix. · `tsfresh/feature_selection/selection.py`

### User-facing API

Convenience wrappers and sklearn-compatible transformers.

- [Bindings](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/convenience/bindings.py): convenience API · Opinionated high-level function bindings for common workflows. · `tsfresh/convenience/bindings.py`
- [Relevant extract](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/convenience/relevant_extraction.py): convenience API · End-to-end helper for extracting only relevant features. · `tsfresh/convenience/relevant_extraction.py`
- [Feature augmenter](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/transformers/feature_augmenter.py): sklearn transformer · Adds extracted features back onto an input frame. · `tsfresh/transformers/feature_augmenter.py`
- [Relevant augmenter](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/transformers/relevant_feature_augmenter.py): sklearn transformer · Augmenter variant that keeps only relevant features. · `tsfresh/transformers/relevant_feature_augmenter.py`
- [Feature selector](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/transformers/feature_selector.py): sklearn transformer · Wraps feature selection for pipeline use. · `tsfresh/transformers/feature_selector.py`
- [Imputer](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/transformers/per_column_imputer.py): pipeline prep · Per-column missing-value filling for sklearn workflows. · `tsfresh/transformers/per_column_imputer.py`

### Support assets

Examples, scripts, utilities, and packaging around the core library.

- [HAR data](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/examples/har_dataset.py): example dataset · Reference dataset loader for human activity recognition. · `tsfresh/examples/har_dataset.py`
- [Drift sim](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/examples/driftbif_simulation.py): example dataset · Synthetic drift/bifurcation simulation example. · `tsfresh/examples/driftbif_simulation.py`
- [Robot failures](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/examples/robot_execution_failures.py): example dataset · Real-world robot execution failure example loader. · `tsfresh/examples/robot_execution_failures.py`
- [Run CLI](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/scripts/run_tsfresh.py): script entrypoint · Main command-line entry point for extraction workflows. · `tsfresh/scripts/run_tsfresh.py`
- [Timing tools](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/scripts/measure_execution_time.py): benchmark script · Measures feature calculator execution time. · `tsfresh/scripts/measure_execution_time.py`
- [DataFrame utils](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/utilities/dataframe_functions.py): pandas utilities · Reshaping, alignment, and long-to-wide helpers. · `tsfresh/utilities/dataframe_functions.py`
- [Distribution](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/utilities/distribution.py): runtime utils · Execution/distribution support helpers. · `tsfresh/utilities/distribution.py`
- [Profiling](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/utilities/profiling.py): runtime utils · Profiling helpers for execution analysis. · `tsfresh/utilities/profiling.py`
- [String utils](https://github.com/blue-yonder/tsfresh/blob/main/tsfresh/utilities/string_manipulation.py): naming utils · Feature naming and identifier manipulation helpers. · `tsfresh/utilities/string_manipulation.py`

### Quality and docs

Tests, documentation, notebooks, and release/CI scaffolding.

- [Unit tests](https://github.com/blue-yonder/tsfresh/tree/main/tests/units): test suite · Subsystem-level behavioral tests. · `tests/units`
- [Integration tests](https://github.com/blue-yonder/tsfresh/tree/main/tests/integrations): test suite · End-to-end tests covering examples, bindings, notebooks, and pipelines. · `tests/integrations`
- [Docs](https://github.com/blue-yonder/tsfresh/tree/main/docs): documentation · User guide, API docs, and conceptual walkthroughs. · `docs`
- [Notebooks](https://github.com/blue-yonder/tsfresh/tree/main/notebooks): examples · Executable walkthroughs for extraction, pipelines, forecasting, and analysis. · `notebooks`

## Connections

- Defaults → Settings: configures · Shared defaults feed extraction presets and behavior.
- DataFrame utils → Data helpers: shapes · DataFrame helpers prepare and align time-series tables.
- Data helpers → Extraction engine: input prep · Extraction engine consumes reshaped series data.
- Settings → Extraction engine: selects · Settings determine which calculators run.
- Calculators → Extraction engine: computes · Calculator definitions are executed by the extraction engine.
- Extraction engine → Relevance: feature matrix · Extracted features are passed to relevance testing.
- Significance tests → Relevance: tests · Statistical tests underpin relevance decisions.
- Relevance → Selection: ranks · Relevance results drive final feature filtering.
- Extraction engine → Bindings: exposes · Convenience functions wrap extraction workflows.
- Selection → Relevant extract: filters · Relevant extraction combines generation and selection.
- Extraction engine → Feature augmenter: wraps · Transformer augments the input with extracted features.
- Extraction engine → Relevant augmenter: wraps · Relevant augmenter reuses extraction plus selection.
- Selection → Feature selector: wraps · Selection is exposed as an sklearn transformer.
- Imputer → Feature augmenter: prepares · Imputation is used before feature augmentation in pipelines.
- HAR data → Docs: illustrates · Example datasets support documentation and demos.
- Drift sim → Docs: illustrates · Synthetic examples are referenced in docs and learning materials.
- Robot failures → Docs: illustrates · Real-world examples support narrative documentation.
- Run CLI → Extraction engine: invokes · CLI entry point runs extraction workflows.
- Run CLI → Selection: invokes · CLI can apply relevance-based reduction.
- Timing tools → Calculators: benchmarks · Timing script measures calculator throughput.
- Profiling → Timing tools: supports · Profiling helpers support performance measurement.
- String utils → Calculators: names · String helpers support generated feature identifiers.
- Unit tests → Extraction engine: covers · Unit tests validate extraction internals.
- Unit tests → Selection: covers · Unit tests validate relevance and selection logic.
- Unit tests → Feature augmenter: covers · Unit tests validate sklearn wrappers and imputers.
- Integration tests → Bindings: covers · Integration tests exercise convenience APIs.
- Integration tests → Notebooks: covers · Integration tests validate notebook-based workflows.
- Integration tests → HAR data: covers · Integration tests validate example loaders.
- Docs → Extraction engine: documents · Documentation explains feature extraction concepts and settings.
- Docs → Relevance: documents · Documentation explains feature filtering and selection.
- Docs → Feature selector: documents · Documentation covers sklearn transformers.
- Docs → DataFrame utils: documents · Documentation covers data formats and reshaping helpers.
