How sb-ai-lab/hypex works
1. What this repo is
HypEx is a Python analytics library, not an application service. Its architecture is centered on experiment analysis pipelines for:
- A/B and A/A testing
- causal inference / matching
- homogeneity checks
- variance reduction methods like CUPED/CUPAC
2. Public entry points
The top-level user API is intentionally small and lives in hypex/:
- hypex.ab, hypex.aa, hypex.matching, hypex.homogeneity
- these expose the main experiment objects used in the README examples: ABTest, AATest, Matching
These are the facade layer that hides the internal pipeline.
3. Core domain model: roles + dataset
The hypex.dataset package is the data boundary. It wraps tabular input and assigns semantic roles to columns:
- ID / info
- treatment
- target
- stratification
- default feature roles
It also has a backend abstraction with a concrete Pandas implementation, so the rest of the library can treat input as a typed dataset rather than raw DataFrames.
4. Preprocessing and feature shaping
hypex.transformers and hypex.preprocessing are the data preparation layer. They handle tasks such as:
- type casting
- NA filling
- filtering / outlier handling
- shuffling
- category aggregation
- CUPED transformation
These run before analysis so downstream comparators and estimators receive cleaned, role-aware columns.
5. Experiment orchestration layer
hypex.experiments and hypex.executor appear to coordinate the workflow:
- select the analysis path
- apply calculators
- produce structured outputs
executor/calculators.py suggests a computation stage separate from orchestration, which is useful for running different statistical routines through a shared execution pipeline.
6. Analysis engines
The hypex.analyzers package holds the main analytical logic for different experiment types:
- aa.py for A/A validation
- ab.py for A/B testing
- matching.py for matching-based causal inference
These analyzers likely consume prepared datasets and produce test statistics, effect estimates, and validation metrics.
7. Matching subsystem
Matching is a distinct subsystem rather than just one test mode. It spans:
- hypex.matching
- hypex.splitters
- hypex.encoders
- hypex.comparators
- hypex.ml and hypex.extensions.faiss
This suggests a pipeline that encodes features, computes distances/similarities, and pairs treated/control units efficiently, with Faiss as the accelerated nearest-neighbor backend.
8. Statistical comparison layer
hypex.comparators contains reusable statistical primitives:
- distance metrics
- hypothesis testing
- power testing
- abstract comparator interfaces
These are shared across AB, AA, and matching workflows, which keeps the statistical logic separate from experiment-specific orchestration.
9. Results and reporting
hypex.reporters and hypex.ui are the presentation layer.
- Reporters format result objects for AA, AB, matching, and homogeneity workflows.
- UI modules likely provide a user-facing wrapper around those reports.
This is the boundary where raw statistical outputs become readable summaries such as resume, variance reduction reports, or match summaries.
10. Extensions and optional dependencies
hypex.extensions isolates integrations with external libraries:
- Faiss for nearest-neighbor search
- SciPy stats/linalg
- Statsmodels
- encoder integrations
- CUPAC support
This is an important architectural boundary: the core API depends on these capabilities, but the implementation is modular so optional scientific packages are isolated.
11. Supporting abstractions
Several packages provide the internal framework:
- operators, factory, forks, utils
- abstract base classes throughout the package
- hypothesis schemes in hypex/hypotheses/schemes/scheme.json
This indicates a plugin-like or strategy-based architecture, where test variants and computational steps are assembled from reusable components.
12. Repository shape around the library
The rest of the repo supports the library itself:
- examples/ contains notebooks and a performance test harness
- docs/ is a full Sphinx documentation site
- tests/ and unitests/ validate examples and tutorial behavior
- CI and packaging are set up via GitHub Actions, pyproject.toml, and tox.ini
So the repo is organized as a distributable Python package with documentation, reproducible examples, and regression tests.
13. Overall data flow
The typical pipeline is:
raw tabular data -> role assignment via Dataset -> preprocessing/transformers -> analyzer/executor/comparator logic -> reporter/UI output.
Matching adds a feature-encoding + distance/search path in the middle, and AB testing can optionally branch through CUPED/CUPAC variance reduction before statistical comparison.
14. Main architectural boundary to draw
For a diagram, the key split is between:
- public experiment facades (ABTest, AATest, Matching, Homogeneity)
- internal execution engine (dataset, transformers, analyzers, comparators, executor, reporters)
- optional acceleration/statistics backends (extensions, ml, Faiss/SciPy/Statsmodels)
That separation is the defining structure of the repository.
Main entry points: experiments Experiments · analyzers Analyzers · matching.py Matching Flow · dataset Dataset
Components 24
Public API
Small facade layer for experiment types and user-facing entry points.
- AB Test · Top-level AB testing entry point.
hypex/ab.py - AA Test · Top-level A/A testing entry point.
hypex/aa.py - Matching · Top-level matching entry point.
hypex/matching.py - Homogeneity · Top-level homogeneity analysis entry point.
hypex/homogeneity.py
Core Engine
Dataset boundary, preprocessing, orchestration, analyzers, comparators, and reporting.
- Dataset · Wraps tabular input and assigns semantic roles through backend abstractions.
hypex/dataset - Transformers · Cleans and reshapes data with casting, NA filling, filtering, shuffling, aggregation, and CUPED.
hypex/transformers - Preprocess · Convenience layer for data preparation before analysis.
hypex/preprocessing.py - Experiments · Coordinates analysis paths and experiment assembly.
hypex/experiments - Executor · Runs calculators and shared execution steps.
hypex/executor - Analyzers · AB, AA, and matching analysis logic that consumes prepared datasets.
hypex/analyzers - Comparators · Distance, hypothesis testing, power testing, and comparison interfaces.
hypex/comparators - Reporters · Formats analysis outputs into readable reports for each workflow.
hypex/reporters - UI · User-facing wrappers around report output.
hypex/ui
Matching Stack
Feature encoding, splitting, and accelerated nearest-neighbor matching path.
- Matching Flow · Feature encoding and matching workflow for causal pairing.
hypex/matching.py - Splitters · Splits units for matching-oriented workflows.
hypex/splitters - Encoders · Encodes features before comparison or nearest-neighbor search.
hypex/encoders - Faiss ML · Nearest-neighbor acceleration used by matching flows.
hypex/ml/faiss.py
Supporting Code
Shared abstractions, utilities, extensions, and hypothesis schemes.
- Extensions · Isolates optional scientific and acceleration integrations such as Faiss, SciPy, Statsmodels, and CUPAC.
hypex/extensions - Operators · Strategy-like operator interfaces and implementations.
hypex/operators - Factory · Creates and wires core experiment components.
hypex/factory - Forks · Workflow variants and specialized branches.
hypex/forks - Hypotheses · Hypothesis models and scheme definitions.
hypex/hypotheses - Utils · Constants, enums, models, errors, typings, and adapter helpers.
hypex/utils
Repo Support
Examples, docs, and tests around the library package.
- Docs & Tests · Documentation site, examples, and regression tests around the library.
docs
Connections 27
- AB Test → Experiments: uses · AB facade delegates orchestration to experiment layer.
- AA Test → Experiments: uses · AA facade delegates orchestration to experiment layer.
- Matching → Matching Flow: uses · Matching facade exposes the matching workflow.
- Homogeneity → Experiments: uses · Homogeneity entry point routes through shared experiment plumbing.
- Experiments → Dataset: consumes · Experiments operate on role-aware datasets.
- Dataset → Transformers: feeds · Prepared dataset flows into preprocessing.
- Preprocess → Transformers: wraps · Preprocessing is the user-facing preparation layer.
- Transformers → Comparators: prepares for · Cleaned data is passed to statistical comparison logic.
- Experiments → Analyzers: selects · Orchestration chooses the relevant analysis engine.
- Analyzers → Executor: delegates · Analyzers rely on shared execution/calculation pipeline.
- Executor → Comparators: runs · Execution stage invokes statistical primitives and calculators.
- Analyzers → Reporters: emits · Analysis results are packaged for reporting.
- Reporters → UI: formats for · Reports are exposed through user-facing UI wrappers.
- Matching Flow → Encoders: encodes · Matching flow starts with feature encoding.
- Matching Flow → Splitters: splits · Matching flow partitions units before pairing.
- Matching Flow → Faiss ML: searches · Matching flow can accelerate pairing with Faiss.
- Encoders → Extensions: adapts · Encoder integrations are isolated in extensions.
- Faiss ML → Extensions: adapts · Faiss backend is wrapped as optional extension.
- Comparators → Extensions: uses · Statistical routines rely on optional SciPy/Statsmodels backends.
- Analyzers → Hypotheses: applies · Analysis flows use hypothesis schemes and definitions.
- Factory → Experiments: assembles · Factory wires experiment objects and components.
- Operators → Analyzers: supports · Operators provide reusable execution abstractions.
- Forks → Experiments: specializes · Forked variants customize experiment behavior.
- Utils → Dataset: supports · Utilities provide models, errors, enums, and adapter helpers.
- Docs & Tests → AB Test: covers · Docs and tests exercise public experiment entry points.
- Docs & Tests → Dataset: covers · Tutorials and tests demonstrate dataset handling.
- Docs & Tests → Matching Flow: covers · Examples document the matching workflow.