# reczoo/fuxictr architecture

> Architecture diagram of the GitHub repository reczoo/fuxictr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/reczoo/fuxictr
- Repository: https://github.com/reczoo/fuxictr

## Overview

##### Project type and purpose
- This is an open-source library (FuxiCTR) for click-through rate (CTR) prediction, providing configurable data preprocessing, model training, hyperparameter tuning, and a rich model zoo across both PyTorch and TensorFlow frameworks.
- Users consume it via command-line scripts (in `demo/`, `model_zoo/.../run_expid.py`, `experiment/run_param_tuner.py`) or import the core API (`fuxictr/`).

##### File structure
- Top-level components:
- `fuxictr/` – core library modules (datasets, preprocessors, feature processors, metrics, framework adapters)
- `model_zoo/` – dozens of subpackages, each encapsulating a specific published CTR model with its own configs and training script
- `demo/` – example scripts plus example configs for quickstart
- `experiment/` – hyperparameter‐tuning orchestration
- `data/` – tiny sample datasets in multiple formats (CSV, NPZ, Parquet, TFRecord)
- `docs/` – static site source for user tutorials and quickstart
- CI/CD workflows, packaging (`setup.py`, `requirements.txt`), tests.

##### README for additional insights
- Key features: Configurable (YAML‐based), Tunable (grid search), Reproducible, Extensible.
- Workflow:
a) Build dataset (via demo or `fuxictr.preprocess.build_dataset`)
b) Load dataset (via `fuxictr.pytorch.dataloaders` or `fuxictr.tensorflow.dataloaders`)
c) Instantiate model (from core or model_zoo)
d) Train/Evaluate (framework trainer + metrics)
e) Tune (in `experiment/`)

1. CLI Layer (scripts): demo examples, run_expid, run_param_tuner
2. Config Manager: YAML parsers for `dataset_config.yaml`, `model_config.yaml`, tuner configs
3. Data Layer:
- Raw data (data storage)
- Preprocessor (`fuxictr.preprocess`)
- Feature Processor (`fuxictr.features`)
- Normalizer/Tokenizer
4. Data Loader Abstractions:
- PyTorch Dataloaders (`npz_dataloader`, `parquet_dataloader`, etc.)
- TF Dataloader (`tf_dataloader`)
5. Model Layer:
- Core model interfaces (`fuxictr.pytorch.models`, `fuxictr.tensorflow.models`)
- Model zoo plug-ins (each `model_zoo/*/src/*.py`)
6. Training Engine:
- PyTorch Trainer (loss, optimizer, metrics)
- TensorFlow Trainer
7. Metric & Logging:
- `fuxictr.metrics`
- Experiment logs, checkpoints
8. Auto-Tuner:
- `fuxictr.autotuner`
- Orchestrator in `experiment/run_param_tuner.py`
9. Documentation & CI/CD (side component): docs site generator, GitHub Actions

b. Relationships & data flows:
1. CLI Layer → reads Config → invokes Preprocessor
2. Preprocessor → writes processed Dataset → passes to Data Loader
3. Data Loader → feeds batched tensors to Model Layer
4. Model Layer → Prediction + Loss → Training Engine
5. Training Engine → updates Model → outputs metrics → logs
6. Auto-Tuner duplicates Config → parallel CLI invocations → aggregate results

c. Architectural patterns & principles:
- Plugin/Factory pattern for data formats and model implementations
- Strategy pattern: selecting PyTorch vs TensorFlow backends
- Separation of concerns: clear modularization of preprocessing, loading, modeling, tuning
- Configuration-driven design: all behaviors driven by YAML files

d. Key technologies & frameworks:
- Python 3.9+
- PyTorch (1.10+)
- TensorFlow (2.1+)
- Parquet / NPZ / CSV / TFRecord data formats
- YAML for configs
- GitHub Actions for CI/CD
- Jupyter Book / Sphinx (docs)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        Demo["Demo Scripts"]:::cli
        ExpRun["experiment/run_expid.py"]:::cli
        ZooRun["ModelZoo run_expid.py"]:::cli
        TunerRun["experiment/run_param_tuner.py"]:::cli
    end

    %% Configuration Manager
    Config[/Configuration Manager/]:::config

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RawData[(Raw Data)]:::data
        Preproc["fuxictr/preprocess/build_dataset"]:::core
        FeatureProc["fuxictr/preprocess/feature_processor"]:::core
        Normalizer["fuxictr/preprocess/normalizer"]:::core
        Tokenizer["fuxictr/preprocess/tokenizer"]:::core
        FeatureMap["fuxictr/features"]:::core
    end

    %% Data Loader Abstractions
    subgraph "Data Loaders"
        direction TB
        subgraph "PyTorch DataLoaders"
            direction TB
            NPZ["npz_dataloader"]:::pytorch
            NPZBlock["npz_block_dataloader"]:::pytorch
            Parquet["parquet_dataloader"]:::pytorch
            ParquetBlock["parquet_block_dataloader"]:::pytorch
            RankLoader["rank_dataloader"]:::pytorch
        end
        TFLoader["tf_dataloader"]:::tensorflow
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        subgraph "Core Models"
            direction TB
            PTCoreInit["fuxictr/pytorch/models/__init__.py"]:::pytorch
            PTMulti["multitask_model.py"]:::pytorch
            PTRank["rank_model.py"]:::pytorch
            TFCoreInit["fuxictr/tensorflow/models/__init__.py"]:::tensorflow
            TFRank["rank_model.py"]:::tensorflow
        end
        subgraph "Model Zoo Plugins"
            direction TB
            ZooPlugins["model_zoo/*/src"]:::modelzoo
            ZooConfigs["model_zoo/*/config"]:::modelzoo
        end
    end

    %% Training Engine
    subgraph "Training Engine & Utils"
        direction TB
        PTUtils["fuxictr/pytorch/torch_utils.py"]:::pytorch
        TFUtils["fuxictr/tensorflow/tf_utils.py"]:::tensorflow
    end

    %% Metrics & Logging
    Metrics["fuxictr/metrics.py"]:::core
    Checkpoints[(Logs & Checkpoints)]:::data

    %% Auto-Tuner
    subgraph "Auto-Tuner"
        direction TB
        AutoTuner["fuxictr/autotuner.py"]:::tuner
        ParamEnum["experiment/enumerate_param_list.py"]:::tuner
    end

    %% Documentation & CI/CD
    Docs[(docs/)]:::docs
    CI[".github/workflows"]:::docs
    Packaging[/setup.py, requirements.txt/]:::docs

    %% Connections
    Demo -->|reads YAML| Config
    ExpRun -->|reads YAML| Config
    ZooRun -->|reads YAML| Config
    TunerRun -->|reads YAML| Config

    Config -->|YAML→Parser| Preproc
    Preproc -->|processed data| RawData
    RawData -->|raw dataset| FeatureProc
    FeatureProc --> Normalizer
    Normalizer --> Tokenizer
    Tokenizer -->|features| FeatureMap
    FeatureMap -->|processed dataset| NPZ
    FeatureMap --> Parquet
    FeatureMap --> TFLoader

    NPZ -->|batches| PTCoreInit
    NPZBlock --> PTCoreInit
    Parquet --> PTCoreInit
    ParquetBlock --> PTCoreInit
    RankLoader --> PTCoreInit
    TFLoader --> TFCoreInit

    PTCoreInit -->|instantiates| PTMulti
    PTCoreInit --> PTRank
    TFCoreInit --> TFRank

    PTMulti -->|forward| PTUtils
    PTRank --> PTUtils
    TFRank --> TFUtils

    PTUtils -->|optimizes| PTMulti
    TFUtils -->|optimizes| TFRank

    PTMulti -->|metrics| Metrics
    PTRank --> Metrics
    TFRank --> Metrics

    Metrics --> Checkpoints
    PTUtils --> Checkpoints
    TFUtils --> Checkpoints

    Config -->|grid search| AutoTuner
    AutoTuner -->|launches| Demo
    AutoTuner -->|launches| ExpRun
    ParamEnum --> AutoTuner

    Docs -->|generates site| Packaging
    CI -->|runs tests| Packaging

    %% Click Events
    click Demo "https://github.com/reczoo/fuxictr/tree/main/demo/"
    click ExpRun "https://github.com/reczoo/fuxictr/blob/main/experiment/run_expid.py"
    click ZooRun "https://github.com/reczoo/fuxictr/tree/main/model_zoo/"
    click TunerRun "https://github.com/reczoo/fuxictr/blob/main/experiment/run_param_tuner.py"
    click Config "https://github.com/reczoo/fuxictr/blob/main/fuxictr/utils.py"
    click RawData "https://github.com/reczoo/fuxictr/tree/main/data/"
    click Preproc "https://github.com/reczoo/fuxictr/blob/main/fuxictr/preprocess/build_dataset.py"
    click FeatureProc "https://github.com/reczoo/fuxictr/blob/main/fuxictr/preprocess/feature_processor.py"
    click Normalizer "https://github.com/reczoo/fuxictr/blob/main/fuxictr/preprocess/normalizer.py"
    click Tokenizer "https://github.com/reczoo/fuxictr/blob/main/fuxictr/preprocess/tokenizer.py"
    click FeatureMap "https://github.com/reczoo/fuxictr/blob/main/fuxictr/features.py"
    click NPZ "https://github.com/reczoo/fuxictr/blob/main/fuxictr/pytorch/dataloaders/npz_dataloader.py"
    click NPZBlock "https://github.com/reczoo/fuxictr/blob/main/fuxictr/pytorch/dataloaders/npz_block_dataloader.py"
    click Parquet "https://github.com/reczoo/fuxictr/blob/main/fuxictr/pytorch/dataloaders/parquet_dataloader.py"
    click ParquetBlock "https://github.com/reczoo/fuxictr/blob/main/fuxictr/pytorch/dataloaders/parquet_block_dataloader.py"
    click RankLoader "https://github.com/reczoo/fuxictr/blob/main/fuxictr/pytorch/dataloaders/rank_dataloader.py"
    click TFLoader "https://github.com/reczoo/fuxictr/blob/main/fuxictr/tensorflow/dataloaders/tf_dataloader.py"
    click PTCoreInit "https://github.com/reczoo/fuxictr/blob/main/fuxictr/pytorch/models/__init__.py"
    click PTMulti "https://github.com/reczoo/fuxictr/blob/main/fuxictr/pytorch/models/multitask_model.py"
    click PTRank "https://github.com/reczoo/fuxictr/blob/main/fuxictr/pytorch/models/rank_model.py"
    click TFCoreInit "https://github.com/reczoo/fuxictr/blob/main/fuxictr/tensorflow/models/__init__.py"
    click TFRank "https://github.com/reczoo/fuxictr/blob/main/fuxictr/tensorflow/models/rank_model.py"
    click ZooPlugins "https://github.com/reczoo/fuxictr/tree/main/model_zoo/"
    click ZooConfigs "https://github.com/reczoo/fuxictr/tree/main/model_zoo/"
    click PTUtils "https://github.com/reczoo/fuxictr/blob/main/fuxictr/pytorch/torch_utils.py"
    click TFUtils "https://github.com/reczoo/fuxictr/blob/main/fuxictr/tensorflow/tf_utils.py"
    click Metrics "https://github.com/reczoo/fuxictr/blob/main/fuxictr/metrics.py"
    click AutoTuner "https://github.com/reczoo/fuxictr/blob/main/fuxictr/autotuner.py"
    click ParamEnum "https://github.com/reczoo/fuxictr/blob/main/experiment/enumerate_param_list.py"
    click Docs "https://github.com/reczoo/fuxictr/tree/main/docs/"
    click CI "https://github.com/reczoo/fuxictr/tree/main/.github/workflows/"
    click Packaging "https://github.com/reczoo/fuxictr/blob/main/setup.py"

    %% Styles
    classDef core fill:#FFEB3B,stroke:#333,stroke-width:1px
    classDef modelzoo fill:#FF9800,stroke:#333,stroke-width:1px
    classDef pytorch fill:#2196F3,stroke:#333,stroke-width:1px
    classDef tensorflow fill:#4CAF50,stroke:#333,stroke-width:1px
    classDef tuner fill:#9C27B0,stroke:#333,stroke-width:1px
    classDef config fill:#BDBDBD,stroke:#333,stroke-width:1px
    classDef data fill:#90A4AE,stroke:#333,stroke-width:1px
    classDef cli fill:#FF5722,stroke:#333,stroke-width:1px
    classDef docs fill:#795548,stroke:#333,stroke-width:1px
```
