How reczoo/fuxictr works
Project type and purpose
• This is an open-source library (FuxiCTR) for click-through rate (CTR) prediction, providing configurable data preprocessing, model training, hyperparameter tuning, and a rich model zoo across both PyTorch and TensorFlow frameworks.
• Users consume it via command-line scripts (in demo/, model_zoo/.../run_expid.py, experiment/run_param_tuner.py) or import the core API (fuxictr/).
File structure
• Top-level components:
• fuxictr/ – core library modules (datasets, preprocessors, feature processors, metrics, framework adapters)
• model_zoo/ – dozens of subpackages, each encapsulating a specific published CTR model with its own configs and training script
• demo/ – example scripts plus example configs for quickstart
• experiment/ – hyperparameter‐tuning orchestration
• data/ – tiny sample datasets in multiple formats (CSV, NPZ, Parquet, TFRecord)
• docs/ – static site source for user tutorials and quickstart
• CI/CD workflows, packaging (setup.py, requirements.txt), tests.
README for additional insights
• Key features: Configurable (YAML‐based), Tunable (grid search), Reproducible, Extensible.
• Workflow:
a) Build dataset (via demo or fuxictr.preprocess.build_dataset)
b) Load dataset (via fuxictr.pytorch.dataloaders or fuxictr.tensorflow.dataloaders)
c) Instantiate model (from core or model_zoo)
d) Train/Evaluate (framework trainer + metrics)
e) Tune (in experiment/)
1. CLI Layer (scripts): demo examples, run_expid, run_param_tuner
2. Config Manager: YAML parsers for dataset_config.yaml, model_config.yaml, tuner configs
3. Data Layer:
• Raw data (data storage)
• Preprocessor (fuxictr.preprocess)
• Feature Processor (fuxictr.features)
• Normalizer/Tokenizer
4. Data Loader Abstractions:
• PyTorch Dataloaders (npz_dataloader, parquet_dataloader, etc.)
• TF Dataloader (tf_dataloader)
5. Model Layer:
• Core model interfaces (fuxictr.pytorch.models, fuxictr.tensorflow.models)
• Model zoo plug-ins (each model_zoo/*/src/*.py)
6. Training Engine:
• PyTorch Trainer (loss, optimizer, metrics)
• TensorFlow Trainer
7. Metric & Logging:
• fuxictr.metrics
• Experiment logs, checkpoints
8. Auto-Tuner:
• fuxictr.autotuner
• Orchestrator in experiment/run_param_tuner.py
9. Documentation & CI/CD (side component): docs site generator, GitHub Actions
b. Relationships & data flows:
1. CLI Layer → reads Config → invokes Preprocessor
2. Preprocessor → writes processed Dataset → passes to Data Loader
3. Data Loader → feeds batched tensors to Model Layer
4. Model Layer → Prediction + Loss → Training Engine
5. Training Engine → updates Model → outputs metrics → logs
6. Auto-Tuner duplicates Config → parallel CLI invocations → aggregate results
c. Architectural patterns & principles:
• Plugin/Factory pattern for data formats and model implementations
• Strategy pattern: selecting PyTorch vs TensorFlow backends
• Separation of concerns: clear modularization of preprocessing, loading, modeling, tuning
• Configuration-driven design: all behaviors driven by YAML files
d. Key technologies & frameworks:
• Python 3.9+
• PyTorch (1.10+)
• TensorFlow (2.1+)
• Parquet / NPZ / CSV / TFRecord data formats
• YAML for configs
• GitHub Actions for CI/CD
• Jupyter Book / Sphinx (docs)