How easezyc/metaheac works
Project type and purpose
• This is a research‑grade, full‑stack machine‑learning framework implementing the “Meta Hybrid Experts and Critics” (MetaHeac) audience‑expansion model for recommendation and advertising.
• It supports two stages:
• Offline meta‑learning across many past campaigns to produce a general model.
• Online per‑campaign fine‑tuning and inference to expand a new seed audience.
File structure
Top level:
• data/
– process.py • raw data → preprocessing → processed_data/
• main.py • entry point to orchestrate both offline and online modes
• run.py • concrete training & evaluation loops
• metamodel.py • meta‑training logic (outer loop of meta‑learning)
• model.py • neural network definitions (experts & critics)
• utils.py • sampling, metrics, logging, argument parsing
• readme.md • high‑level description, requirements, and usage
README for additional insights
• Confirms two‑stage framework, required Python libs (PyTorch, Pandas, NumPy), CLI parameters for tasks, experts, critics, batch size, GPU.
• Data flow: raw data → python process.py → data/processed_data → python main.py for training.
a. Main components
1. Data Ingestion & Preprocessing
– process.py reads raw dataset → writes to processed_data.
2. Processed Data Store
– filesystem or object store (e.g., HDFS/S3).
3. Offline Meta‑Learning Pipeline
– metamodel.py + run.py
– loads processed data in mini‑batches of tasks → meta‑train general model → checkpoint registry.
4. Model Registry
– storage for “general” MetaHeac model artifacts.
5. Online Fine‑Tuning & Inference Pipeline
– main.py (online mode)
– loads general model + new campaign seed users → fine‑tunes critic/expert weights → inference.
6. Audience Output & Delivery
– expanded user list → marketing platform API or file export.
7. Compute Environment
– GPU cluster or single node with Python 3.6+, PyTorch 1.x.
8. Utilities
– utils.py for batching, logging, metrics; config via CLI args.
b. Relationships & data flow
• Raw data → Preprocessing → Processed Data Store
• Processed Data → Offline Meta‑Learning → General Model
• General Model + Seed Users → Online Fine‑Tuning → Expanded Audience
• Metrics from both stages feed back to logs/monitoring.
c. Architectural patterns & design principles
• Two‑stage pipeline pattern (offline versus online).
• Modular code separation: data, model definitions, meta‑training, orchestration.
• Reusable model artifacts (registry).
• Configurable via CLI for reproducibility.
d. Key technologies & libraries
• Python 3.6
• PyTorch (>1.0) for NN experts & critics
• Pandas, NumPy for data handling
• Filesystem or cloud object store for data/model artifacts
Essential details to capture
• Separation of offline meta‑learning vs. online fine‑tuning
• Data flow from raw → processed → model → output
• How metamodel.py and model.py collaborate under run.py/main.py
• External dependencies (PyTorch, GPU) and storage layers