# easezyc/metaheac architecture

> Architecture diagram of the GitHub repository easezyc/metaheac, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/easezyc/metaheac
- Repository: https://github.com/easezyc/metaheac

## Overview

##### Project type and purpose
- This is a research‑grade, full‑stack machine‑learning framework implementing the “Meta Hybrid Experts and Critics” (MetaHeac) audience‑expansion model for recommendation and advertising.
- It supports two stages:
- Offline meta‑learning across many past campaigns to produce a general model.
- Online per‑campaign fine‑tuning and inference to expand a new seed audience.

##### File structure
Top level:
- data/
– process.py      • raw data → preprocessing → processed_data/
- main.py           • entry point to orchestrate both offline and online modes
- run.py            • concrete training & evaluation loops
- metamodel.py      • meta‑training logic (outer loop of meta‑learning)
- model.py          • neural network definitions (experts & critics)
- utils.py          • sampling, metrics, logging, argument parsing
- readme.md         • high‑level description, requirements, and usage

##### README for additional insights
- Confirms two‑stage framework, required Python libs (PyTorch, Pandas, NumPy), CLI parameters for tasks, experts, critics, batch size, GPU.
- Data flow: raw data → `python process.py` → `data/processed_data` → `python main.py` for training.

a. Main components
1. Data Ingestion & Preprocessing
– process.py reads raw dataset → writes to processed_data.
2. Processed Data Store
– filesystem or object store (e.g., HDFS/S3).
3. Offline Meta‑Learning Pipeline
– metamodel.py + run.py
– loads processed data in mini‑batches of tasks → meta‑train general model → checkpoint registry.
4. Model Registry
– storage for “general” MetaHeac model artifacts.
5. Online Fine‑Tuning & Inference Pipeline
– main.py (online mode)
– loads general model + new campaign seed users → fine‑tunes critic/expert weights → inference.
6. Audience Output & Delivery
– expanded user list → marketing platform API or file export.
7. Compute Environment
– GPU cluster or single node with Python 3.6+, PyTorch 1.x.
8. Utilities
– utils.py for batching, logging, metrics; config via CLI args.

b. Relationships & data flow
- Raw data → Preprocessing → Processed Data Store
- Processed Data → Offline Meta‑Learning → General Model
- General Model + Seed Users → Online Fine‑Tuning → Expanded Audience
- Metrics from both stages feed back to logs/monitoring.

c. Architectural patterns & design principles
- Two‑stage pipeline pattern (offline versus online).
- Modular code separation: data, model definitions, meta‑training, orchestration.
- Reusable model artifacts (registry).
- Configurable via CLI for reproducibility.

d. Key technologies & libraries
- Python 3.6
- PyTorch (>1.0) for NN experts & critics
- Pandas, NumPy for data handling
- Filesystem or cloud object store for data/model artifacts

##### Essential details to capture
- Separation of offline meta‑learning vs. online fine‑tuning
- Data flow from raw → processed → model → output
- How metamodel.py and model.py collaborate under run.py/main.py
- External dependencies (PyTorch, GPU) and storage layers

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    DI["Data Ingestion & Preprocessing"]:::data
    PDS[(Processed Data Store)]:::storage

    %% Offline Stage
    Offline["Offline Meta-Learning Pipeline"]:::offline
    MT["Meta-Training Logic"]:::offline
    MD["Model Definitions\n(Experts & Critics)"]:::offline
    MR[/Model Registry/]:::artifact

    %% Online Stage
    Online["Online Fine-Tuning & Inference Pipeline"]:::online
    AO["Audience Output & Delivery"]:::online

    %% Utilities and Docs
    UTIL["Utilities (sampling, metrics,\nlogging, CLI args)"]:::util
    DOC["Documentation & Usage"]:::util

    %% Compute
    COMP["Compute Environment\n(GPU Cluster)"]:::compute

    %% Data flow
    DI -->|"processed_data"| PDS
    PDS -->|loads processed_data| Offline
    Offline -->|trains meta_model.pt| MR
    MR -->|provides general model| Online
    MT -->|"outer loop tasks"| Offline
    MD -->|"defines networks"| Offline
    MD -->|"defines networks"| Online

    %% Utilities and Compute
    UTIL --> Offline
    UTIL --> Online
    COMP --> Offline
    COMP --> Online

    %% Online Output
    Online -->|"expanded_audience.csv"| AO

    %% Documentation
    DOC --> DI
    DOC --> Offline
    DOC --> Online

    %% Click Events
    click DI "https://github.com/easezyc/metaheac/blob/master/data/process.py"
    click PDS "https://github.com/easezyc/metaheac/tree/master/data/"
    click Offline "https://github.com/easezyc/metaheac/blob/master/run.py"
    click MT "https://github.com/easezyc/metaheac/blob/master/metamodel.py"
    click MD "https://github.com/easezyc/metaheac/blob/master/model.py"
    click Online "https://github.com/easezyc/metaheac/blob/master/main.py"
    click UTIL "https://github.com/easezyc/metaheac/blob/master/utils.py"
    click DOC "https://github.com/easezyc/metaheac/blob/master/readme.md"

    %% Styles
    classDef data fill:#FADBD8,stroke:#C0392B
    classDef storage fill:#FCF3CF,stroke:#B7950B
    classDef offline fill:#D6EAF8,stroke:#2980B9
    classDef online fill:#EAFAF1,stroke:#27AE60
    classDef util fill:#F4ECF7,stroke:#8E44AD
    classDef compute fill:#E8F8F5,stroke:#148F77
    classDef artifact fill:#D5F5E3,stroke:#1E8449
```
