How lileipisces/pod works
Major Components
• Data Sources & Input:
– The three domains (Sports, Beauty, Toys) serve as separate data sources.
– Domain-related shell scripts (sports.sh, beauty.sh, toys.sh) act as entry points that trigger experiments for their respective datasets.
• Data Processing & Preprocessing:
– Although not shown as dedicated folders, several Python files (e.g., utils.py, templates.py, seq.py) likely support data transformations, prompt templates, and preprocessing routines.
• Model Training & Distillation:
– pretrain.py serves as the module that orchestrates model pretraining using prompt distillation techniques.
– module.py (and potentially exp.py) encapsulate the core model definition and experiment logic.
• Evaluation:
– Files like bleu.py, rouge.py, and topn.py indicate the calculation of evaluation metrics (BLEU, ROUGE, Top-N metrics) to assess recommendation performance.
• Logging & Experiment Monitoring:
– The presence of domain-specific log files (sports.log, beauty.log, toys.log) shows that the experiments are monitored and output is logged for later review.
Establish Relationships and Data Flow
• Entry Points (sports.sh, beauty.sh, toys.sh) initiate domain-specific experiment runs.
• These scripts likely call the centralized experiment script (exp.py) which then directs data preprocessing functions (utils.py, templates.py, seq.py).
• The preprocessed data is fed into the model pipeline defined in module.py and carried through pretrain.py which handles model training with libraries such as PyTorch and transformers.
• After training, evaluation is performed with scripts (bleu.py, rouge.py, topn.py) that compute quality metrics on model outputs.
• Each stage generates log files (e.g., sports.log) that capture runtime data and performance metrics.
Architectural Patterns and Design Principles
• The architecture is modular: distinct scripts for data handling, training, and evaluation encourage separation of concerns.
• Experiment reusability is implied by having domain-specific shell scripts which exploit a common codebase and shared modules.
• There is a clear data flow: input (Amazon data) → preprocessing → model training via prompt distillation → evaluation and logging.
Technologies and Libraries to Highlight
• Python 3.6, PyTorch 1.6, transformers 4.18.0 (for model distillation and LLM-based tasks).
• External data sources as provided in the README for Sports, Beauty, and Toys.
• Any custom utilities (from utils.py, templates.py, seq.py) that assist in processing or model interfacing.