How bmikaberidze/micm-nlp works
micm-nlp is a YAML-driven NLP research framework. A user initializes a workspace and runs a unit config; the pipeline resolves configuration, tokenizes and preprocesses data, builds a model, trains/evaluates it, and returns run outputs. Group configs orchestrate multiple unit runs. The README also documents tokenizer training and PEFT/XPE capabilities, represented as distinct branches. Edges follow the pipeline excerpt, SOURCE INDEX references, or explicit README statements; unsampled internals are not inferred beyond those references.
Main entry points: pipeline.py Unit pipeline · runner.py Training runner · group.py Group orchestration · dataset.py Dataset preparation
Components 22
Entry and setup
- CLI
src/micm_nlp/cli.py - Workspace setup
src/micm_nlp/bootstrap.py
Run orchestration
- Unit configuration
src/micm_nlp/config.py - Unit pipeline
src/micm_nlp/pipeline.py - Group orchestration
src/micm_nlp/group.py - Plugin resolution
src/micm_nlp/plugins.py - Workspace paths
src/micm_nlp/path.py
Data and tokenization
- Tokenizer loading
src/micm_nlp/tokenizers/tokenizer.py - Tokenizer training
src/micm_nlp/tokenizers/tokenizer.py - Dataset preparation
src/micm_nlp/datasets/dataset.py
Models and training
- Model construction
src/micm_nlp/models/model.py - PEFT dispatch
src/micm_nlp/models/peft.py - Cross-Prompt Encoder
src/micm_nlp/models/xpe/encoder.py - XPE model factory
src/micm_nlp/models/xpe/factory.py - XPE encoder heads
src/micm_nlp/models/xpe/heads.py - XPE save/load
src/micm_nlp/models/xpe/save_load.py - Training runner
src/micm_nlp/training/runner.py
Evaluation and outputs
- Metrics
src/micm_nlp/evals/eval.py - Evaluation results
src/micm_nlp/evals/results.py - Run artifacts
src/micm_nlp/training/run_output.py
Other components
- Researcher
- Hugging Face Hub
Connections 24
- Researcher → CLI: starts run
README.md - Researcher → Workspace setup: initializes workspace
README.md - CLI → Unit pipeline: dispatches run
- Workspace setup → Workspace paths: sets workspace root
src/micm_nlp/path.py - Unit pipeline → Unit configuration: parses config
src/micm_nlp/pipeline.py - Unit pipeline → Tokenizer loading: loads tokenizer
src/micm_nlp/pipeline.py - Unit pipeline → Dataset preparation: constructs and preprocesses
src/micm_nlp/pipeline.py - Dataset preparation → Tokenizer loading: tokenizes data
src/micm_nlp/datasets/dataset.py - Unit pipeline → Model construction: builds model
src/micm_nlp/pipeline.py - Model construction → PEFT dispatch: configures adapters
src/micm_nlp/models/model.py - PEFT dispatch → XPE model factory: dispatches XPE
src/micm_nlp/models/peft.py - XPE model factory → Cross-Prompt Encoder: builds encoder
src/micm_nlp/models/xpe/factory.py - Cross-Prompt Encoder → XPE encoder heads: uses encoder heads
src/micm_nlp/models/xpe/encoder.py - XPE model factory → XPE save/load: loads adapter state
src/micm_nlp/models/xpe/factory.py - Unit pipeline → Training runner: runs training
src/micm_nlp/pipeline.py - Training runner → Metrics: computes metrics
src/micm_nlp/training/runner.py - Training runner → Evaluation results: records evaluations
src/micm_nlp/training/runner.py - Training runner → Run artifacts: writes run artifacts
src/micm_nlp/training/runner.py - Unit pipeline → Group orchestration: supports group runs
README.md - Group orchestration → Unit configuration: resolves unit configs
src/micm_nlp/group.py - Group orchestration → Run artifacts: collects run results
src/micm_nlp/group.py - Unit pipeline → Plugin resolution: supports custom classes
README.md - Tokenizer loading → Hugging Face Hub: loads pretrained assets
README.md - Dataset preparation → Hugging Face Hub: loads datasets
README.md