# bmikaberidze/micm-nlp architecture

> Architecture diagram of the GitHub repository bmikaberidze/micm-nlp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bmikaberidze/micm-nlp
- Repository: https://github.com/bmikaberidze/micm-nlp

## Overview

micm-nlp is a YAML-driven NLP research framework. A user initializes a workspace and runs a unit config; the pipeline resolves configuration, tokenizes and preprocesses data, builds a model, trains/evaluates it, and returns run outputs. Group configs orchestrate multiple unit runs. The README also documents tokenizer training and PEFT/XPE capabilities, represented as distinct branches. Edges follow the pipeline excerpt, SOURCE INDEX references, or explicit README statements; unsampled internals are not inferred beyond those references.

Main entry points: [`src/micm_nlp/pipeline.py`](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/pipeline.py) (Unit pipeline), [`src/micm_nlp/training/runner.py`](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/training/runner.py) (Training runner), [`src/micm_nlp/group.py`](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/group.py) (Group orchestration), [`src/micm_nlp/datasets/dataset.py`](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/datasets/dataset.py) (Dataset preparation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry and setup"]
  node_cli["CLI<br/>[cli.py]"]
  node_bootstrap["Workspace setup<br/>[bootstrap.py]"]
end

subgraph group_workflow["Run orchestration"]
  node_config["Unit configuration<br/>[config.py]"]
  node_pipeline["Unit pipeline<br/>[pipeline.py]"]
  node_group["Group orchestration<br/>[group.py]"]
  node_plugins["Plugin resolution<br/>[plugins.py]"]
  node_paths["Workspace paths<br/>[path.py]"]
end

subgraph group_data["Data and tokenization"]
  node_tokenizer["Tokenizer loading<br/>[tokenizer.py]"]
  node_tokentrain["Tokenizer training<br/>[tokenizer.py]"]
  node_dataset["Dataset preparation<br/>[dataset.py]"]
end

subgraph group_learning["Models and training"]
  node_model["Model construction<br/>[model.py]"]
  node_peft["PEFT dispatch<br/>[peft.py]"]
  node_xpe["Cross-Prompt Encoder<br/>[encoder.py]"]
  node_xpefactory["XPE model factory<br/>[factory.py]"]
  node_xpeheads["XPE encoder heads<br/>[heads.py]"]
  node_xpesave["XPE save/load<br/>[save_load.py]"]
  node_trainer["Training runner<br/>[runner.py]"]
end

subgraph group_results["Evaluation and outputs"]
  node_metrics["Metrics<br/>[eval.py]"]
  node_evalresults["Evaluation results<br/>[results.py]"]
  node_runoutput["Run artifacts<br/>[run_output.py]"]
end

node_researcher(("Researcher"))
node_hub["Hugging Face Hub"]

node_researcher -->|"starts run"| node_cli
node_researcher -->|"initializes workspace"| node_bootstrap
node_cli -.->|"dispatches run"| node_pipeline
node_bootstrap -->|"sets workspace root"| node_paths
node_pipeline -->|"parses config"| node_config
node_pipeline -->|"loads tokenizer"| node_tokenizer
node_pipeline -->|"constructs and preprocesses"| node_dataset
node_dataset -->|"tokenizes data"| node_tokenizer
node_pipeline -->|"builds model"| node_model
node_model -->|"configures adapters"| node_peft
node_peft -->|"dispatches XPE"| node_xpefactory
node_xpefactory -->|"builds encoder"| node_xpe
node_xpe -->|"uses encoder heads"| node_xpeheads
node_xpefactory -->|"loads adapter state"| node_xpesave
node_pipeline -->|"runs training"| node_trainer
node_trainer -->|"computes metrics"| node_metrics
node_trainer -->|"records evaluations"| node_evalresults
node_trainer -->|"writes run artifacts"| node_runoutput
node_pipeline -->|"supports group runs"| node_group
node_group -->|"resolves unit configs"| node_config
node_group -->|"collects run results"| node_runoutput
node_pipeline -->|"supports custom classes"| node_plugins
node_tokenizer -.->|"loads pretrained assets"| node_hub
node_dataset -.->|"loads datasets"| node_hub

click node_cli "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/cli.py"
click node_bootstrap "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/bootstrap.py"
click node_config "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/config.py"
click node_pipeline "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/pipeline.py"
click node_group "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/group.py"
click node_plugins "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/plugins.py"
click node_paths "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/path.py"
click node_tokenizer "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/tokenizers/tokenizer.py"
click node_tokentrain "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/tokenizers/tokenizer.py"
click node_dataset "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/datasets/dataset.py"
click node_model "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/model.py"
click node_peft "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/peft.py"
click node_xpe "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/xpe/encoder.py"
click node_xpefactory "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/xpe/factory.py"
click node_xpeheads "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/xpe/heads.py"
click node_xpesave "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/xpe/save_load.py"
click node_trainer "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/training/runner.py"
click node_metrics "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/evals/eval.py"
click node_evalresults "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/evals/results.py"
click node_runoutput "https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/training/run_output.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_bootstrap toneBlue
class node_config,node_pipeline,node_group,node_plugins,node_paths toneAmber
class node_tokenizer,node_tokentrain,node_dataset toneMint
class node_model,node_peft,node_xpe,node_xpefactory,node_xpeheads,node_xpesave,node_trainer toneRose
class node_metrics,node_evalresults,node_runoutput,node_researcher,node_hub toneIndigo
```

## Components

### Entry and setup

- [CLI](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/cli.py): `src/micm_nlp/cli.py`
- [Workspace setup](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/bootstrap.py): `src/micm_nlp/bootstrap.py`

### Run orchestration

- [Unit configuration](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/config.py): `src/micm_nlp/config.py`
- [Unit pipeline](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/pipeline.py): `src/micm_nlp/pipeline.py`
- [Group orchestration](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/group.py): `src/micm_nlp/group.py`
- [Plugin resolution](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/plugins.py): `src/micm_nlp/plugins.py`
- [Workspace paths](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/path.py): `src/micm_nlp/path.py`

### Data and tokenization

- [Tokenizer loading](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/tokenizers/tokenizer.py): `src/micm_nlp/tokenizers/tokenizer.py`
- [Tokenizer training](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/tokenizers/tokenizer.py): `src/micm_nlp/tokenizers/tokenizer.py`
- [Dataset preparation](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/datasets/dataset.py): `src/micm_nlp/datasets/dataset.py`

### Models and training

- [Model construction](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/model.py): `src/micm_nlp/models/model.py`
- [PEFT dispatch](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/peft.py): `src/micm_nlp/models/peft.py`
- [Cross-Prompt Encoder](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/xpe/encoder.py): `src/micm_nlp/models/xpe/encoder.py`
- [XPE model factory](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/xpe/factory.py): `src/micm_nlp/models/xpe/factory.py`
- [XPE encoder heads](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/xpe/heads.py): `src/micm_nlp/models/xpe/heads.py`
- [XPE save/load](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/models/xpe/save_load.py): `src/micm_nlp/models/xpe/save_load.py`
- [Training runner](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/training/runner.py): `src/micm_nlp/training/runner.py`

### Evaluation and outputs

- [Metrics](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/evals/eval.py): `src/micm_nlp/evals/eval.py`
- [Evaluation results](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/evals/results.py): `src/micm_nlp/evals/results.py`
- [Run artifacts](https://github.com/bmikaberidze/micm-nlp/blob/main/src/micm_nlp/training/run_output.py): `src/micm_nlp/training/run_output.py`

### Other components

- **Researcher**
- **Hugging Face Hub**

## Connections

- Researcher → CLI: starts run (evidence: `README.md`)
- Researcher → Workspace setup: initializes workspace (evidence: `README.md`)
- CLI → Unit pipeline: dispatches run
- Workspace setup → Workspace paths: sets workspace root (evidence: `src/micm_nlp/path.py`)
- Unit pipeline → Unit configuration: parses config (evidence: `src/micm_nlp/pipeline.py`)
- Unit pipeline → Tokenizer loading: loads tokenizer (evidence: `src/micm_nlp/pipeline.py`)
- Unit pipeline → Dataset preparation: constructs and preprocesses (evidence: `src/micm_nlp/pipeline.py`)
- Dataset preparation → Tokenizer loading: tokenizes data (evidence: `src/micm_nlp/datasets/dataset.py`)
- Unit pipeline → Model construction: builds model (evidence: `src/micm_nlp/pipeline.py`)
- Model construction → PEFT dispatch: configures adapters (evidence: `src/micm_nlp/models/model.py`)
- PEFT dispatch → XPE model factory: dispatches XPE (evidence: `src/micm_nlp/models/peft.py`)
- XPE model factory → Cross-Prompt Encoder: builds encoder (evidence: `src/micm_nlp/models/xpe/factory.py`)
- Cross-Prompt Encoder → XPE encoder heads: uses encoder heads (evidence: `src/micm_nlp/models/xpe/encoder.py`)
- XPE model factory → XPE save/load: loads adapter state (evidence: `src/micm_nlp/models/xpe/factory.py`)
- Unit pipeline → Training runner: runs training (evidence: `src/micm_nlp/pipeline.py`)
- Training runner → Metrics: computes metrics (evidence: `src/micm_nlp/training/runner.py`)
- Training runner → Evaluation results: records evaluations (evidence: `src/micm_nlp/training/runner.py`)
- Training runner → Run artifacts: writes run artifacts (evidence: `src/micm_nlp/training/runner.py`)
- Unit pipeline → Group orchestration: supports group runs (evidence: `README.md`)
- Group orchestration → Unit configuration: resolves unit configs (evidence: `src/micm_nlp/group.py`)
- Group orchestration → Run artifacts: collects run results (evidence: `src/micm_nlp/group.py`)
- Unit pipeline → Plugin resolution: supports custom classes (evidence: `README.md`)
- Tokenizer loading → Hugging Face Hub: loads pretrained assets (evidence: `README.md`)
- Dataset preparation → Hugging Face Hub: loads datasets (evidence: `README.md`)
