# thammegowda/mtdata architecture

> Architecture diagram of the GitHub repository thammegowda/mtdata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thammegowda/mtdata
- Repository: https://github.com/thammegowda/mtdata

## Overview

##### Project Type & Purpose
- MTData is an open-source Python CLI tool and library for automating the download, extraction, parsing, and preparation of parallel machine-translation datasets. It provides:
- A command-line interface (`mtdata` / `python -m mtdata`)
- A Python API (`mtdata` package)
- An extensible index of datasets, extraction & parsing pipelines, ISO/BPC-47 language mapping, caching, and recipe management.

##### High-Level Architectural Components
1. User Entry Points
- CLI Frontend (`mtdata list`, `mtdata get`, `mtdata list-recipe`, `mtdata score`, `mtdata iso`, etc.)
- Python API (importable modules under `mtdata/`)
2. Core Controller
- `mtdata/main.py` and `__main__.py` parse arguments and dispatch to subcommands.
3. Dataset Index & Metadata
- `mtdata/index/` modules (one file per source, e.g., `opus_index.py`, `huggingface.py`)
- Exposes dataset IDs, URLs, versions, file types.
4. Recipe Manager
- `mtdata/recipe/recipes.yml` and recipe-loading logic
- Maps recipe IDs to sets of dataset IDs for train/dev/test.
5. Cache & Storage
- `mtdata/cache.py`: local cache directory (`~/.mtdata` or custom via $MTDATA)
- Manages downloaded archives, avoids re-download.
6. Downloader & Extractor
- HTTP/HTTPS fetch using `requests` or `aiohttp` + support for compressed formats (`.zip`, `.tar.gz`, `.tgz`)
- Integration with `pigz` for parallel decompression.
7. Parser & Converter
- File-type parsers under `mtdata/` (e.g., `tmx.py`, `sgm.py`, `tmx.py`, `tsv` parsing in `data.py`)
- Validates segment counts, handles language-pair ordering.
8. ISO & BCP-47 Module
- `mtdata/iso/` & top-level `iso-langs/` for language-code registry, mapping input codes to ISO-639-3 and BCP-47-style tags.
9. Crawler Submodule (Scrapy)
- `crawler/` (a git submodule): spiders (`OpusSpider.py`, `TildeSpider.py`), pipelines, middlewares.
- Used for dynamic data-site crawling when HTTP endpoint conventions change.
10. Utilities & Scripts
- `mtdata/utils.py`, `mtdata/scripts/`, recipe-stats, TSV-to-XLS converters.
11. External Services
- Dataset host servers (OPUS, ParaCrawl, EU sites, etc.)
- PyPI (for installation).
12. Tests & Documentation
- `tests/` directory (pytest suite)
- `docs/` folder with Asciidoctor HTML.

##### Component Interactions & Data Flow
- User invokes CLI → CLI parser selects subcommand → Core controller instantiates relevant module
- For `get` or `get-recipe`:
→ Query Index to resolve dataset IDs → Check Cache for existing archives
→ If missing: Downloader fetches archives from remote URLs
→ Extractor decompresses archive to temp dir
→ Parser reads extracted files (.tmx, .tsv, .sgm) → Creates parallel segment files
→ Output writer writes train/dev/test files, merges if requested, writes `mtdata.signature.txt`
→ Update Cache
- For `list`:
→ Controller queries Index modules → Compiles available dataset IDs and metadata
- For ISO lookups:
→ Controller calls `iso` or `bcp47` functions → Outputs standardized tags.
- For `score`:
→ Pipes source↔target through an external QE metric subprocess.

##### Architectural Patterns & Design Principles
- Plugin-style Index: each data source in its own module under `mtdata/index/`
- Separation of concerns: downloader vs. extractor vs. parser
- Cache-aside pattern for downloads
- Declarative recipes enabling reproducibility
- Scrapy crawler submodule follows pipeline architecture (items → pipelines → exports)
- Command dispatch uses Python’s `argparse` subcommands.

##### Technologies, Frameworks & Libraries
- Python 3.x
- Scrapy (crawler submodule)
- requests or aiohttp for HTTP
- pigz for fast gzip decompression
- PyYAML for recipes
- pytest for tests
- Asciidoctor for docs

##### Essential Elements
- Component names exactly as in code (e.g., `mtdata/index/opus/opus_index.py`)
- Subcommand entry points (`mtdata get`, `mtdata list`)
- Filesystem cache directory and output directory
- External HTTP dataset endpoints

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "CLI / API Layer"
        CLI["CLI Entry Points\n(mtdata/__main__.py, mtdata/entry.py, mtdata/main.py)"]:::entry
        API["Python API\n(mtdata/)"]:::entry
    end

    subgraph "Core Engine"
        CTRL["Controller\n(mtdata/main.py)"]:::core
        IDX["Dataset Index\n(mtdata/index/)"]:::core
        RECIPE["Recipe Manager\n(mtdata/resource/recipes.yml\nmtdata/recipe/)"]:::core
        CACHE["Cache & Storage\n(mtdata/cache.py)"]:::core
        DL["Downloader & Extractor\n(mtdata/data.py\nmtdata/pigz.py)"]:::core
        PARSE["Parser & Converter\n(mtdata/tmx.py\nmtdata/sgm.py\nmtdata/data.py)"]:::core
        OUTPUT["Output Writer\n(train/dev/test files)"]:::core
    end

    subgraph "ISO / BCP-47 Module"
        ISO["ISO/BPC-47\n(mtdata/iso/ , iso-langs/)"]:::plugin
    end

    subgraph "Crawler Submodule"
        CRAWL["Crawler\n(crawler/)"]:::plugin
    end

    subgraph "Utilities & Scripts"
        UT["Utils & Scripts\n(mtdata/utils.py\nmtdata/scripts/)"]:::plugin
    end

    subgraph "External Services"
        HTTP["Dataset Servers\n(OPUS, ParaCrawl, EU, etc.)"]:::external
        PYPI["PyPI"]:::external
        QE["QE Metric Subprocess"]:::external
    end

    CLI -->|"invokes"| CTRL
    API -->|"imports/uses"| CTRL

    CTRL -->|"resolves dataset IDs"| IDX
    CTRL -->|"queries recipes"| RECIPE
    CTRL -->|"checks cache"| CACHE
    CTRL -->|"lookup codes"| ISO
    CTRL -->|"crawl dynamic"| CRAWL
    CTRL -->|"run utilities"| UT
    CTRL -->|"score data"| QE

    IDX -->|"fetch metadata"| HTTP
    CTRL -->|"install via"| PYPI

    CACHE -->|"cached? skip download"| DL
    DL -->|"fetch URL"| HTTP
    DL -->|"decompress"| DL
    DL -->|"output files"| PARSE
    PARSE -->|"parse segments"| PARSE
    PARSE -->|"write output"| OUTPUT
    OUTPUT -->|"update cache"| CACHE

    CRAWL -->|"pipelines → store"| CACHE

    classDef entry fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef core fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef plugin fill:#FFFFE0,stroke:#000,stroke-width:1px;
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px;

    click CLI "https://github.com/thammegowda/mtdata/blob/main/mtdata/__main__.py"
    click CLI "https://github.com/thammegowda/mtdata/blob/main/mtdata/entry.py"
    click CTRL "https://github.com/thammegowda/mtdata/blob/main/mtdata/main.py"
    click IDX "https://github.com/thammegowda/mtdata/tree/main/mtdata/index/"
    click RECIPE "https://github.com/thammegowda/mtdata/blob/main/mtdata/resource/recipes.yml"
    click RECIPE "https://github.com/thammegowda/mtdata/blob/main/mtdata/recipe/__init__.py"
    click CACHE "https://github.com/thammegowda/mtdata/blob/main/mtdata/cache.py"
    click DL "https://github.com/thammegowda/mtdata/blob/main/mtdata/data.py"
    click DL "https://github.com/thammegowda/mtdata/blob/main/mtdata/pigz.py"
    click PARSE "https://github.com/thammegowda/mtdata/blob/main/mtdata/tmx.py"
    click PARSE "https://github.com/thammegowda/mtdata/blob/main/mtdata/sgm.py"
    click PARSE "https://github.com/thammegowda/mtdata/blob/main/mtdata/data.py"
    click ISO "https://github.com/thammegowda/mtdata/tree/main/mtdata/iso/"
    click ISO "https://github.com/thammegowda/mtdata/tree/main/iso-langs/"
    click CRAWL "https://github.com/thammegowda/mtdata/tree/main/crawler/"
    click UT "https://github.com/thammegowda/mtdata/blob/main/mtdata/utils.py"
    click UT "https://github.com/thammegowda/mtdata/blob/main/mtdata/scripts/recipe_stats.py"
    click UT "https://github.com/thammegowda/mtdata/blob/main/mtdata/scripts/tsv2xls.py"
```
