How thammegowda/mtdata works
Project Type & Purpose
• MTData is an open-source Python CLI tool and library for automating the download, extraction, parsing, and preparation of parallel machine-translation datasets. It provides:
• A command-line interface (mtdata / python -m mtdata)
• A Python API (mtdata package)
• An extensible index of datasets, extraction & parsing pipelines, ISO/BPC-47 language mapping, caching, and recipe management.
High-Level Architectural Components
1. User Entry Points
• CLI Frontend (mtdata list, mtdata get, mtdata list-recipe, mtdata score, mtdata iso, etc.)
• Python API (importable modules under mtdata/)
2. Core Controller
• mtdata/main.py and __main__.py parse arguments and dispatch to subcommands.
3. Dataset Index & Metadata
• mtdata/index/ modules (one file per source, e.g., opus_index.py, huggingface.py)
• Exposes dataset IDs, URLs, versions, file types.
4. Recipe Manager
• mtdata/recipe/recipes.yml and recipe-loading logic
• Maps recipe IDs to sets of dataset IDs for train/dev/test.
5. Cache & Storage
• mtdata/cache.py: local cache directory (~/.mtdata or custom via $MTDATA)
• Manages downloaded archives, avoids re-download.
6. Downloader & Extractor
• HTTP/HTTPS fetch using requests or aiohttp + support for compressed formats (.zip, .tar.gz, .tgz)
• Integration with pigz for parallel decompression.
7. Parser & Converter
• File-type parsers under mtdata/ (e.g., tmx.py, sgm.py, tmx.py, tsv parsing in data.py)
• Validates segment counts, handles language-pair ordering.
8. ISO & BCP-47 Module
• mtdata/iso/ & top-level iso-langs/ for language-code registry, mapping input codes to ISO-639-3 and BCP-47-style tags.
9. Crawler Submodule (Scrapy)
• crawler/ (a git submodule): spiders (OpusSpider.py, TildeSpider.py), pipelines, middlewares.
• Used for dynamic data-site crawling when HTTP endpoint conventions change.
10. Utilities & Scripts
• mtdata/utils.py, mtdata/scripts/, recipe-stats, TSV-to-XLS converters.
11. External Services
• Dataset host servers (OPUS, ParaCrawl, EU sites, etc.)
• PyPI (for installation).
12. Tests & Documentation
• tests/ directory (pytest suite)
• docs/ folder with Asciidoctor HTML.
Component Interactions & Data Flow
• User invokes CLI → CLI parser selects subcommand → Core controller instantiates relevant module
• For get or get-recipe:
→ Query Index to resolve dataset IDs → Check Cache for existing archives
→ If missing: Downloader fetches archives from remote URLs
→ Extractor decompresses archive to temp dir
→ Parser reads extracted files (.tmx, .tsv, .sgm) → Creates parallel segment files
→ Output writer writes train/dev/test files, merges if requested, writes mtdata.signature.txt
→ Update Cache
• For list:
→ Controller queries Index modules → Compiles available dataset IDs and metadata
• For ISO lookups:
→ Controller calls iso or bcp47 functions → Outputs standardized tags.
• For score:
→ Pipes source↔target through an external QE metric subprocess.
Architectural Patterns & Design Principles
• Plugin-style Index: each data source in its own module under mtdata/index/
• Separation of concerns: downloader vs. extractor vs. parser
• Cache-aside pattern for downloads
• Declarative recipes enabling reproducibility
• Scrapy crawler submodule follows pipeline architecture (items → pipelines → exports)
• Command dispatch uses Python’s argparse subcommands.
Technologies, Frameworks & Libraries
• Python 3.x
• Scrapy (crawler submodule)
• requests or aiohttp for HTTP
• pigz for fast gzip decompression
• PyYAML for recipes
• pytest for tests
• Asciidoctor for docs
Essential Elements
• Component names exactly as in code (e.g., mtdata/index/opus/opus_index.py)
• Subcommand entry points (mtdata get, mtdata list)
• Filesystem cache directory and output directory
• External HTTP dataset endpoints