How potamides/datikz works
Project Type and Purpose
• DaTi*k*Z is an open-source Python CLI tool that crawls, extracts, and compiles Ti*k*Z drawings from multiple data sources, then packages them into train/test Parquet datasets. It is intended for researchers who need large collections of LaTeX vector‐graphic examples.
High-Level Architecture Components
a. CLI / Entry Point
• main.py: Parses command-line arguments (e.g., --arxiv_files, --size) and invokes the dataset generation pipeline.
b. Core Library
• datikz/datikz.py: Implements the DaTiKZ class or functions that orchestrate loading, parsing, and exporting.
c. Loader Plugins
• datikz/loaders/*.py and subdirectories: Each loader module (arxiv, chatgpt, gpt4, tex_stackexchange_com, tikz_favorites, etc.) implements a common interface to fetch raw Ti*k*Z snippets and metadata from one source.
d. External Dependencies & Services
• TeX Live: Compiles LaTeX sources to PDF.
• ghostscript & poppler: Rasterizes or parses PDFs for image previews and further processing.
• Data Sources: arXiv JSONL files, web scraping endpoints (texample.net, TikZ.net, PGF manual), OpenAI GPT APIs for “chatgpt” and “gpt4” loaders.
e. Data Pipeline
• Fetching: Loader plugins retrieve raw data.
• Parsing & Filtering: Clean up, de-macro, or transform Ti*k*Z code (e.g., demacro.py).
• Packaging: Collect processed records and write out datikz-train.parquet and datikz-test.parquet.
Relationships & Data Flow
• main.py → invokes DaTiKZ core
• DaTiKZ core → dynamically loads each loader plugin
• Loader plugins → fetch data from external sources (websites, JSONL, APIs)
• Fetched Ti*k*Z code → passed back to core for parsing and external tool invocation
• Parsed & rendered records → aggregated by core → exported to Parquet files
Architectural Patterns & Principles
• Plugin Pattern: The datikz/loaders directory defines a uniform interface; new sources can be added by dropping in a new module.
• Separation of Concerns:
• CLI parsing (main.py)
• Orchestration & business logic (datikz/datikz.py)
• Source-specific fetching and normalization (loaders/)
• External tool integration is kept at the periphery (TeX Live, ghostscript, poppler).
• Dependency Injection / Dynamic Discovery: Core loads loader classes by module introspection rather than hard-coding each source.
Detail Level
• Plugin interface pattern—show an interface or abstract loader class that each loader implements.
• That new loaders can be dropped into datikz/loaders/ without modifying the core.