# potamides/datikz architecture

> Architecture diagram of the GitHub repository potamides/datikz, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/potamides/datikz
- Repository: https://github.com/potamides/datikz

## Overview

##### Project Type and Purpose
- DaTi*k*Z is an open-source Python CLI tool that crawls, extracts, and compiles Ti*k*Z drawings from multiple data sources, then packages them into train/test Parquet datasets. It is intended for researchers who need large collections of LaTeX vector‐graphic examples.

##### High-Level Architecture Components
a. CLI / Entry Point
- main.py: Parses command-line arguments (e.g., --arxiv_files, --size) and invokes the dataset generation pipeline.
b. Core Library
- datikz/datikz.py: Implements the DaTiKZ class or functions that orchestrate loading, parsing, and exporting.
c. Loader Plugins
- datikz/loaders/*.py and subdirectories: Each loader module (arxiv, chatgpt, gpt4, tex_stackexchange_com, tikz_favorites, etc.) implements a common interface to fetch raw Ti*k*Z snippets and metadata from one source.
d. External Dependencies & Services
- TeX Live: Compiles LaTeX sources to PDF.
- ghostscript & poppler: Rasterizes or parses PDFs for image previews and further processing.
- Data Sources: arXiv JSONL files, web scraping endpoints (texample.net, TikZ.net, PGF manual), OpenAI GPT APIs for “chatgpt” and “gpt4” loaders.
e. Data Pipeline
- Fetching: Loader plugins retrieve raw data.
- Parsing & Filtering: Clean up, de-macro, or transform Ti*k*Z code (e.g., demacro.py).
- Packaging: Collect processed records and write out `datikz-train.parquet` and `datikz-test.parquet`.

##### Relationships & Data Flow
- main.py → invokes DaTiKZ core
- DaTiKZ core → dynamically loads each loader plugin
- Loader plugins → fetch data from external sources (websites, JSONL, APIs)
- Fetched Ti*k*Z code → passed back to core for parsing and external tool invocation
- Parsed & rendered records → aggregated by core → exported to Parquet files

##### Architectural Patterns & Principles
- Plugin Pattern: The `datikz/loaders` directory defines a uniform interface; new sources can be added by dropping in a new module.
- Separation of Concerns:
- CLI parsing (main.py)
- Orchestration & business logic (datikz/datikz.py)
- Source-specific fetching and normalization (loaders/)
- External tool integration is kept at the periphery (TeX Live, ghostscript, poppler).
- Dependency Injection / Dynamic Discovery: Core loads loader classes by module introspection rather than hard-coding each source.

##### Detail Level
- Plugin interface pattern—show an interface or abstract loader class that each loader implements.
- That new loaders can be dropped into `datikz/loaders/` without modifying the core.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI and Core
    subgraph "Core Layer"
        CLI["CLI (main.py)"]:::internal
        Core["DaTiKZ Core (datikz/datikz.py)"]:::internal
    end

    %% Loader Plugins
    subgraph "Loader Plugins" 
        arxiv_finder["arXiv Finder (finder.py)"]:::plugin
        arxiv_demacro["arXiv Demacro (demacro.py)"]:::plugin
        chatgpt["ChatGPT Loader (chatgpt.py)"]:::plugin
        gpt4_init["GPT-4 Loader (__init__.py)"]:::plugin
        gpt4_json["GPT-4 Config (gpt4.json)"]:::plugin
        tex_se["TeX SE Loader (tex_stackexchange_com.py)"]:::plugin
        tikz_fav["TikZ Favorites Loader (tikz_favorites.py)"]:::plugin
        tikz_net["TikZ.net Loader (tikz_net.py)"]:::plugin
        texample["TEXample.net Loader (texample_net.py)"]:::plugin
        pgfmanual["PGF Manual Loader (pgfmanual.py)"]:::plugin
        latex_examples["LaTeX Examples Loader (latex_examples.py)"]:::plugin
        janosh["Janosh TikZ Loader (janosh_tikz.py)"]:::plugin
        petarv["PetarV TikZ Loader (petarv_tikz.py)"]:::plugin
        openaiwatch["OpenAI Watcher Loader (openaiwatch.py)"]:::plugin
    end

    %% External Systems
    subgraph "External Systems"
        arxiv_jsonl["arXiv JSONL"]:::external
        web_apis["Web Sites & APIs"]:::external
        tex_toolchain["TeX Live & PDF Toolchain"]:::external
    end

    %% Output
    output["Parquet Dataset (train/test)"]:::output

    %% Connections
    CLI -->|"start & parse args"| Core
    Core -->|"fetch Ti*k*Z snippets"| arxiv_finder
    Core -->|"fetch Ti*k*Z snippets"| arxiv_demacro
    Core -->|"fetch Ti*k*Z snippets"| chatgpt
    Core -->|"fetch Ti*k*Z snippets"| gpt4_init
    Core -->|"fetch Ti*k*Z snippets"| tex_se
    Core -->|"fetch Ti*k*Z snippets"| tikz_fav
    Core -->|"fetch Ti*k*Z snippets"| tikz_net
    Core -->|"fetch Ti*k*Z snippets"| texample
    Core -->|"fetch Ti*k*Z snippets"| pgfmanual
    Core -->|"fetch Ti*k*Z snippets"| latex_examples
    Core -->|"fetch Ti*k*Z snippets"| janosh
    Core -->|"fetch Ti*k*Z snippets"| petarv
    Core -->|"fetch Ti*k*Z snippets"| openaiwatch

    arxiv_finder -->|"data"| arxiv_jsonl
    arxiv_demacro -->|"data"| arxiv_jsonl
    chatgpt -->|"API calls"| web_apis
    gpt4_init -->|"API calls"| web_apis
    tex_se -->|"scraping"| web_apis
    tikz_fav -->|"scraping"| web_apis
    tikz_net -->|"scraping"| web_apis
    texample -->|"scraping"| web_apis
    pgfmanual -->|"scraping"| web_apis
    latex_examples -->|"scraping"| web_apis
    janosh -->|"scraping"| web_apis
    petarv -->|"scraping"| web_apis
    openaiwatch -->|"monitoring"| web_apis

    Core -->|"compile & render"| tex_toolchain
    Core -->|"write parquet"| output

    %% Click Events
    click CLI "https://github.com/potamides/datikz/blob/main/main.py"
    click Core "https://github.com/potamides/datikz/blob/main/datikz/datikz.py"
    click arxiv_finder "https://github.com/potamides/datikz/blob/main/datikz/loaders/arxiv/finder.py"
    click arxiv_demacro "https://github.com/potamides/datikz/blob/main/datikz/loaders/arxiv/demacro.py"
    click chatgpt "https://github.com/potamides/datikz/blob/main/datikz/loaders/chatgpt.py"
    click gpt4_init "https://github.com/potamides/datikz/blob/main/datikz/loaders/gpt4/__init__.py"
    click gpt4_json "https://github.com/potamides/datikz/blob/main/datikz/loaders/gpt4/gpt4.json"
    click tex_se "https://github.com/potamides/datikz/blob/main/datikz/loaders/tex_stackexchange_com.py"
    click tikz_fav "https://github.com/potamides/datikz/blob/main/datikz/loaders/tikz_favorites.py"
    click tikz_net "https://github.com/potamides/datikz/blob/main/datikz/loaders/tikz_net.py"
    click texample "https://github.com/potamides/datikz/blob/main/datikz/loaders/texample_net.py"
    click pgfmanual "https://github.com/potamides/datikz/blob/main/datikz/loaders/pgfmanual.py"
    click latex_examples "https://github.com/potamides/datikz/blob/main/datikz/loaders/latex_examples.py"
    click janosh "https://github.com/potamides/datikz/blob/main/datikz/loaders/janosh_tikz.py"
    click petarv "https://github.com/potamides/datikz/blob/main/datikz/loaders/petarv_tikz.py"
    click openaiwatch "https://github.com/potamides/datikz/blob/main/datikz/loaders/openaiwatch.py"

    %% Styles
    classDef internal fill:#cce5ff,stroke:#004085,color:#004085;
    classDef plugin fill:#d4edda,stroke:#155724,color:#155724;
    classDef external fill:#fff3cd,stroke:#856404,color:#856404;
    classDef output fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
```
