How microsoft/table-transformer works
Main Components
• Data Sources & Ingestion:
• External Data: PubTables-1M dataset, FinTabNet, SciTSR, etc. (downloaded from Microsoft Research Open Data or Hugging Face).
• Pre-processing Scripts: Located in the "scripts" directory (e.g., process_fintabnet.py, process_icdar2013.py) that prepare raw data.
• Data Pipeline & Configuration:
• Data configuration files in "src" (detection_config.json and structure_config.json) that define hyperparameters and operational settings.
• The environment management via environment.yml and conda setup.
• Model Components and Training:
• Training codes: “main.py” in both the “detr” and “src” directories (with flags to choose between “detection” and “structure” tasks) indicate the training pipeline.
• Supporting utilities: “detr/util” contains helper routines (e.g., box_ops, plot_utils).
• Inference and Evaluation:
• Inference Scripts: “src/inference.py” provides a simple pipeline for applying the trained model to input images.
• Evaluation Modules: “src/eval.py” and “test_all.py” in the detr folder which compute detection metrics (AP, AR) as well as GriTS metrics for table structure recognition.
• Debug/Visualization: The debug functionality in the training scripts (flags for debug visualizations) shows how inference outputs and intermediate results are logged.
• External Integrations/Dependencies:
• Pre-trained weights hosted on Hugging Face (links provided in the README).
• External libraries: PyTorch, Torchvision, and likely detectron2 (in the detr/d2 sub-component).
Relationships and Data Flow
• Data flows from external sources to pre-processing scripts, then becomes input for both training and evaluation modules.
• Training and inference modules depend on configuration files, which govern hyperparameters and dataset paths.
• Utilities and debug outputs support the training and evaluation loops.
• External sources (like Hugging Face for weights or Microsoft Research Open Data for the raw dataset) can be shown as separate blocks feeding into the system.
Architectural Patterns & Design Principles
• Modularity: Separate directories for models, datasets, engine (training/evaluation loops), and utilities keep the design modular.
• Config-driven: The use of JSON config files and environment setup shows dependency injection which allows different training scenarios.
• Pipeline/Flow Architecture: The separation between data ingestion, training, and inference indicates a pipeline-oriented design.
• Reuse of components: The architecture is designed to be extensible (e.g., different model weights, possibility of fine-tuning by loading checkpoints).
Relevant Technologies to Highlight
• Deep Learning Framework: PyTorch/Torchvision is at the core of training and inference.
• Docker & CI/CD: Presence of a Dockerfile (for environment consistency) and GitHub workflows (CI) in the “.github” directory.
• Configuration Management: JSON config files and environment.yml for reproducibility.