How velocitatem/dapper works
Main Components
• Data Processing Pipeline: Represent the components responsible for loading, processing (resizing images, splitting datasets), and uploading images to MinIO. Key files include those in classification/data, configuration files (e.g., lsnet.yaml, settings.yaml), and dataset loaders.
• Model Training Pipeline: Highlight the training process using the custom MinioImageDataset, the training scripts (train.py), model definitions (in training/models.py, training/resnet.py), and configuration files under classification/training/configs.
• Extraction Module: Although not highlighted in the README, the existence of the extraction/ directory with rule_based_extractor.py suggests a module that might be used for additional document extraction tasks.
• Dashboard and External Interfaces: The long list of scripts like dashboard.py, and potential interaction points from external services (via docker-compose and configuration settings), should be clearly drawn.
Analyzing the File Structure for Architectural Clues
• Folder “classification”: This is the core of the document classification workflow. Inside, the separation into subdirectories for data, training, datasets, and utils indicates a modular design.
• Subdirectories “training” and “datasets”: These indicate clear separation between dataset management and the actual model training routines.
• Configurations: Multiple YAML files under training/configs and classification/config suggest that the system is highly configurable with parameters for model and data handling.
• Extraction: The extraction subpackage (with both a base extractor and a rule-based extractor) implies an additional function layer that might process or post-process the document outputs.
• GitHub workflows and docker-compose: These files indicate CI/CD and deployment practices that can be depicted externally for integration with infrastructure.
a. Main Containers/Components:
• Frontend/Interface Layer (if applicable): Although there is a dashboard script, it might represent a lightweight UI or API.
• Backend/Processing Layer: This includes the “classification” component that is responsible for data processing and training.
▫ Data Processing Subsystem: Represent the scripts for loading and processing image data, including the augmentation and resizing operations.
▫ Model Training Subsystem: Include components for model selection (ResNet18/transfer learning), training, evaluation, and model persistence (saving the model.pth).
b. Relationships and Data Flow:
• Optionally, show external datasets and sources (RVL-CDIP, invoices, Kaggle) as inputs into the data processing component.
c. Architectural Patterns/Principles:
• Modular Design: The clear separation between data handling, training, and extraction modules.
• Pipeline Architecture: A sequential workflow moving from data ingestion and processing to model training and evaluation.
• External Integration: The use of external storage (MinIO) and external dataset sources should be visibly integrated.
d. Technologies and Frameworks:
• Mention Python as the primary programming language, MinIO for object storage, and the use of machine learning frameworks (implied from the training scripts). Also note Docker (via docker-compose) for orchestration.