How ds-hub-sochi/animal_classification works
Main System Components
• CLI Layer
– irbis_classifier/cli/...
– Sub-modules: data_addition, preprocessing, reports, training, saving, synthetic_data_generation, testing
• Core Library
– irbis_classifier/src/...
– Sub-packages: models, training, testing, synthetic_data, utils
• Configuration Store
– configs/*.json
• Data Storage
– data/raw, data/interim, data/processed, data/external
• Models Output
– models/ (traced model artifacts)
• Reports Output
– reports/figures (generated plots)
• Tests
– tests/ (unit tests and assets)
Relationships and Data Flows
a. Data Ingestion & Augmentation
– Raw data (data/raw) → CLI/data_addition → interim
– Synthetic data generator can inject into data/raw or interim
b. Preprocessing Pipeline
– interim → filter_broken_images → filter_duplicates → sample_from_long_series → train/val/test split → match_splits_with_markup → processed
c. Reporting
– processed (and intermediate stages) → CLI/reports → reports/figures
d. Training
– processed/train.csv → CLI/training → Core training modules (datasets, trainer, experiment, losses, transforms) → saved model artifact
e. Saving/Exporting Model
– training output → CLI/saving → models/traced_model.pt
f. Testing/Evaluation
– processed/test.csv + traced_model.pt → CLI/testing → metrics, testers → CLI/reports or command-line output
Architectural Patterns & Design Principles
– Layered Architecture:
• Presentation/CLI layer on top
• Core domain modules (data processing, model training, evaluation)
• Infrastructure: file-system storage for data, models, reports
– Separation of Concerns:
• Each CLI command calls into a single-purpose library function
• Configurations are externalized into JSON files under configs/
– Extensibility:
• New CLI commands can be added under irbis_classifier/cli
• New model architectures via irbis_classifier/src/models/factory
Relevant Technologies & Frameworks
– Python 3.x
– PyTorch (models, trainer, tracing)
– pandas / numpy (data handling)
– matplotlib / seaborn (plots)
– pytest (unit tests)
– pre-commit, pylint, tox (linting & CI)