How wtglad/deep_sheep works
Project type and purpose
• This is a machine-learning classification system (“Deep Sheep”) for sorting sheep vs. non-sheep water-drinker images.
• It provides both training and inference functionality, with packaging for distribution (via PyPI) and a special trainer implementation targeting Google Cloud Platform (GCP).
File structure
Top‐level files:
• basic_test/… – local test assets or unit tests for quick iteration
• classifier.py – the inference interface (loads a trained model, runs prediction)
• deep_sheep.py – CLI entry point for end users
• model_trainer.py – local trainer abstraction for offline experimentation
• requirements.txt – Python dependencies
• readme.md – high-level overview and documentation pointer
ds-gcp package (for GCP training + distribution):
• setup.py – packaging config for PyPI
• dist/… – built sdist (deep_sheep-0.1.tar.gz)
• deep_sheep.egg-info/… – metadata for the package
• trainer/model_trainer_gcp.py – trainer implementation that spins up GCP resources (e.g., Cloud Storage, AI Platform).
README for additional context
• Confirms the goal: sort sheep images vs non-sheep.
• Points to a Google Doc for more detailed architecture/usage.
• Encourages community feedback.
Main components
a. Data Sources & Storage
• Raw image store (local folder or Google Cloud Storage bucket).
• Labeled dataset repository (e.g., CSV or TFRecords).
b. Data Preprocessing
• Scripts (inside model_trainer.py or GCP trainer) that resize/augment images.
c. Model Training
• Local Trainer (model_trainer.py) – runs on developer’s machine.
• GCP Trainer (trainer/model_trainer_gcp.py) – submits jobs to GCP AI Platform, reads/writes to Cloud Storage.
d. Model Artifact Storage
• Local FS (for dev) or GCP bucket (for production).
e. Packaging & Deployment
• setup.py + dist/… → PyPI package “deep_sheep”.
• Versioning and metadata (deep_sheep.egg-info).
f. Inference / Classification
• CLI entry point (deep_sheep.py) calls classifier.py
• classifier.py loads model artifact, processes input images, returns labels.
g. Optional Monitoring / Logging (implied for production GCP jobs).
Relationships and data flows
1) Data ingestion → preprocessing → model training
2) Training job writes model artifact → artifact stored in bucket
3) setup.py packages code → published to PyPI → installed by end users
4) CLI (deep_sheep.py) uses installed package → classifier loads model from local or remote storage → returns predictions
Architectural patterns and design principles
• Separation of concerns: training vs inference code in different modules.
• Modular packaging: all core logic lives under a distributable Python package (ds-gcp).
• Cloud‐native training: trainer abstraction lets you swap local vs GCP without touching core logic.
• CLI abstraction: single entry point (deep_sheep.py) for end users.
Detail level
• Under “GCP Trainer” indicate external services: “Cloud Storage”, “AI Platform Training Job”.
• Local developer environment separately from cloud infrastructure.