How ldebele/end-to-end-airbus-ship-detection works
Airbus-Ship-Detection trains a U-Net ship-segmentation model and serves predictions through a separate event-driven inference path. In training, Airflow orchestrates dataset download, preprocessing into TFRecords, model fitting, and evaluation. In inference, a producer fetches satellite imagery and publishes it to Kafka; a consumer forwards images to the prediction API. The source excerpts show these stages and key calls; the relationship between saved training models and API loading is documented only at a high level, so it is dashed.
Main entry points: training_pipelines.py Airflow DAG · train.py Model training · preprocess.py Image preprocessing · evaluate.py Model evaluation
Components 18
Training orchestration
- Airflow DAG
src/pipelines/training_pipelines.py
Data preparation
- Dataset ingestion
src/components/data_ingestion - Image preprocessing
src/components/preprocessing/preprocess.py - Mask utilities
src/components/preprocessing/utils.py - TFRecord writer
src/components/preprocessing/save_tfrecord.py
Model lifecycle
- TFRecord reader
src/components/model_training/read_tfrecord.py - Model training
src/components/model_training/train.py - U-Net model
src/components/model_training/unet_model.py - Training utilities
src/components/model_training/utils.py - Model evaluation
src/components/model_evaluation/evaluate.py - MLflow tracker
Inference
- Image producer
src/inference/producer/producer.py - Kafka topic
- Image consumer
src/inference/consumer/consumer.py - Prediction API
src/inference/api/main.py
Other components
- Satellite image API
- Prediction requester
- Kaggle dataset
Connections 21
- Airflow DAG → Dataset ingestion: dispatches
src/pipelines/training_pipelines.py - Airflow DAG → Image preprocessing: dispatches
src/pipelines/training_pipelines.py - Airflow DAG → Model training: dispatches
src/pipelines/training_pipelines.py - Airflow DAG → Model evaluation: dispatches
src/pipelines/training_pipelines.py - Airflow DAG → Image preprocessing: orders after ingestion
src/pipelines/training_pipelines.py - Airflow DAG → Model training: orders after preprocessing
src/pipelines/training_pipelines.py - Airflow DAG → Model evaluation: orders after training
src/pipelines/training_pipelines.py - Kaggle dataset → Dataset ingestion: provides dataset
README.md - Image preprocessing → Mask utilities: uses
src/components/preprocessing/preprocess.py - Image preprocessing → TFRecord writer: writes records
src/components/preprocessing/preprocess.py - Model training → TFRecord reader: loads records
src/components/model_training/train.py - Model training → U-Net model: builds
src/components/model_training/train.py - Model training → Training utilities: uses
src/components/model_training/train.py - Model training → MLflow tracker: logs runs
src/components/model_training/train.py - Model evaluation → MLflow tracker: sets experiment
src/components/model_evaluation/evaluate.py - Satellite image API → Image producer: serves images
src/inference/producer/producer.py - Image producer → Kafka topic: publishes images
src/inference/producer/producer.py - Kafka topic → Image consumer: delivers images
src/inference/consumer/consumer.py - Image consumer → Prediction API: posts images
src/inference/consumer/consumer.py - Prediction requester → Prediction API: requests prediction
src/inference/api/main.py - Model training → Prediction API: model availability
README.md