# ldebele/end-to-end-airbus-ship-detection architecture

> Architecture diagram of the GitHub repository ldebele/end-to-end-airbus-ship-detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ldebele/end-to-end-airbus-ship-detection
- Repository: https://github.com/ldebele/end-to-end-airbus-ship-detection

## Overview

Airbus-Ship-Detection trains a U-Net ship-segmentation model and serves predictions through a separate event-driven inference path. In training, Airflow orchestrates dataset download, preprocessing into TFRecords, model fitting, and evaluation. In inference, a producer fetches satellite imagery and publishes it to Kafka; a consumer forwards images to the prediction API. The source excerpts show these stages and key calls; the relationship between saved training models and API loading is documented only at a high level, so it is dashed.

Main entry points: [`src/pipelines/training_pipelines.py`](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/pipelines/training_pipelines.py) (Airflow DAG), [`src/components/model_training/train.py`](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_training/train.py) (Model training), [`src/components/preprocessing/preprocess.py`](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/preprocessing/preprocess.py) (Image preprocessing), [`src/components/model_evaluation/evaluate.py`](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_evaluation/evaluate.py) (Model evaluation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training_orchestration["Training orchestration"]
  node_airflow["Airflow DAG"]
end

subgraph group_data_preparation["Data preparation"]
  node_data_ingestion["Dataset ingestion"]
  node_preprocessing["Image preprocessing<br/>[preprocess.py]"]
  node_mask_utils["Mask utilities<br/>[utils.py]"]
  node_tfrecord_writer["TFRecord writer<br/>[save_tfrecord.py]"]
end

subgraph group_model_lifecycle["Model lifecycle"]
  node_tfrecord_reader["TFRecord reader<br/>[read_tfrecord.py]"]
  node_training["Model training<br/>[train.py]"]
  node_unet["U-Net model<br/>[unet_model.py]"]
  node_training_utils["Training utilities<br/>[utils.py]"]
  node_evaluation["Model evaluation<br/>[evaluate.py]"]
  node_mlflow["MLflow tracker"]
end

subgraph group_inference["Inference"]
  node_producer["Image producer<br/>[producer.py]"]
  node_kafka["Kafka topic"]
  node_consumer["Image consumer<br/>[consumer.py]"]
  node_prediction_api["Prediction API<br/>[main.py]"]
end

node_satellite_api["Satellite image API"]
node_prediction_client(("Prediction requester"))
node_kaggle["Kaggle dataset"]

node_airflow -->|"dispatches"| node_data_ingestion
node_airflow -->|"dispatches"| node_preprocessing
node_airflow -->|"dispatches"| node_training
node_airflow -->|"dispatches"| node_evaluation
node_airflow -->|"orders after ingestion"| node_preprocessing
node_airflow -->|"orders after preprocessing"| node_training
node_airflow -->|"orders after training"| node_evaluation
node_kaggle -->|"provides dataset"| node_data_ingestion
node_preprocessing -->|"uses"| node_mask_utils
node_preprocessing -->|"writes records"| node_tfrecord_writer
node_training -->|"loads records"| node_tfrecord_reader
node_training -->|"builds"| node_unet
node_training -->|"uses"| node_training_utils
node_training -->|"logs runs"| node_mlflow
node_evaluation -->|"sets experiment"| node_mlflow
node_satellite_api -->|"serves images"| node_producer
node_producer -->|"publishes images"| node_kafka
node_kafka -->|"delivers images"| node_consumer
node_consumer -->|"posts images"| node_prediction_api
node_prediction_client -->|"requests prediction"| node_prediction_api
node_training -.->|"model availability"| node_prediction_api

click node_airflow "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/pipelines/training_pipelines.py"
click node_data_ingestion "https://github.com/ldebele/end-to-end-airbus-ship-detection/tree/main/src/components/data_ingestion"
click node_preprocessing "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/preprocessing/preprocess.py"
click node_mask_utils "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/preprocessing/utils.py"
click node_tfrecord_writer "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/preprocessing/save_tfrecord.py"
click node_tfrecord_reader "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_training/read_tfrecord.py"
click node_training "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_training/train.py"
click node_unet "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_training/unet_model.py"
click node_training_utils "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_training/utils.py"
click node_evaluation "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_evaluation/evaluate.py"
click node_producer "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/inference/producer/producer.py"
click node_consumer "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/inference/consumer/consumer.py"
click node_prediction_api "https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/inference/api/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow toneBlue
class node_data_ingestion,node_preprocessing,node_mask_utils,node_tfrecord_writer toneAmber
class node_tfrecord_reader,node_training,node_unet,node_training_utils,node_evaluation,node_mlflow,node_satellite_api,node_prediction_client toneMint
class node_producer,node_kafka,node_consumer,node_prediction_api toneRose
class node_kaggle toneIndigo
```

## Components

### Training orchestration

- [Airflow DAG](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/pipelines/training_pipelines.py): `src/pipelines/training_pipelines.py`

### Data preparation

- [Dataset ingestion](https://github.com/ldebele/end-to-end-airbus-ship-detection/tree/main/src/components/data_ingestion): `src/components/data_ingestion`
- [Image preprocessing](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/preprocessing/preprocess.py): `src/components/preprocessing/preprocess.py`
- [Mask utilities](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/preprocessing/utils.py): `src/components/preprocessing/utils.py`
- [TFRecord writer](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/preprocessing/save_tfrecord.py): `src/components/preprocessing/save_tfrecord.py`

### Model lifecycle

- [TFRecord reader](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_training/read_tfrecord.py): `src/components/model_training/read_tfrecord.py`
- [Model training](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_training/train.py): `src/components/model_training/train.py`
- [U-Net model](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_training/unet_model.py): `src/components/model_training/unet_model.py`
- [Training utilities](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_training/utils.py): `src/components/model_training/utils.py`
- [Model evaluation](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/components/model_evaluation/evaluate.py): `src/components/model_evaluation/evaluate.py`
- **MLflow tracker**

### Inference

- [Image producer](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/inference/producer/producer.py): `src/inference/producer/producer.py`
- **Kafka topic**
- [Image consumer](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/inference/consumer/consumer.py): `src/inference/consumer/consumer.py`
- [Prediction API](https://github.com/ldebele/end-to-end-airbus-ship-detection/blob/main/src/inference/api/main.py): `src/inference/api/main.py`

### Other components

- **Satellite image API**
- **Prediction requester**
- **Kaggle dataset**

## Connections

- Airflow DAG → Dataset ingestion: dispatches (evidence: `src/pipelines/training_pipelines.py`)
- Airflow DAG → Image preprocessing: dispatches (evidence: `src/pipelines/training_pipelines.py`)
- Airflow DAG → Model training: dispatches (evidence: `src/pipelines/training_pipelines.py`)
- Airflow DAG → Model evaluation: dispatches (evidence: `src/pipelines/training_pipelines.py`)
- Airflow DAG → Image preprocessing: orders after ingestion (evidence: `src/pipelines/training_pipelines.py`)
- Airflow DAG → Model training: orders after preprocessing (evidence: `src/pipelines/training_pipelines.py`)
- Airflow DAG → Model evaluation: orders after training (evidence: `src/pipelines/training_pipelines.py`)
- Kaggle dataset → Dataset ingestion: provides dataset (evidence: `README.md`)
- Image preprocessing → Mask utilities: uses (evidence: `src/components/preprocessing/preprocess.py`)
- Image preprocessing → TFRecord writer: writes records (evidence: `src/components/preprocessing/preprocess.py`)
- Model training → TFRecord reader: loads records (evidence: `src/components/model_training/train.py`)
- Model training → U-Net model: builds (evidence: `src/components/model_training/train.py`)
- Model training → Training utilities: uses (evidence: `src/components/model_training/train.py`)
- Model training → MLflow tracker: logs runs (evidence: `src/components/model_training/train.py`)
- Model evaluation → MLflow tracker: sets experiment (evidence: `src/components/model_evaluation/evaluate.py`)
- Satellite image API → Image producer: serves images (evidence: `src/inference/producer/producer.py`)
- Image producer → Kafka topic: publishes images (evidence: `src/inference/producer/producer.py`)
- Kafka topic → Image consumer: delivers images (evidence: `src/inference/consumer/consumer.py`)
- Image consumer → Prediction API: posts images (evidence: `src/inference/consumer/consumer.py`)
- Prediction requester → Prediction API: requests prediction (evidence: `src/inference/api/main.py`)
- Model training → Prediction API: model availability (evidence: `README.md`)
