How textailes/hestia works
HESTIA is a multimodal data pipeline and archive portal. The principal workflow is: an authenticated client submits artifacts or structured readings to the Flask API; resource handlers persist binaries in MinIO and publish events, while Kafka consumers and Connect synchronize metadata into Postgres. The same API exposes reconstructions, scenes, imagery, jobs, and AmalthAI assets. A Directus extension provides authenticated archive browsing and management, with optional EGI SSO and ATON viewing. Some integrations are represented from README documentation because their implementation was not sampled.
Components 27
Ingestion API
- Flask API
api/api.py - API Key Security
api/middleware/security.py - Artifact Resource
api/resources/artifact.py - Sensor Resource
api/resources/sensor.py
Domain Resources
- Reconstruction Resource
api/resources/reconstruction.py - THOTH Scene Resource
api/resources/scene.py - NEFELE Jobs
api/resources/nefele_job.py - AmalthAI Assets
api/resources/amalthai_dataset.py - AmalthAI Inference
api/resources/amalthai_inference.py - Image Resources
api/resources/multispectral.py
Data Platform
- Database Service
api/services/database.py - Storage Service
api/services/storage.py - Messaging Service
api/services/messaging.py - MinIO Storage
- Postgres Database
- Kafka Events
Archive Portal
- Archive Extension
docker/directus_extensions/endpoints/archive/src/index.js - Archive Routes
docker/directus_extensions/endpoints/archive/src/routes - Archive Auth
docker/directus_extensions/endpoints/archive/src/utils/auth.js - Directus CMS
Processing Integrations
- Artifact Consumer
api/artifact_consumer.py - Kafka Connect
- Thumbnail Rendering
api/scripts/generate_thumbnail.py
Other components
- Data Client
- Portal User
- EGI Identity
- ATON Viewer
Connections 33
- Data Client → Flask API: submits data
- Flask API → Artifact Resource: dispatches uploads
- Flask API → Sensor Resource: dispatches readings
- Flask API → Reconstruction Resource: dispatches models
- Flask API → THOTH Scene Resource: dispatches scenes
- Flask API → NEFELE Jobs: dispatches jobs
- Flask API → AmalthAI Assets: dispatches assets
- Flask API → AmalthAI Inference: dispatches inference
- Flask API → Image Resources: dispatches images
- Artifact Resource → API Key Security: checks key
- Sensor Resource → API Key Security: checks key
- Artifact Resource → Storage Service: uploads binaries
- Reconstruction Resource → Storage Service: stores models
- THOTH Scene Resource → Storage Service: stores scenes
- AmalthAI Inference → Storage Service: stores images
- Artifact Resource → Messaging Service: publishes events
- Sensor Resource → Messaging Service: publishes readings
- AmalthAI Inference → Database Service: writes runs
- Storage Service → MinIO Storage: reads and writes
- Database Service → Postgres Database: opens connections
- Messaging Service → Kafka Events: streams events
- Kafka Events → Artifact Consumer: delivers uploads
- Kafka Events → Kafka Connect: feeds sink
- Kafka Connect → Postgres Database: writes metadata
- Reconstruction Resource → Thumbnail Rendering: requests thumbnails
- Portal User → Archive Extension: browses archive
- Archive Extension → Archive Routes: registers routes
- Archive Routes → Archive Auth: checks sessions
- Archive Routes → Directus CMS: queries content
- Archive Auth → Directus CMS: issues sessions
- Archive Auth → EGI Identity: uses SSO
- Archive Routes → ATON Viewer: opens 3D viewer
- Flask API → Data Client: returns results