How sematic-ai/sematic works
Sematic turns Python functions into typed, traceable pipeline graphs, executes them locally or across cloud/Kubernetes resources, persists run state and artifacts, and exposes monitoring and lineage through a web dashboard. The main path is authoring, graph construction, execution/resolution, persistence, then inspection or rerun. The README documents caching, retries, dynamic graphs, resource plugins, and integrations; those branches are represented at subsystem level where their internal wiring was not sampled. Exact paths come from the tree; implementation detail for unsampled areas is limited.
Main entry points: resolver.py Step resolver · runner.py Runner entry · app.py HTTP API · api_client.py API client
Components 24
Pipeline authoring
- Python SDK
sematic/function.py - Pipeline graph
sematic/graph.py - Types and serialization
sematic/types/serialization.py
Pipeline execution
- Runner entry
sematic/runner.py - Local runner
sematic/runners/local_runner.py - Cloud runner
sematic/runners/cloud_runner.py - Step resolver
sematic/resolver.py - Job scheduler
sematic/scheduling/job_scheduler.py - Kubernetes
sematic/scheduling/kubernetes.py - Compute plugins
sematic/plugins/abstract_external_resource.py - Step caching
sematic/caching/caching.py
State and artifacts
- Platform database
sematic/db/db.py - Artifact storage
sematic/plugins/abstract_storage.py - S3 backend
sematic/plugins/storage/s3_storage.py - Metrics storage
sematic/plugins/metrics_storage/sql/sql_metrics_storage.py
Platform services
- CLI
sematic/cli/main.py - HTTP API
sematic/api/app.py - API client
sematic/api_client.py - Git metadata
sematic/caching/git.py
User experience
- Web dashboard
sematic/ui/packages/common/src/pages/RunDetails/index.tsx - Pipeline views
sematic/ui/packages/common/src/pages/PipelineList/PipelineList.tsx - Run inspection
sematic/ui/packages/common/src/pages/RunDetails/RunRouter.tsx
Other components
- ML engineer
- Kubernetes cluster
Connections 27
- ML engineer → Python SDK: defines pipeline
- Python SDK → Pipeline graph: builds graph
- Python SDK → Types and serialization: checks and serializes
- ML engineer → CLI: starts or runs
- CLI → Runner entry: invokes
- Runner entry → Local runner: selects local
- Runner entry → Cloud runner: selects cloud
- Local runner → Step resolver: resolves steps
- Cloud runner → Step resolver: resolves steps
- Cloud runner → Job scheduler: submits jobs
- Job scheduler → Kubernetes: dispatches jobs
- Kubernetes → Kubernetes cluster: runs workloads
- Step resolver → Compute plugins: manages resources
- Step resolver → Step caching: checks cached steps
- Runner entry → HTTP API: reports run state
- HTTP API → Platform database: reads and writes
- HTTP API → Artifact storage: requests destinations
- Artifact storage → S3 backend: uses backend
- Step resolver → Artifact storage: transfers artifacts
- Step resolver → Metrics storage: records metrics
- Metrics storage → Platform database: stores metric data
- Web dashboard → API client: requests platform data
- Pipeline views → API client: loads pipeline runs
- Run inspection → API client: loads run details
- API client → HTTP API: calls API
- Git metadata → Platform database: tracks run metadata
- Web dashboard → Run inspection: opens run inspection