How rahulkris1/metadata-driven-etl-backend works
Purpose and API entry
This is a FastAPI foundation for a metadata-driven ETL platform. Its current exposed surface is operational: liveness and metadata-database readiness endpoints. Application construction starts in app/main.py, with versioned API composition in app/api/router.py and route handlers under app/api/routes/health.py.
Application composition and request boundary
app/main.py assembles configuration, logging, middleware, exception handling, and the API router. Cross-cutting HTTP behavior belongs at this boundary: request IDs and process-time response headers, CORS, and normalized error responses. Relevant paths: app/core/middleware.py, app/core/exceptions.py, app/core/logging.py.
Configuration and integration contracts
Runtime configuration is environment-driven, optionally loading .env; no integration secrets are embedded in source. app/core/config.py is the central contract for application settings, metadata DB connectivity, and future Azure, Databricks, Airflow, and storage integrations. .env.example defines deployer-facing configuration, while docker-compose.yml supplies a PostgreSQL-oriented local container setup.
Metadata persistence layer
The metadata database is the durable system-of-record boundary for ETL metadata. SQLAlchemy-style model definitions are organized in app/models/base.py, app/models/metadata.py, and app/models/enums.py; validation-oriented model concerns are separated in app/models/validation.py. Database engine/session lifecycle is centralized in app/core/database.py.
Repository access boundary
Repositories isolate persistence operations from HTTP handlers and future ETL orchestration/services. app/repositories/base.py provides shared repository behavior, while app/repositories/metadata.py owns metadata-specific access patterns. API handlers obtain database resources through app/api/dependencies/database.py, avoiding direct connection lifecycle management in routes.
Schema and API response boundary
Pydantic/API schemas remain distinct from ORM models, preventing database representations from becoming the public contract. Current health response contracts live in app/schemas/health.py; route implementation in app/api/routes/health.py uses these contracts to report process health and database readiness.
Runtime, deployment, and schema evolution
The service runs as an ASGI application (Uvicorn locally) and is containerized through Dockerfile and scripts/start.sh. Alembic governs metadata-schema evolution via alembic.ini, migrations/env.py, and versioned revisions such as migrations/versions/a2b7ec202dc3_create_metadata_model.py. SQLite is the default local database; PostgreSQL is the intended Compose/example-environment metadata backend.
Main entry points: main.py Application factory · database.py Engine and sessions · metadata.py Metadata repositories · health.py Health endpoints
Components 18
FastAPI service
HTTP application and API boundary.
- Application factory · Assembles settings, logging, middleware, errors, and API routes.
app/main.py - Versioned API router
app/api/router.py - Health endpoints · Exposes liveness and metadata-database readiness.
app/api/routes/health.py - Health response schemas · Public health response contracts.
app/schemas/health.py - HTTP middleware · Adds request IDs, process-time headers, and CORS behavior.
app/core/middleware.py - Exception handling · Normalizes HTTP error responses.
app/core/exceptions.py - Application logging
app/core/logging.py - Runtime settings · Defines database and future integration contracts.
app/core/config.py - Database dependency · Supplies database resources to routes.
app/api/dependencies/database.py
Metadata persistence
Durable metadata system of record.
- Engine and sessions · Centralizes database engine and session management.
app/core/database.py - Metadata repositories · Isolates metadata persistence operations.
app/repositories/metadata.py - Repository base
app/repositories/base.py - Metadata ORM models · Metadata entities, base definitions, enums, and validation concerns.
app/models/metadata.py - Metadata database · Durable ETL metadata system of record.
Runtime and delivery
Local, container, and schema-evolution tooling.
- Alembic migrations · Applies versioned metadata schema revisions.
migrations/env.py - Container runtime · Starts the Uvicorn-hosted application.
Dockerfile - Local PostgreSQL · PostgreSQL-oriented local metadata backend.
docker-compose.yml - Environment template · Documents deployer-facing settings.
.env.example
Connections 19
- Application factory → Versioned API router: mounts
- Versioned API router → Health endpoints: routes
- Health endpoints → Health response schemas: returns
- Application factory → HTTP middleware: installs
- Application factory → Exception handling: registers
- Application factory → Application logging: configures
- Application factory → Runtime settings: loads
- Health endpoints → Database dependency: requests session
- Database dependency → Engine and sessions: uses
- Metadata repositories → Repository base: extends
- Metadata repositories → Engine and sessions: persists through
- Metadata repositories → Metadata ORM models: maps
- Engine and sessions → Metadata database: connects to
- Runtime settings → Engine and sessions: provides connectivity settings
- Alembic migrations → Metadata ORM models: evolves schema for
- Alembic migrations → Metadata database: applies revisions
- Container runtime → Application factory: runs ASGI app
- Local PostgreSQL → Metadata database: provides local backend
- Environment template → Runtime settings: documents settings