# rahulkris1/metadata-driven-etl-backend architecture

> Architecture diagram of the GitHub repository rahulkris1/metadata-driven-etl-backend, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rahulkris1/metadata-driven-etl-backend
- Repository: https://github.com/rahulkris1/metadata-driven-etl-backend

## Overview

#### Purpose and API entry
This is a FastAPI foundation for a metadata-driven ETL platform. Its current exposed surface is operational: liveness and metadata-database readiness endpoints. Application construction starts in `app/main.py`, with versioned API composition in `app/api/router.py` and route handlers under `app/api/routes/health.py`.

#### Application composition and request boundary
`app/main.py` assembles configuration, logging, middleware, exception handling, and the API router. Cross-cutting HTTP behavior belongs at this boundary: request IDs and process-time response headers, CORS, and normalized error responses. Relevant paths: `app/core/middleware.py`, `app/core/exceptions.py`, `app/core/logging.py`.

#### Configuration and integration contracts
Runtime configuration is environment-driven, optionally loading `.env`; no integration secrets are embedded in source. `app/core/config.py` is the central contract for application settings, metadata DB connectivity, and future Azure, Databricks, Airflow, and storage integrations. `.env.example` defines deployer-facing configuration, while `docker-compose.yml` supplies a PostgreSQL-oriented local container setup.

#### Metadata persistence layer
The metadata database is the durable system-of-record boundary for ETL metadata. SQLAlchemy-style model definitions are organized in `app/models/base.py`, `app/models/metadata.py`, and `app/models/enums.py`; validation-oriented model concerns are separated in `app/models/validation.py`. Database engine/session lifecycle is centralized in `app/core/database.py`.

#### Repository access boundary
Repositories isolate persistence operations from HTTP handlers and future ETL orchestration/services. `app/repositories/base.py` provides shared repository behavior, while `app/repositories/metadata.py` owns metadata-specific access patterns. API handlers obtain database resources through `app/api/dependencies/database.py`, avoiding direct connection lifecycle management in routes.

#### Schema and API response boundary
Pydantic/API schemas remain distinct from ORM models, preventing database representations from becoming the public contract. Current health response contracts live in `app/schemas/health.py`; route implementation in `app/api/routes/health.py` uses these contracts to report process health and database readiness.

#### Runtime, deployment, and schema evolution
The service runs as an ASGI application (Uvicorn locally) and is containerized through `Dockerfile` and `scripts/start.sh`. Alembic governs metadata-schema evolution via `alembic.ini`, `migrations/env.py`, and versioned revisions such as `migrations/versions/a2b7ec202dc3_create_metadata_model.py`. SQLite is the default local database; PostgreSQL is the intended Compose/example-environment metadata backend.

Main entry points: [`app/main.py`](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/main.py) (Application factory), [`app/core/database.py`](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/database.py) (Engine and sessions), [`app/repositories/metadata.py`](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/repositories/metadata.py) (Metadata repositories), [`app/api/routes/health.py`](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/api/routes/health.py) (Health endpoints)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_app["FastAPI service"]
  node_main{{"Application factory<br/>FastAPI composition<br/>[main.py]"}}
  node_router["Versioned API router<br/>[router.py]"]
  node_health_routes["Health endpoints<br/>route handlers<br/>[health.py]"]
  node_health_schemas["Health response schemas<br/>Pydantic schemas<br/>[health.py]"]
  node_middleware["HTTP middleware<br/>cross-cutting HTTP<br/>[middleware.py]"]
  node_exceptions["Exception handling<br/>error boundary<br/>[exceptions.py]"]
  node_logging["Application logging<br/>[logging.py]"]
  node_config["Runtime settings<br/>environment configuration<br/>[config.py]"]
  node_db_dependency["Database dependency<br/>request-scoped dependency<br/>[database.py]"]
end

subgraph group_persistence["Metadata persistence"]
  node_database["Engine and sessions<br/>SQLAlchemy lifecycle<br/>[database.py]"]
  node_repositories["Metadata repositories<br/>repository access<br/>[metadata.py]"]
  node_repository_base["Repository base<br/>shared repository behavior<br/>[base.py]"]
  node_orm_models["Metadata ORM models<br/>SQLAlchemy models<br/>[metadata.py]"]
  node_metadata_db[("Metadata database<br/>SQLite / PostgreSQL")]
end

subgraph group_runtime["Runtime and delivery"]
  node_migrations["Alembic migrations<br/>schema evolution<br/>[env.py]"]
  node_container_runtime["Container runtime<br/>Docker ASGI service"]
  node_compose_db[("Local PostgreSQL<br/>Compose database<br/>[docker-compose.yml]")]
  node_env_example["Environment template<br/>deployment configuration<br/>[.env.example]"]
end

node_main -->|"mounts"| node_router
node_router -->|"routes"| node_health_routes
node_health_routes -->|"returns"| node_health_schemas
node_main -->|"installs"| node_middleware
node_main -->|"registers"| node_exceptions
node_main -->|"configures"| node_logging
node_main -->|"loads"| node_config
node_health_routes -->|"requests session"| node_db_dependency
node_db_dependency -->|"uses"| node_database
node_repositories -->|"extends"| node_repository_base
node_repositories -->|"persists through"| node_database
node_repositories -->|"maps"| node_orm_models
node_database -->|"connects to"| node_metadata_db
node_config -->|"provides connectivity settings"| node_database
node_migrations -->|"evolves schema for"| node_orm_models
node_migrations -->|"applies revisions"| node_metadata_db
node_container_runtime -->|"runs ASGI app"| node_main
node_compose_db -.->|"provides local backend"| node_metadata_db
node_env_example -.->|"documents settings"| node_config

click node_main "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/main.py"
click node_router "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/api/router.py"
click node_health_routes "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/api/routes/health.py"
click node_health_schemas "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/schemas/health.py"
click node_middleware "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/middleware.py"
click node_exceptions "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/exceptions.py"
click node_logging "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/logging.py"
click node_config "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/config.py"
click node_db_dependency "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/api/dependencies/database.py"
click node_database "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/database.py"
click node_repositories "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/repositories/metadata.py"
click node_repository_base "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/repositories/base.py"
click node_orm_models "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/models/metadata.py"
click node_migrations "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/migrations/env.py"
click node_container_runtime "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/Dockerfile"
click node_compose_db "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/docker-compose.yml"
click node_env_example "https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/.env.example"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_router,node_health_routes,node_health_schemas,node_middleware,node_exceptions,node_logging,node_config,node_db_dependency toneBlue
class node_database,node_repositories,node_repository_base,node_orm_models,node_metadata_db toneAmber
class node_migrations,node_container_runtime,node_compose_db,node_env_example toneMint
```

## Components

### FastAPI service

HTTP application and API boundary.

- [Application factory](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/main.py): FastAPI composition · Assembles settings, logging, middleware, errors, and API routes. · `app/main.py`
- [Versioned API router](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/api/router.py): router · `app/api/router.py`
- [Health endpoints](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/api/routes/health.py): route handlers · Exposes liveness and metadata-database readiness. · `app/api/routes/health.py`
- [Health response schemas](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/schemas/health.py): Pydantic schemas · Public health response contracts. · `app/schemas/health.py`
- [HTTP middleware](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/middleware.py): cross-cutting HTTP · Adds request IDs, process-time headers, and CORS behavior. · `app/core/middleware.py`
- [Exception handling](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/exceptions.py): error boundary · Normalizes HTTP error responses. · `app/core/exceptions.py`
- [Application logging](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/logging.py): logging · `app/core/logging.py`
- [Runtime settings](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/config.py): environment configuration · Defines database and future integration contracts. · `app/core/config.py`
- [Database dependency](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/api/dependencies/database.py): request-scoped dependency · Supplies database resources to routes. · `app/api/dependencies/database.py`

### Metadata persistence

Durable metadata system of record.

- [Engine and sessions](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/core/database.py): SQLAlchemy lifecycle · Centralizes database engine and session management. · `app/core/database.py`
- [Metadata repositories](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/repositories/metadata.py): repository access · Isolates metadata persistence operations. · `app/repositories/metadata.py`
- [Repository base](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/repositories/base.py): shared repository behavior · `app/repositories/base.py`
- [Metadata ORM models](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/app/models/metadata.py): SQLAlchemy models · Metadata entities, base definitions, enums, and validation concerns. · `app/models/metadata.py`
- **Metadata database**: SQLite / PostgreSQL · Durable ETL metadata system of record.

### Runtime and delivery

Local, container, and schema-evolution tooling.

- [Alembic migrations](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/migrations/env.py): schema evolution · Applies versioned metadata schema revisions. · `migrations/env.py`
- [Container runtime](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/Dockerfile): Docker ASGI service · Starts the Uvicorn-hosted application. · `Dockerfile`
- [Local PostgreSQL](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/docker-compose.yml): Compose database · PostgreSQL-oriented local metadata backend. · `docker-compose.yml`
- [Environment template](https://github.com/rahulkris1/metadata-driven-etl-backend/blob/main/.env.example): deployment configuration · Documents deployer-facing settings. · `.env.example`

## Connections

- Application factory → Versioned API router: mounts
- Versioned API router → Health endpoints: routes
- Health endpoints → Health response schemas: returns
- Application factory → HTTP middleware: installs
- Application factory → Exception handling: registers
- Application factory → Application logging: configures
- Application factory → Runtime settings: loads
- Health endpoints → Database dependency: requests session
- Database dependency → Engine and sessions: uses
- Metadata repositories → Repository base: extends
- Metadata repositories → Engine and sessions: persists through
- Metadata repositories → Metadata ORM models: maps
- Engine and sessions → Metadata database: connects to
- Runtime settings → Engine and sessions: provides connectivity settings
- Alembic migrations → Metadata ORM models: evolves schema for
- Alembic migrations → Metadata database: applies revisions
- Container runtime → Application factory: runs ASGI app
- Local PostgreSQL → Metadata database: provides local backend
- Environment template → Runtime settings: documents settings