# mjulian-alv/middleware architecture

> Architecture diagram of the GitHub repository mjulian-alv/middleware, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mjulian-alv/middleware
- Repository: https://github.com/mjulian-alv/middleware

## Overview

DataPipeline is a configurable ETL system: users define pipelines and reusable sources, then run them on demand or by schedule to load data into a serving store. The runtime hydrates source configuration, reads batches, applies transforms, and writes results while tracking run counts and incremental watermarks. The API and CLI provide execution and management entry points; persistence stores metadata and run history. The tree also shows CSV/XLSX inputs, schema discovery, connection testing, and admin services. No README was provided, so capabilities and relationships here are based on the source index and shown excerpts.

Main entry points: [`DataPipeline.Engine/PipelineRunner.cs`](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Engine/PipelineRunner.cs) (Pipeline runner), [`DataPipeline.Persistence/EfPipelineStore.cs`](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Persistence/EfPipelineStore.cs) (Pipeline store), [`DataPipeline.Api/Controllers/RunsController.cs`](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Controllers/RunsController.cs) (Run management), [`DataPipeline.Api/Controllers/DataSourcesController.cs`](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Controllers/DataSourcesController.cs) (Source management)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_management["Management API"]
  node_api["REST API<br/>[Program.cs]"]
  node_auth["Basic auth"]
  node_data_api["Data access<br/>[DataController.cs]"]
  node_sources_api["Source management"]
  node_pipelines_api["Pipeline management"]
  node_runs_api["Run management<br/>[RunsController.cs]"]
  node_cli["CLI<br/>[Program.cs]"]
  node_admin_services["Admin services<br/>[Services]"]
end

subgraph group_runtime["Pipeline runtime"]
  node_pipeline_def["Pipeline definition"]
  node_hydrator["Source hydrator"]
  node_runner["Pipeline runner<br/>[PipelineRunner.cs]"]
  node_scheduler["Scheduler"]
  node_scheduler_host["Scheduler host"]
end

subgraph group_connectors["Connectors"]
  node_source_factory["Source factory<br/>[SourceFactory.cs]"]
  node_transform_factory["Transform factory"]
  node_source_connectors["SQL, CSV, XLSX<br/>[Sources]"]
  node_transforms["Column mapping"]
  node_testing["Connection and schema checks<br/>[Testing]"]
  node_encryption["Encryption"]
end

subgraph group_storage["Persistence"]
  node_pipeline_store["Pipeline store<br/>[EfPipelineStore.cs]"]
  node_data_source_store["Source store"]
  node_run_history["Run history"]
  node_metadata_db[("Metadata database")]
  node_serving_store[("SQLite serving store<br/>[SqliteDataStore.cs]")]
end

node_operator(("Operator"))

node_operator -->|"manages and runs"| node_api
node_api -->|"configures auth"| node_auth
node_runs_api -->|"hydrates sources"| node_hydrator
node_runs_api -->|"starts runs"| node_runner
node_runs_api -->|"loads definitions"| node_pipeline_store
node_runs_api -->|"records runs"| node_run_history
node_data_api -->|"loads definitions"| node_pipeline_store
node_data_api -->|"reads data"| node_serving_store
node_sources_api -->|"manages sources"| node_data_source_store
node_sources_api -->|"tests and discovers"| node_testing
node_sources_api -->|"encrypts secrets"| node_encryption
node_pipelines_api -->|"manages definitions"| node_pipeline_store
node_hydrator -->|"loads source config"| node_data_source_store
node_hydrator -->|"decrypts credentials"| node_encryption
node_runner -->|"executes definition"| node_pipeline_def
node_runner -->|"creates source"| node_source_factory
node_runner -->|"creates transforms"| node_transform_factory
node_runner -->|"writes batches"| node_serving_store
node_source_factory -->|"creates connectors"| node_source_connectors
node_transform_factory -->|"creates transform"| node_transforms
node_scheduler_host -->|"hosts scheduler"| node_scheduler
node_cli -->|"runs pipelines"| node_runner
node_cli -->|"loads definitions"| node_pipeline_store
node_admin_services -.->|"uses API"| node_api
node_pipeline_store -->|"reads and writes"| node_metadata_db
node_data_source_store -->|"persists sources"| node_metadata_db
node_run_history -->|"persists history"| node_metadata_db

click node_api "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Program.cs"
click node_auth "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Auth/BasicAuthHandler.cs"
click node_data_api "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Controllers/DataController.cs"
click node_sources_api "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Controllers/DataSourcesController.cs"
click node_pipelines_api "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Controllers/PipelinesController.cs"
click node_runs_api "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Controllers/RunsController.cs"
click node_cli "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Cli/Program.cs"
click node_admin_services "https://github.com/mjulian-alv/middleware/tree/main/DataPipeline.Admin/Services"
click node_pipeline_def "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Core/Pipelines/PipelineDefinition.cs"
click node_hydrator "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Engine/PipelineHydrator.cs"
click node_runner "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Engine/PipelineRunner.cs"
click node_scheduler "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Engine/EjecutorProgramado.cs"
click node_scheduler_host "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Hosting/PipelineSchedulerHostedService.cs"
click node_source_factory "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Connectors/Factories/SourceFactory.cs"
click node_transform_factory "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Connectors/Factories/TransformFactory.cs"
click node_source_connectors "https://github.com/mjulian-alv/middleware/tree/main/DataPipeline.Connectors/Sources"
click node_transforms "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Connectors/Transforms/ColumnMapTransform.cs"
click node_testing "https://github.com/mjulian-alv/middleware/tree/main/DataPipeline.Connectors/Testing"
click node_encryption "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Connectors/Security/EncryptionService.cs"
click node_pipeline_store "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Persistence/EfPipelineStore.cs"
click node_data_source_store "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Persistence/EfDataSourceStore.cs"
click node_run_history "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Persistence/EfHistorialCorridas.cs"
click node_metadata_db "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Persistence/MetadataDbContext.cs"
click node_serving_store "https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Connectors/Stores/SqliteDataStore.cs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_auth,node_data_api,node_sources_api,node_pipelines_api,node_runs_api,node_cli,node_admin_services toneBlue
class node_pipeline_def,node_hydrator,node_runner,node_scheduler,node_scheduler_host toneAmber
class node_source_factory,node_transform_factory,node_source_connectors,node_transforms,node_testing,node_encryption toneMint
class node_pipeline_store,node_data_source_store,node_run_history,node_metadata_db,node_serving_store toneRose
class node_operator toneIndigo
```

## Components

### Management API

- [REST API](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Program.cs): `DataPipeline.Api/Program.cs`
- [Basic auth](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Auth/BasicAuthHandler.cs): `DataPipeline.Api/Auth/BasicAuthHandler.cs`
- [Data access](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Controllers/DataController.cs): `DataPipeline.Api/Controllers/DataController.cs`
- [Source management](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Controllers/DataSourcesController.cs): `DataPipeline.Api/Controllers/DataSourcesController.cs`
- [Pipeline management](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Controllers/PipelinesController.cs): `DataPipeline.Api/Controllers/PipelinesController.cs`
- [Run management](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Controllers/RunsController.cs): `DataPipeline.Api/Controllers/RunsController.cs`
- [CLI](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Cli/Program.cs): `DataPipeline.Cli/Program.cs`
- [Admin services](https://github.com/mjulian-alv/middleware/tree/main/DataPipeline.Admin/Services): `DataPipeline.Admin/Services`

### Pipeline runtime

- [Pipeline definition](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Core/Pipelines/PipelineDefinition.cs): `DataPipeline.Core/Pipelines/PipelineDefinition.cs`
- [Source hydrator](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Engine/PipelineHydrator.cs): `DataPipeline.Engine/PipelineHydrator.cs`
- [Pipeline runner](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Engine/PipelineRunner.cs): `DataPipeline.Engine/PipelineRunner.cs`
- [Scheduler](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Engine/EjecutorProgramado.cs): `DataPipeline.Engine/EjecutorProgramado.cs`
- [Scheduler host](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Api/Hosting/PipelineSchedulerHostedService.cs): `DataPipeline.Api/Hosting/PipelineSchedulerHostedService.cs`

### Connectors

- [Source factory](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Connectors/Factories/SourceFactory.cs): `DataPipeline.Connectors/Factories/SourceFactory.cs`
- [Transform factory](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Connectors/Factories/TransformFactory.cs): `DataPipeline.Connectors/Factories/TransformFactory.cs`
- [SQL, CSV, XLSX](https://github.com/mjulian-alv/middleware/tree/main/DataPipeline.Connectors/Sources): `DataPipeline.Connectors/Sources`
- [Column mapping](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Connectors/Transforms/ColumnMapTransform.cs): `DataPipeline.Connectors/Transforms/ColumnMapTransform.cs`
- [Connection and schema checks](https://github.com/mjulian-alv/middleware/tree/main/DataPipeline.Connectors/Testing): `DataPipeline.Connectors/Testing`
- [Encryption](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Connectors/Security/EncryptionService.cs): `DataPipeline.Connectors/Security/EncryptionService.cs`

### Persistence

- [Pipeline store](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Persistence/EfPipelineStore.cs): `DataPipeline.Persistence/EfPipelineStore.cs`
- [Source store](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Persistence/EfDataSourceStore.cs): `DataPipeline.Persistence/EfDataSourceStore.cs`
- [Run history](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Persistence/EfHistorialCorridas.cs): `DataPipeline.Persistence/EfHistorialCorridas.cs`
- [Metadata database](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Persistence/MetadataDbContext.cs): `DataPipeline.Persistence/MetadataDbContext.cs`
- [SQLite serving store](https://github.com/mjulian-alv/middleware/blob/main/DataPipeline.Connectors/Stores/SqliteDataStore.cs): `DataPipeline.Connectors/Stores/SqliteDataStore.cs`

### Other components

- **Operator**

## Connections

- Operator → REST API: manages and runs
- REST API → Basic auth: configures auth (evidence: `DataPipeline.Api/Program.cs`)
- Run management → Source hydrator: hydrates sources (evidence: `DataPipeline.Api/Controllers/RunsController.cs`)
- Run management → Pipeline runner: starts runs (evidence: `DataPipeline.Api/Controllers/RunsController.cs`)
- Run management → Pipeline store: loads definitions (evidence: `DataPipeline.Api/Controllers/RunsController.cs`)
- Run management → Run history: records runs (evidence: `DataPipeline.Api/Controllers/RunsController.cs`)
- Data access → Pipeline store: loads definitions (evidence: `DataPipeline.Api/Controllers/DataController.cs`)
- Data access → SQLite serving store: reads data (evidence: `DataPipeline.Api/Controllers/DataController.cs`)
- Source management → Source store: manages sources (evidence: `DataPipeline.Api/Controllers/DataSourcesController.cs`)
- Source management → Connection and schema checks: tests and discovers (evidence: `DataPipeline.Api/Controllers/DataSourcesController.cs`)
- Source management → Encryption: encrypts secrets (evidence: `DataPipeline.Api/Controllers/DataSourcesController.cs`)
- Pipeline management → Pipeline store: manages definitions (evidence: `DataPipeline.Api/Controllers/PipelinesController.cs`)
- Source hydrator → Source store: loads source config (evidence: `DataPipeline.Engine/PipelineHydrator.cs`)
- Source hydrator → Encryption: decrypts credentials (evidence: `DataPipeline.Engine/PipelineHydrator.cs`)
- Pipeline runner → Pipeline definition: executes definition (evidence: `DataPipeline.Engine/PipelineRunner.cs`)
- Pipeline runner → Source factory: creates source (evidence: `DataPipeline.Engine/PipelineRunner.cs`)
- Pipeline runner → Transform factory: creates transforms (evidence: `DataPipeline.Engine/PipelineRunner.cs`)
- Pipeline runner → SQLite serving store: writes batches (evidence: `DataPipeline.Engine/PipelineRunner.cs`)
- Source factory → SQL, CSV, XLSX: creates connectors (evidence: `DataPipeline.Connectors/Factories/SourceFactory.cs`)
- Transform factory → Column mapping: creates transform (evidence: `DataPipeline.Connectors/Factories/TransformFactory.cs`)
- Scheduler host → Scheduler: hosts scheduler (evidence: `DataPipeline.Api/Hosting/PipelineSchedulerHostedService.cs`)
- CLI → Pipeline runner: runs pipelines (evidence: `DataPipeline.Cli/Program.cs`)
- CLI → Pipeline store: loads definitions (evidence: `DataPipeline.Cli/Program.cs`)
- Admin services → REST API: uses API
- Pipeline store → Metadata database: reads and writes (evidence: `DataPipeline.Persistence/EfPipelineStore.cs`)
- Source store → Metadata database: persists sources (evidence: `DataPipeline.Api/Program.cs`)
- Run history → Metadata database: persists history (evidence: `DataPipeline.Api/Program.cs`)
