How mjulian-alv/middleware works
DataPipeline is a configurable ETL system: users define pipelines and reusable sources, then run them on demand or by schedule to load data into a serving store. The runtime hydrates source configuration, reads batches, applies transforms, and writes results while tracking run counts and incremental watermarks. The API and CLI provide execution and management entry points; persistence stores metadata and run history. The tree also shows CSV/XLSX inputs, schema discovery, connection testing, and admin services. No README was provided, so capabilities and relationships here are based on the source index and shown excerpts.
Main entry points: PipelineRunner.cs Pipeline runner · EfPipelineStore.cs Pipeline store · RunsController.cs Run management · DataSourcesController.cs Source management
Components 25
Management API
- REST API
DataPipeline.Api/Program.cs - Basic auth
DataPipeline.Api/Auth/BasicAuthHandler.cs - Data access
DataPipeline.Api/Controllers/DataController.cs - Source management
DataPipeline.Api/Controllers/DataSourcesController.cs - Pipeline management
DataPipeline.Api/Controllers/PipelinesController.cs - Run management
DataPipeline.Api/Controllers/RunsController.cs - CLI
DataPipeline.Cli/Program.cs - Admin services
DataPipeline.Admin/Services
Pipeline runtime
- Pipeline definition
DataPipeline.Core/Pipelines/PipelineDefinition.cs - Source hydrator
DataPipeline.Engine/PipelineHydrator.cs - Pipeline runner
DataPipeline.Engine/PipelineRunner.cs - Scheduler
DataPipeline.Engine/EjecutorProgramado.cs - Scheduler host
DataPipeline.Api/Hosting/PipelineSchedulerHostedService.cs
Connectors
- Source factory
DataPipeline.Connectors/Factories/SourceFactory.cs - Transform factory
DataPipeline.Connectors/Factories/TransformFactory.cs - SQL, CSV, XLSX
DataPipeline.Connectors/Sources - Column mapping
DataPipeline.Connectors/Transforms/ColumnMapTransform.cs - Connection and schema checks
DataPipeline.Connectors/Testing - Encryption
DataPipeline.Connectors/Security/EncryptionService.cs
Persistence
- Pipeline store
DataPipeline.Persistence/EfPipelineStore.cs - Source store
DataPipeline.Persistence/EfDataSourceStore.cs - Run history
DataPipeline.Persistence/EfHistorialCorridas.cs - Metadata database
DataPipeline.Persistence/MetadataDbContext.cs - SQLite serving store
DataPipeline.Connectors/Stores/SqliteDataStore.cs
Other components
- Operator
Connections 27
- Operator → REST API: manages and runs
- REST API → Basic auth: configures auth
DataPipeline.Api/Program.cs - Run management → Source hydrator: hydrates sources
DataPipeline.Api/Controllers/RunsController.cs - Run management → Pipeline runner: starts runs
DataPipeline.Api/Controllers/RunsController.cs - Run management → Pipeline store: loads definitions
DataPipeline.Api/Controllers/RunsController.cs - Run management → Run history: records runs
DataPipeline.Api/Controllers/RunsController.cs - Data access → Pipeline store: loads definitions
DataPipeline.Api/Controllers/DataController.cs - Data access → SQLite serving store: reads data
DataPipeline.Api/Controllers/DataController.cs - Source management → Source store: manages sources
DataPipeline.Api/Controllers/DataSourcesController.cs - Source management → Connection and schema checks: tests and discovers
DataPipeline.Api/Controllers/DataSourcesController.cs - Source management → Encryption: encrypts secrets
DataPipeline.Api/Controllers/DataSourcesController.cs - Pipeline management → Pipeline store: manages definitions
DataPipeline.Api/Controllers/PipelinesController.cs - Source hydrator → Source store: loads source config
DataPipeline.Engine/PipelineHydrator.cs - Source hydrator → Encryption: decrypts credentials
DataPipeline.Engine/PipelineHydrator.cs - Pipeline runner → Pipeline definition: executes definition
DataPipeline.Engine/PipelineRunner.cs - Pipeline runner → Source factory: creates source
DataPipeline.Engine/PipelineRunner.cs - Pipeline runner → Transform factory: creates transforms
DataPipeline.Engine/PipelineRunner.cs - Pipeline runner → SQLite serving store: writes batches
DataPipeline.Engine/PipelineRunner.cs - Source factory → SQL, CSV, XLSX: creates connectors
DataPipeline.Connectors/Factories/SourceFactory.cs - Transform factory → Column mapping: creates transform
DataPipeline.Connectors/Factories/TransformFactory.cs - Scheduler host → Scheduler: hosts scheduler
DataPipeline.Api/Hosting/PipelineSchedulerHostedService.cs - CLI → Pipeline runner: runs pipelines
DataPipeline.Cli/Program.cs - CLI → Pipeline store: loads definitions
DataPipeline.Cli/Program.cs - Admin services → REST API: uses API
- Pipeline store → Metadata database: reads and writes
DataPipeline.Persistence/EfPipelineStore.cs - Source store → Metadata database: persists sources
DataPipeline.Api/Program.cs - Run history → Metadata database: persists history
DataPipeline.Api/Program.cs