How jgmelon22/miniofileprocessor works
Purpose and Runtime
System A is a .NET HTTP API that accepts CSV uploads, performs lightweight admission validation, stores the file in S3-compatible object storage, persists import metadata in MySQL, and emits a Kafka notification for asynchronous processing by System B. The solution entry point is FileUploaderPartA.sln; API startup is src/FileUploaderPartA.API/Program.cs.
API Boundary
ImportsController is the inbound HTTP boundary for import creation. It receives upload requests and delegates work through the application layer rather than directly accessing storage or persistence.
• src/FileUploaderPartA.API/Controllers/ImportsController.cs
• src/FileUploaderPartA.API/Program.cs
• src/FileUploaderPartA.API/Extensions/IocExtensions.cs
Swagger is part of the API runtime/documentation surface, as indicated by the README and API project.
Application Command Orchestration
Import submission is modeled as a create command and handler. The handler coordinates validation, object upload, metadata creation/persistence, and Kafka publication through abstractions supplied by infrastructure DI.
• src/FileUploaderPartA.Application/Imports/Commands/CreateImportCommand.cs
• src/FileUploaderPartA.Application/Imports/Commands/Handlers/CreateImportCommandHandler.cs
• src/FileUploaderPartA.Application/FileUploaderPartA.Application.csproj
This is the primary control-flow boundary between HTTP transport and business workflow.
Domain Model and Validation Gate
The core defines the import entity, request DTO, lifecycle status, mapping logic, and shared result semantics. Validation intentionally remains shallow and synchronous: CSV MIME type, expected header structure, and maximum 7 MB size. Full row-level/business processing is deferred to System B.
• src/FileUploaderPartA.Core/Domains/Imports/Entities/Import.cs
• src/FileUploaderPartA.Core/Domains/Imports/Enums/Status.cs
• src/FileUploaderPartA.Core/Shared/Validations/ FileValidations/CsvValidations.cs
The stored Status supports downstream idempotency/processing checks.
Object Storage Boundary
Files are uploaded through an S3 service abstraction and implementation. Configuration names Amazon S3, while the deployed/local target is MinIO, which exposes an S3-compatible API. The persisted import record retains the resulting storage path rather than file contents.
• src/FileUploaderPartA.Infrastructure/Interfaces/Services/IS3Service.cs
• src/FileUploaderPartA.Infrastructure/Services/S3Service.cs
• src/FileUploaderPartA.Infrastructure/Configurations/AmazonS3Configuration.cs
Persistence Boundary
MySQL stores import metadata and processing status. The infrastructure uses Dapper via a database context and an import-specific repository interface/implementation, keeping SQL persistence outside the domain and application workflow.
• src/FileUploaderPartA.Infrastructure/Data/DapperDbContext.cs
• src/FileUploaderPartA.Infrastructure/Interfaces/Repository/IImportRepository.cs
• src/FileUploaderPartA.Infrastructure/Repositories/ImportRepository.cs
Event Publication and Downstream Handoff
After upload and metadata persistence, the application publishes an import event through Kafka. System B consumes the topic, checks the database status, and processes file content asynchronously. This decouples costly CSV processing from request latency, but makes storage, database persistence, and message publication a multi-system consistency boundary.
• src/FileUploaderPartA.Infrastructure/Interfaces/Services/IKafkaProducerService.cs
• src/FileUploaderPartA.Infrastructure/Services/KafkaProducerService.cs
• src/FileUploaderPartA.Infrastructure/Configurations/KafkaConfiguration.cs
Main entry points: CreateImportCommandHandler.cs Create Import Handler · IocExtensions.cs IoC Extensions · S3Service.cs S3 Service · ImportRepository.cs Import Repository
Components 18
API Runtime
- API Startup & DI · Configures HTTP runtime, Swagger, and infrastructure bindings.
src/FileUploaderPartA.API/Program.cs - IoC Extensions
src/FileUploaderPartA.API/Extensions/IocExtensions.cs - Imports Controller · Inbound endpoint for CSV import creation.
src/FileUploaderPartA.API/Controllers/ImportsController.cs
Application
- Create Import Command
src/FileUploaderPartA.Application/Imports/Commands/CreateImportCommand.cs - Create Import Handler · Coordinates validation, upload, persistence, and event publication.
src/FileUploaderPartA.Application/Imports/Commands/Handlers/CreateImportCommandHandler.cs
Core Domain
- Import Entity & Status · Metadata includes storage path and processing lifecycle status.
src/FileUploaderPartA.Core/Domains/Imports/Entities/Import.cs - CSV Admission Validation · Checks MIME type, header structure, and 7 MB maximum size.
src/FileUploaderPartA.Core/Shared/Validations/ FileValidations/CsvValidations.cs
Infrastructure
- S3 Service · Uploads files through the S3 abstraction.
src/FileUploaderPartA.Infrastructure/Services/S3Service.cs - S3 Service Contract
src/FileUploaderPartA.Infrastructure/Interfaces/Services/IS3Service.cs - Dapper DB Context
src/FileUploaderPartA.Infrastructure/Data/DapperDbContext.cs - Import Repository · Persists import metadata and status.
src/FileUploaderPartA.Infrastructure/Repositories/ImportRepository.cs - Import Repository Contract
src/FileUploaderPartA.Infrastructure/Interfaces/Repository/IImportRepository.cs - Kafka Producer · Publishes the import notification after persistence.
src/FileUploaderPartA.Infrastructure/Services/KafkaProducerService.cs - Kafka Producer Contract
src/FileUploaderPartA.Infrastructure/Interfaces/Services/IKafkaProducerService.cs
External Systems
- MinIO / S3 Storage · S3-compatible file storage.
- MySQL · Stores import metadata and processing status.
- Kafka Topic · Asynchronous handoff for import processing.
- System B · Consumes events, checks status, and processes CSV rows.
Connections 23
- API Startup & DI → IoC Extensions: loads registrations
- API Startup & DI → Imports Controller: hosts
- IoC Extensions → Create Import Handler: binds
- IoC Extensions → S3 Service: binds
- IoC Extensions → Import Repository: binds
- IoC Extensions → Kafka Producer: binds
- Imports Controller → Create Import Command: creates
- Create Import Command → Create Import Handler: handled by
- Create Import Handler → CSV Admission Validation: validates
- Create Import Handler → Import Entity & Status: creates metadata
- Create Import Handler → S3 Service Contract: uploads file
- S3 Service Contract → S3 Service: implemented by
- S3 Service → MinIO / S3 Storage: stores object
- Create Import Handler → Import Repository Contract: persists metadata
- Import Repository Contract → Import Repository: implemented by
- Import Repository → Dapper DB Context: uses
- Dapper DB Context → MySQL: writes
- Create Import Handler → Kafka Producer Contract: publishes import event
- Kafka Producer Contract → Kafka Producer: implemented by
- Kafka Producer → Kafka Topic: produces
- Kafka Topic → System B: consumed by
- System B → MySQL: checks status
- System B → MinIO / S3 Storage: reads CSV