# luisgarcia206/globant_data-engineering-challenge architecture

> Architecture diagram of the GitHub repository luisgarcia206/globant_data-engineering-challenge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/luisgarcia206/globant_data-engineering-challenge
- Repository: https://github.com/luisgarcia206/globant_data-engineering-challenge

## Overview

This is a FastAPI hiring-data ingestion and analytics service. Clients upload CSVs or submit JSON batches; the API delegates parsing and batch processing, which validates records, quarantines rejects, and persists accepted rows in PostgreSQL. Separate API routes provide Avro backup/restore and hiring reports. The graph also shows configuration, storage adapters, and SQL-defined reporting views. README and sampled source support the shown relationships; unsampled deployment and service internals are not expanded.

Main entry points: [`core/database.py`](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/core/database.py) (Database access), [`services/ingestion_service.py`](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/services/ingestion_service.py) (CSV and batch processing), [`api/main.py`](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/api/main.py) (FastAPI application), [`core/storage.py`](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/core/storage.py) (Storage adapters)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API"]
  node_application["FastAPI application<br/>[main.py]"]
  node_ingestion_routes["Ingestion routes<br/>[ingestion.py]"]
  node_backup_routes["Backup routes<br/>[backup.py]"]
  node_report_routes["Report routes<br/>[reports.py]"]
end

subgraph group_ingestion["Ingestion"]
  node_csv_batch["CSV and batch processing"]
  node_validation["Record validation"]
  node_contracts["Record contracts<br/>[records.py]"]
end

subgraph group_platform["Data platform"]
  node_database_access["Database access<br/>[database.py]"]
  node_postgres[("PostgreSQL")]
  node_configuration["Runtime configuration<br/>[config.py]"]
  node_storage_adapter["Storage adapters<br/>[storage.py]"]
  node_rejection_log["Rejected-record log<br/>[logging.py]"]
end

subgraph group_recovery_analytics["Recovery and analytics"]
  node_avro["Avro snapshots<br/>[avro_service.py]"]
  node_avro_schemas["Avro schemas"]
  node_report_views["Analytics views<br/>[init.sql]"]
end

node_client(("Data client"))
node_gcs[("Google Cloud Storage")]

node_client -->|"sends requests"| node_application
node_application -->|"registers routes"| node_ingestion_routes
node_application -->|"registers routes"| node_backup_routes
node_application -->|"registers routes"| node_report_routes
node_application -->|"checks health"| node_database_access
node_ingestion_routes -->|"delegates processing"| node_csv_batch
node_csv_batch -->|"validates rows"| node_validation
node_validation -->|"validates against"| node_contracts
node_csv_batch -->|"maps columns"| node_contracts
node_validation -->|"loads foreign keys"| node_database_access
node_validation -->|"logs rejects"| node_rejection_log
node_csv_batch -->|"inserts valid rows"| node_database_access
node_csv_batch -->|"saves raw CSV"| node_storage_adapter
node_csv_batch -->|"syncs rejects"| node_rejection_log
node_database_access -->|"manages connections"| node_postgres
node_database_access -->|"reads settings"| node_configuration
node_storage_adapter -->|"selects backend"| node_configuration
node_rejection_log -->|"reads log settings"| node_configuration
node_rejection_log -->|"syncs log file"| node_storage_adapter
node_storage_adapter -.->|"stores objects"| node_gcs
node_backup_routes -->|"delegates backup and restore"| node_avro
node_avro -->|"reads and restores tables"| node_database_access
node_avro -->|"reads and writes snapshots"| node_storage_adapter
node_avro -->|"loads schemas"| node_avro_schemas
node_report_routes -->|"queries reports"| node_database_access
node_report_routes -->|"queries views"| node_report_views
node_report_views -->|"defines SQL views"| node_postgres

click node_application "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/api/main.py"
click node_ingestion_routes "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/api/routes/ingestion.py"
click node_backup_routes "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/api/routes/backup.py"
click node_report_routes "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/api/routes/reports.py"
click node_csv_batch "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/services/ingestion_service.py"
click node_validation "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/services/validator_service.py"
click node_contracts "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/schemas/records.py"
click node_database_access "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/core/database.py"
click node_configuration "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/core/config.py"
click node_storage_adapter "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/core/storage.py"
click node_rejection_log "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/core/logging.py"
click node_avro "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/services/avro_service.py"
click node_avro_schemas "https://github.com/luisgarcia206/globant_data-engineering-challenge/tree/main/schemas/avro"
click node_report_views "https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/sql/init.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_application,node_ingestion_routes,node_backup_routes,node_report_routes,node_client toneBlue
class node_csv_batch,node_validation,node_contracts,node_gcs toneAmber
class node_database_access,node_postgres,node_configuration,node_storage_adapter,node_rejection_log toneMint
class node_avro,node_avro_schemas,node_report_views toneRose
```

## Components

### API

- [FastAPI application](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/api/main.py): `api/main.py`
- [Ingestion routes](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/api/routes/ingestion.py): `api/routes/ingestion.py`
- [Backup routes](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/api/routes/backup.py): `api/routes/backup.py`
- [Report routes](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/api/routes/reports.py): `api/routes/reports.py`

### Ingestion

- [CSV and batch processing](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/services/ingestion_service.py): `services/ingestion_service.py`
- [Record validation](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/services/validator_service.py): `services/validator_service.py`
- [Record contracts](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/schemas/records.py): `schemas/records.py`

### Data platform

- [Database access](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/core/database.py): `core/database.py`
- **PostgreSQL**
- [Runtime configuration](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/core/config.py): `core/config.py`
- [Storage adapters](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/core/storage.py): `core/storage.py`
- [Rejected-record log](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/core/logging.py): `core/logging.py`

### Recovery and analytics

- [Avro snapshots](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/services/avro_service.py): `services/avro_service.py`
- [Avro schemas](https://github.com/luisgarcia206/globant_data-engineering-challenge/tree/main/schemas/avro): `schemas/avro`
- [Analytics views](https://github.com/luisgarcia206/globant_data-engineering-challenge/blob/main/sql/init.sql): `sql/init.sql`

### Other components

- **Data client**
- **Google Cloud Storage**

## Connections

- Data client → FastAPI application: sends requests (evidence: `README.md`)
- FastAPI application → Ingestion routes: registers routes (evidence: `api/main.py`)
- FastAPI application → Backup routes: registers routes (evidence: `api/main.py`)
- FastAPI application → Report routes: registers routes (evidence: `api/main.py`)
- FastAPI application → Database access: checks health (evidence: `api/main.py`)
- Ingestion routes → CSV and batch processing: delegates processing (evidence: `api/routes/ingestion.py`)
- CSV and batch processing → Record validation: validates rows (evidence: `services/ingestion_service.py`)
- Record validation → Record contracts: validates against (evidence: `services/validator_service.py`)
- CSV and batch processing → Record contracts: maps columns (evidence: `services/ingestion_service.py`)
- Record validation → Database access: loads foreign keys (evidence: `services/validator_service.py`)
- Record validation → Rejected-record log: logs rejects (evidence: `services/validator_service.py`)
- CSV and batch processing → Database access: inserts valid rows (evidence: `services/ingestion_service.py`)
- CSV and batch processing → Storage adapters: saves raw CSV (evidence: `services/ingestion_service.py`)
- CSV and batch processing → Rejected-record log: syncs rejects (evidence: `services/ingestion_service.py`)
- Database access → PostgreSQL: manages connections (evidence: `README.md`)
- Database access → Runtime configuration: reads settings (evidence: `core/database.py`)
- Storage adapters → Runtime configuration: selects backend (evidence: `core/storage.py`)
- Rejected-record log → Runtime configuration: reads log settings (evidence: `core/logging.py`)
- Rejected-record log → Storage adapters: syncs log file (evidence: `core/logging.py`)
- Storage adapters → Google Cloud Storage: stores objects (evidence: `core/storage.py`)
- Backup routes → Avro snapshots: delegates backup and restore (evidence: `api/routes/backup.py`)
- Avro snapshots → Database access: reads and restores tables (evidence: `services/avro_service.py`)
- Avro snapshots → Storage adapters: reads and writes snapshots (evidence: `services/avro_service.py`)
- Avro snapshots → Avro schemas: loads schemas (evidence: `services/avro_service.py`)
- Report routes → Database access: queries reports (evidence: `api/routes/reports.py`)
- Report routes → Analytics views: queries views (evidence: `api/routes/reports.py`)
- Analytics views → PostgreSQL: defines SQL views (evidence: `README.md`)