How luisgarcia206/globant_data-engineering-challenge works
This is a FastAPI hiring-data ingestion and analytics service. Clients upload CSVs or submit JSON batches; the API delegates parsing and batch processing, which validates records, quarantines rejects, and persists accepted rows in PostgreSQL. Separate API routes provide Avro backup/restore and hiring reports. The graph also shows configuration, storage adapters, and SQL-defined reporting views. README and sampled source support the shown relationships; unsampled deployment and service internals are not expanded.
Main entry points: database.py Database access · ingestion_service.py CSV and batch processing · main.py FastAPI application · storage.py Storage adapters
Components 17
API
- FastAPI application
api/main.py - Ingestion routes
api/routes/ingestion.py - Backup routes
api/routes/backup.py - Report routes
api/routes/reports.py
Ingestion
- CSV and batch processing
services/ingestion_service.py - Record validation
services/validator_service.py - Record contracts
schemas/records.py
Data platform
- Database access
core/database.py - PostgreSQL
- Runtime configuration
core/config.py - Storage adapters
core/storage.py - Rejected-record log
core/logging.py
Recovery and analytics
- Avro snapshots
services/avro_service.py - Avro schemas
schemas/avro - Analytics views
sql/init.sql
Other components
- Data client
- Google Cloud Storage
Connections 27
- Data client → FastAPI application: sends requests
README.md - FastAPI application → Ingestion routes: registers routes
api/main.py - FastAPI application → Backup routes: registers routes
api/main.py - FastAPI application → Report routes: registers routes
api/main.py - FastAPI application → Database access: checks health
api/main.py - Ingestion routes → CSV and batch processing: delegates processing
api/routes/ingestion.py - CSV and batch processing → Record validation: validates rows
services/ingestion_service.py - Record validation → Record contracts: validates against
services/validator_service.py - CSV and batch processing → Record contracts: maps columns
services/ingestion_service.py - Record validation → Database access: loads foreign keys
services/validator_service.py - Record validation → Rejected-record log: logs rejects
services/validator_service.py - CSV and batch processing → Database access: inserts valid rows
services/ingestion_service.py - CSV and batch processing → Storage adapters: saves raw CSV
services/ingestion_service.py - CSV and batch processing → Rejected-record log: syncs rejects
services/ingestion_service.py - Database access → PostgreSQL: manages connections
README.md - Database access → Runtime configuration: reads settings
core/database.py - Storage adapters → Runtime configuration: selects backend
core/storage.py - Rejected-record log → Runtime configuration: reads log settings
core/logging.py - Rejected-record log → Storage adapters: syncs log file
core/logging.py - Storage adapters → Google Cloud Storage: stores objects
core/storage.py - Backup routes → Avro snapshots: delegates backup and restore
api/routes/backup.py - Avro snapshots → Database access: reads and restores tables
services/avro_service.py - Avro snapshots → Storage adapters: reads and writes snapshots
services/avro_service.py - Avro snapshots → Avro schemas: loads schemas
services/avro_service.py - Report routes → Database access: queries reports
api/routes/reports.py - Report routes → Analytics views: queries views
api/routes/reports.py - Analytics views → PostgreSQL: defines SQL views
README.md