How stagepfe2025/custom-etl-application works
Project type and purpose
• This is a full-stack, enterprise-grade ETL (Extract-Transform-Load) web application written in Python.
• It uses Flask to expose a RESTful API and UI, Celery for background/parallel tasks, Redis as broker and cache, and Postgres/Elasticsearch for storage.
File structure
• Top level:
• Dockerfile, docker-compose.dev.yml, manage.py → containerized deployment and management
• celery_app.py, config.py → Celery and application configuration
• app/ → Flask application:
– app/api → HTTP endpoints (admin/routes)
– app/main → web UI routes and template rendering
– app/tasks → Celery ETL and scheduled tasks
– app/extractors, app/transformers, app/loaders → modular ETL pipeline
– app/extensions.py → Flask extensions (DB, Redis, Celery)
– app/templates → Jinja2 templates for dashboard and monitoring
• docs/ and TROUBLESHOOTING.md provide deep dives on scaling, task management, large file uploads, etc.
• tests/ and test_*.py show the integration/ unit tests around APIs and monitoring.
Extract additional insights from the README
• Architecture overview shows three main containers: Web Service, Celery Workers (standard + chunked), Redis (stats + queues) and a Flower monitor.
• Processing workflow stages: Upload → Validate/Chunk → Queue → Email Discovery (4 finders in parallel) → Transform/Enrich → Load → Report.
• Data stores: Postgres for relational outputs, Elasticsearch for search/analytics, Redis for real-time stats and job queues.
a. Main components:
1. Client (Browser or API consumer)
2. Web Service (Flask App)
3. Task Broker & Cache (Redis)
4. Celery Workers (Standard + Chunked + Scheduled)
5. Extractors (File, S3, API)
6. Transformers (Company enrichment, Data cleaning)
7. Loaders (Postgres Loader, Elasticsearch Loader, CSV Loader)
8. External Systems (S3 buckets, external APIs, WHOIS services, SMTP servers)
9. Databases: Postgres, Elasticsearch
10. Monitoring: Flower, Live Dashboard (served by Flask), Logs
b. Component interactions / data flow:
1. Client → HTTP POST /api/v1/jobs → Web Service
2. Web Service
• Validates file (CSV/JSON), chunks large files
• Pushes ETL tasks into Redis queue via Celery
3. Celery Workers fetch tasks from Redis
• Extractors read from file system, S3, or external APIs
• Transformers enrich and clean data
• Email discovery sub-tasks run in parallel using multiple methods, push stats to Redis
• Loaders write results to Postgres & Elasticsearch
4. Web Service / Dashboard polls Redis & DB to show real-time stats, job status, logs
5. Flower (Celery monitor) connects to Redis to show worker status & queue health
c. Architectural patterns & design principles:
• Modular ETL pipeline: extract → transform → load components in separate modules
• Producer–Consumer (Flask produces tasks, Celery consumes)
• Parallel/Asynchronous processing to maximize throughput
• MVC for Flask (routes → templates → extensions)
• Caching & pooling for performance (Redis cache, HTTP session pooling)
d. Key technologies/frameworks:
• Flask (app/api, app/main)
• Celery for distributed tasks
• Redis as both broker and in-memory stats store
• Postgres & Elasticsearch for persistent storage
• Jinja2 templates for UI
• Docker & docker-compose for container orchestration
• Python libraries: requests (pooled HTTP), whois, smtplib
Detail
• Both standard and chunked worker pools if desired