How zeus-deus/data-engineering_workflow works
Main Components
• Data Ingestion Module: Represent components that fetch raw data from external sources. This is implemented under src/data_ingestion (scripts like reasoning_errors_to_azure.py and reasoning_errors_to_mongodb.py). Show that raw data is pushed to either MongoDB or Azure Blob Storage.
• ETL Pipelines: Clearly indicate the ETL process present under src/pipelines/reasoning_errors (modules for extract, transform, and load). This pipeline moves data between storage solutions (e.g., MongoDB/Azure Blob for raw data and PostgreSQL/Azure PostgreSQL for structured data).
• Workflow Orchestration: The project uses Prefect for task automation and orchestration. Include the Prefect flows from the prefect/ directory (such as prefect_flow.py, prefect_vectorization_flow.py) that manage the execution and scheduling of tasks.
• Vectorization and RAG Integration: Highlight that there is a dedicated module (src/vectorization) for converting text data to vectors and a separate module under src/rag for integrating RAG-based question answering that leverages Qdrant and the OpenAI API.
• Containerization: Docker is used to create consistent, isolated environments. Use the docker directory and docker-compose.yml file to illustrate how multiple containers (application, MongoDB, PostgreSQL, etc.) interact.
• External Services: Identify integration points with external systems and APIs such as:
• Azure Services (Blob Storage and Azure PostgreSQL) – indicated by the environmental toggles.
• Qdrant – as a vector store, and the use of the OpenAI API for embedding generation.
• Other libraries/tools (BeautifulSoup, pandas, SQLAlchemy).
Relationships and Interactions
• ETL pipelines should be depicted as transforming data from one format/location (raw in MongoDB/Azure Blob) to processed data stored in PostgreSQL or Azure PostgreSQL.
• Dependencies between the Prefect orchestrator and individual pipelines, hinting at monitoring and automation.
• Integration between the vectorization component and the RAG module where processed or vectorized data is sent to Qdrant for enabling advanced query matching.
• That Docker components encapsulate many of these services to allow for uniform deployment across environments.
Architectural Patterns and Design Principles
• Modularity: The separation of data ingestion, ETL, vectorization, and workflow orchestration.
• Decoupling: The common utilities and configuration management (e.g., reading environment variables to switch between local and cloud databases).
• Containerization: Showing services as isolated containers that communicate via defined interfaces.
• Orchestration: The use of Prefect to manage and monitor pipeline execution, which enforces a clear separation of tasks and error handling.
• Extensibility: The structure supports plugging in additional data sources or additional pipeline stages with minimal changes.
Relevant Technologies
• Python and its libraries (pandas, BeautifulSoup, SQLAlchemy).
• NoSQL (MongoDB) and RDBMS (PostgreSQL / Azure PostgreSQL).
• Cloud services (Azure Blob Storage).
• Prefect for orchestration.
• Qdrant and OpenAI API for vectorization and RAG.
• Docker and Docker Compose for containerizing and managing multi-container setups.
Detail and Clarity
• Every logical unit and interaction is represented as accurately as possible.
• Don’t shy away from decomposing the project into many small blocks if that accurately reflects the modular nature of the architecture.