How john-mwangi/data_pipeline works
• Data Sources: Represent the origins of your data. Include components like CSV Files, JSON Files, and Remote URLs since these are mentioned in both the file_tree and the README.
• Fetch Data – implemented in the main pipeline (main.py and pipeline.py).
• Process & Clean Data – includes operations like name splitting and date conversions.
• Validate Data – capture usage of Pandera for runtime type checking.
• Calculate Data Quality Metrics – represents the computation of categorical counts, min/max values, and null counts.
• SQLite Database: Display the database layer where processed data and metrics are stored (as seen in the file tree with pipeline.db and references in the README to processed_data and data_quality tables).
• API Layer: Illustrate the REST API architecture built on FastAPI with additional functions:
• API Server (FastAPI implementation in api.py)
• Authentication (Basic Auth configured via config.yaml)
• Rate Limiting and Pagination support, illustrating middleware or dedicated components that manage these aspects.
Interactions and data flow between components
• Within the ETL pipeline, chain the processes from fetching → processing → validation → metrics computation.
• Database to the API layer, showing that the FastAPI server retrieves data for client requests.
Incorporate architectural patterns and design principles
• Separation of concerns: ETL processing is decoupled from the API serving logic.
• Modularity in the codebase (separate directories for data, src, tests, and output) to reflect maintainability.
• Use consistent naming and clear boundaries (e.g., Data Sources vs. ETL Pipeline vs. API Layer).
Specific technologies
• Use of Polars for data processing.
• Call out FastAPI for the API server.
• Explicitly note the use of Pandera in the data validation step.