How fcrispcoach/airflow-bank-transactions works
Project type and purpose
• This is a data-engineering + ML pipeline orchestrated with Apache Airflow.
• Purpose: generate and ingest synthetic time-series and bank-transaction data, validate data quality, detect anomalies/fraud via ML (Isolation Forest & Prophet), store results in PostgreSQL, and serve interactive dashboards via Streamlit.
File structure
• Top-level:
• airflow.cfg, airflow.db, webserver_config.py → Airflow setup and metadata
• requirements.txt → dependencies
• check_reports.py → custom monitoring or report checker
• config/: Python modules for path and global configuration
• dags/: Airflow DAG definitions
• anomaly_detection.py
• bank_transactions_etl.py
• bank_transactions_validation.py
• plugins/: custom Airflow operators
• custom_operators/data_quality_operator.py (wraps Great Expectations and Cerberus checks)
• data/: raw JSON/CSV stores and data population scripts (synthetic generation)
• database/: SQL schema and sample-data population scripts for PostgreSQL
• models/: ML training scripts (Isolation Forest model)
• schemas/: Cerberus JSON schema definitions for transaction validation
• dashboard/: Streamlit app for interactive visualization
• tests/: unit tests for validation and ETL logic
Extract insights from the README
• Orchestration: Apache Airflow schedules and coordinates tasks
• Data generation: Faker + custom scripts for both time series and transactions
• Validation: Great Expectations (data quality) + Cerberus (schema)
• ML: Scikit-learn (Isolation Forest for fraud/anomaly) & Prophet (time-series forecasting)
• Storage: PostgreSQL for structured data
• Visualization: Streamlit dashboard
a. Main components
1. Data Generation
– Synthetic time-series script
– Faker-based transaction generator
2. Raw Data Storage
– Flat files (JSON/CSV in data/)
3. Orchestrator
– Apache Airflow (Scheduler + Webserver)
– DAGs: ETL, validation, anomaly detection
– Custom Operators (data_quality_operator)
4. Validation Layer
– Great Expectations checks
– Cerberus schema validation
5. Processing & ML
– ETL scripts (pandas)
– Model training (Isolation Forest, Prophet)
6. Database
– PostgreSQL (initialized by database/init_db.sql)
7. Dashboard
– Streamlit app (dashboard/app.py)
8. Supporting Modules
– config/paths.py, schemas/transaction_schema.py, tests/
b. Relationships & interactions
• Data generators → raw data files → Airflow DAG start
• Airflow Scheduler → runs DAG tasks in order
• Validation tasks → fail early or proceed to ETL
• ETL tasks → load cleansed data into PostgreSQL
• ML tasks → read from PostgreSQL, train model, write anomaly flags back
• Dashboard → reads from PostgreSQL or summary tables, renders charts
c. Architectural patterns & principles
– Modular DAGs in Airflow (separate validation, ETL, ML)
– Separation of concerns: generation vs. validation vs. processing vs. serving
– Operator extensibility (custom Airflow operators for quality checks)
– CI via tests/ for critical pipelines
d. Key technologies & frameworks
– Apache Airflow (orchestrator)
– pandas (data processing)
– Great Expectations, Cerberus (validation)
– scikit-learn, Prophet (ML)
– PostgreSQL (storage)
– Streamlit (visualization)
– Faker (synthetic data)
• Use clear, concise labels: “Synthetic Generator,” “Validation Operator,” “Isolation Forest Task,” “PostgreSQL”
• Orchestration (e.g., blue)
• Storage (e.g., green)
• Processing/ML (e.g., orange)
• Visualization (e.g., purple)
• Rectangles for scripts or services
• Hexagons for custom operators