How darshilparmar/twitter-airflow-data-engineering-project works
This small Airflow project schedules a daily Twitter extraction, refines returned tweets into selected fields, and writes them to a CSV file. The DAG invokes the ETL function; that function performs the API request and handles refinement and export. The README provides little architectural detail, and although s3fs is imported, the sampled ETL code does not use it, so no S3 integration is shown.
Main entry points: twitter_dag.py Daily DAG · twitter_etl.py ETL function
Components 6
Orchestration
- Daily DAG
twitter_dag.py
Tweet ETL
- ETL function
twitter_etl.py - Tweet refinement
twitter_etl.py - Refined CSV
Other components
- Airflow scheduler
- Twitter API
Connections 5
- Airflow scheduler → Daily DAG: triggers schedule
- Daily DAG → ETL function: invokes
- ETL function → Twitter API: requests timeline
- Twitter API → Tweet refinement: returns tweets
- Tweet refinement → Refined CSV: writes CSV