How akbaritabar/internal-and-international-migration-of-scientists works
Main Components
• Data Sources & Resources
• Raw data files and mapping resources in the “resources/mapping” folder (e.g., GeoNames, shapefiles, and CSV files). These represent the input geospatial and migration metadata.
• Migration and processed data folders under “resources” (and possibly “results”) that hold the aggregated data and outputs.
• Configuration & Environment Setup
• A “config” folder houses configuration files (i.e., config.py) with parameters that likely drive the pipeline.
• The conda environment configuration (provided in the README using “requirements.yml”) specifies the Python version and libraries (pandas, geopandas, duckdb, etc.) that are used to run the pipeline.
• Workflow Engine (SnakeMake Pipeline)
• The “workflow” folder contains the Snakefile that defines the reproducible pipeline. It orchestrates the various data processing and analysis rules, including those for mapping, aggregation, and preparation of migration measures.
• Under “workflow/rules” you see individual rule scripts (e.g., snakefile_duckdb_data_processing.smk, snakefile_prepare_mapping_data.smk) that detail specific processing steps.
• Python Scripts & Data Preparation
• In the “workflow/scripts” folder there are numerous Python files (e.g., prepare_data_for_mapping.py, subnational_regions_nmr_line_plots.py) that serve as modules/tasks in the pipeline.
• They handle data cleaning, mapping, and seem to interact with both local data (CSV/parquet) and geospatial libraries.
• R Scripts for Statistical Analysis & Visualization
• Located in “workflow/scripts/Danko”, R scripts perform statistical analyses (e.g., fit_Kendall.R, make_tables.R) and produce figures that are later stored in results (as seen in “results/figures”).
• These scripts utilize packages like brms, mgcv, and others mentioned in the README.
• Output Results & Publication Materials
• The “results” folder contains processed datasets, figures (maps, graphs for Figures 1–4), and tables (e.g., descriptive_table*.tex). These are the end products of the pipeline.
• The “docs” folder (and particularly the publication figures in “_publication_figures”) provide the external visualization artifacts referenced in the manuscript.
Relationships and Data Flow
• Data Input flows from raw resources (geospatial files, migration data) and preprocessed configurations.
• The SnakeMake pipeline (in the “workflow” folder) orchestrates the execution of Python scripts that read these data sources, perform data transformation, and generate output files.
• These outputs (CSV, parquet, figures) are stored in the “results” folder, which then serve as input for the R scripts.
• The R scripts (under “workflow/scripts/Danko”) perform statistical modeling and generate additional tables/plots, which are also stored in “results” and referenced in the manuscript.
• Finally, the generated documentation (in “docs”) and publication figures (e.g., in “_publication_figures”) provide the final public-facing assets.
Architectural Patterns & Technologies
• The project follows a reproducible pipeline design using SnakeMake for dependency management (DAG of processing steps).
• There is a clear separation of concerns: Python for data processing/mapping and R for statistical analysis and modeling.
Detailed Elements
• Clearly label every node so that viewers understand what each folder or process is responsible for.
– Data sources (e.g., rectangular nodes)
– Workflow components and scripts (e.g., circles or hexagons)
Emphasize
• The reproducible and modular design, where the SnakeMake workflow ensures that dependencies (in the form of file inputs and outputs) are respected.
• The integration of multiple languages (Python and R) within a single pipeline.
• How the configuration and environment settings are spread across the system in a modular but tightly coupled way for reproducibility.
• That the final product is both a demonstration of complex data processing (geospatial mapping and statistical modeling of migration flows) and an end-to-end reproducible research workflow.