# akbaritabar/internal-and-international-migration-of-scientists architecture

> Architecture diagram of the GitHub repository akbaritabar/internal-and-international-migration-of-scientists, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akbaritabar/internal-and-international-migration-of-scientists
- Repository: https://github.com/akbaritabar/internal-and-international-migration-of-scientists

## Overview

##### Main Components
- Data Sources & Resources
- Raw data files and mapping resources in the “resources/mapping” folder (e.g., GeoNames, shapefiles, and CSV files). These represent the input geospatial and migration metadata.
- Migration and processed data folders under “resources” (and possibly “results”) that hold the aggregated data and outputs.

- Configuration & Environment Setup
- A “config” folder houses configuration files (i.e., config.py) with parameters that likely drive the pipeline.
- The conda environment configuration (provided in the README using “requirements.yml”) specifies the Python version and libraries (pandas, geopandas, duckdb, etc.) that are used to run the pipeline.

- Workflow Engine (SnakeMake Pipeline)
- The “workflow” folder contains the Snakefile that defines the reproducible pipeline. It orchestrates the various data processing and analysis rules, including those for mapping, aggregation, and preparation of migration measures.
- Under “workflow/rules” you see individual rule scripts (e.g., snakefile_duckdb_data_processing.smk, snakefile_prepare_mapping_data.smk) that detail specific processing steps.

- Python Scripts & Data Preparation
- In the “workflow/scripts” folder there are numerous Python files (e.g., prepare_data_for_mapping.py, subnational_regions_nmr_line_plots.py) that serve as modules/tasks in the pipeline.
- They handle data cleaning, mapping, and seem to interact with both local data (CSV/parquet) and geospatial libraries.

- R Scripts for Statistical Analysis & Visualization
- Located in “workflow/scripts/Danko”, R scripts perform statistical analyses (e.g., fit_Kendall.R, make_tables.R) and produce figures that are later stored in results (as seen in “results/figures”).
- These scripts utilize packages like brms, mgcv, and others mentioned in the README.

- Output Results & Publication Materials
- The “results” folder contains processed datasets, figures (maps, graphs for Figures 1–4), and tables (e.g., descriptive_table*.tex). These are the end products of the pipeline.
- The “docs” folder (and particularly the publication figures in “_publication_figures”) provide the external visualization artifacts referenced in the manuscript.

##### Relationships and Data Flow
- Data Input flows from raw resources (geospatial files, migration data) and preprocessed configurations.
- The SnakeMake pipeline (in the “workflow” folder) orchestrates the execution of Python scripts that read these data sources, perform data transformation, and generate output files.
- These outputs (CSV, parquet, figures) are stored in the “results” folder, which then serve as input for the R scripts.
- The R scripts (under “workflow/scripts/Danko”) perform statistical modeling and generate additional tables/plots, which are also stored in “results” and referenced in the manuscript.
- Finally, the generated documentation (in “docs”) and publication figures (e.g., in “_publication_figures”) provide the final public-facing assets.

##### Architectural Patterns & Technologies
- The project follows a reproducible pipeline design using SnakeMake for dependency management (DAG of processing steps).
- There is a clear separation of concerns: Python for data processing/mapping and R for statistical analysis and modeling.

##### Detailed Elements
- Clearly label every node so that viewers understand what each folder or process is responsible for.
– Data sources (e.g., rectangular nodes)
– Workflow components and scripts (e.g., circles or hexagons)

##### Emphasize
- The reproducible and modular design, where the SnakeMake workflow ensures that dependencies (in the form of file inputs and outputs) are respected.
- The integration of multiple languages (Python and R) within a single pipeline.
- How the configuration and environment settings are spread across the system in a modular but tightly coupled way for reproducibility.
- That the final product is both a demonstration of complex data processing (geospatial mapping and statistical modeling of migration flows) and an end-to-end reproducible research workflow.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        DS1["Mapping Files"]:::data
        DS2["Migration Data"]:::data
        DS3["Processed Data"]:::data
    end

    %% Configuration & Environment Setup
    subgraph "Configuration & Environment Setup"
        CE1["Config File (config.py)"]:::config
        CE2["Conda Environment (workflow/envs)"]:::config
    end

    %% Workflow Engine (SnakeMake Pipeline)
    subgraph "Workflow Engine (SnakeMake Pipeline)"
        WE1["Snakefile"]:::pipeline
        WE2["Snake Rules"]:::pipeline
    end

    %% Python Data Processing Scripts
    P["Python Data Processing Scripts"]:::python

    %% Intermediate Results
    IR["Intermediate Results (results)"]:::results

    %% R Statistical Analysis Scripts
    R["R Statistical Analysis Scripts"]:::rscript

    %% Final Outputs & Publication Materials
    subgraph "Final Outputs & Publication Materials"
        FO1["Publication Figures (_publication_figures)"]:::output
        FO2["Documentation (docs)"]:::output
    end

    %% Data Flow Connections
    DS1 --->|"provides"| WE1
    DS2 --->|"provides"| WE1
    DS3 --->|"provides"| WE1

    CE1 --->|"configures"| WE1
    CE2 --->|"configures"| WE1

    WE1 --->|"orchestrates"| P
    WE2 --->|"defines rules for"| P

    P --->|"generates"| IR
    IR --->|"feeds"| R
    R --->|"produces"| FO1
    R --->|"produces"| FO2

    %% Click Events for Component Mapping
    click DS1 "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/resources/mapping"
    click DS2 "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/resources/migration"
    click DS3 "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/resources/scp_processed"

    click CE1 "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/blob/main/config/config.py"
    click CE2 "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/workflow/envs"

    click WE1 "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/workflow/Snakefile"
    click WE2 "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/workflow/rules"

    click P "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/workflow/scripts"
    click R "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/workflow/scripts/Danko"
    click IR "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/results"
    click FO1 "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/_publication_figures"
    click FO2 "https://github.com/akbaritabar/internal-and-international-migration-of-scientists/tree/main/docs"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef config fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef pipeline fill:#FFDAB9,stroke:#000,stroke-width:1px;
    classDef python fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef results fill:#DDA0DD,stroke:#000,stroke-width:1px;
    classDef rscript fill:#FFB6C1,stroke:#000,stroke-width:1px;
    classDef output fill:#F0E68C,stroke:#000,stroke-width:1px;
```
