# kanaji2002/ci-skip architecture

> Architecture diagram of the GitHub repository kanaji2002/ci-skip, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kanaji2002/ci-skip
- Repository: https://github.com/kanaji2002/ci-skip

## Overview

##### Project type and purpose
- This is a data-replication / data-curation pipeline written in Python.
- Its goal is to fetch raw data (commit histories, CI data, file metadata) from GitHub (and possibly CI services), transform and merge these pieces, and produce final datasets and analysis artifacts for a research paper’s replication appendix.

##### File structure
- Top-level:
- data/replication – all replication materials and pipelines
- README.md – (placeholder)
- Under data/replication/data-curation:
- “pipeline” scripts: 1_1, 1_2, 2_… – staged ETL steps
- clone_and_get_data.py, get_ci_data.py, get_dependencies_for_each_commit.py, get_version_changing_commits.py – discrete data-fetch modules
- data_downloader/script – wrappers around external APIs (GitHub, giPython)
- create_dataset.py – consolidates processed inputs into final dataset
- Other files: zipped archives (datasets.zip, data-analysis.zip), Online Appendix.pdf
- No server or UI code – purely batch pipelines and data artifacts.

##### README for additional insights
- (README not yet populated)
- Rely on file names to infer that each numbered pipeline step builds on the previous.

##### Essential details to capture
- All numbered pipeline steps in order
- The two wrapper layers (common vs. specific) under data_downloader
- The transformation from raw fetch to merged dataset
- Final outputs (datasets.zip, data-analysis.zip, Online Appendix.pdf)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    GH_API["GitHub API"]:::external
    CI_API["CI Service API"]:::external

    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        BEGIN["Downloader Entry: begin_download.py"]:::pipeline
        WRAPPERS["Wrappers Module"]:::pipeline
        COMMON["Common Utils: common.py"]:::pipeline
    end

    %% ETL Stages
    subgraph "ETL Pipeline Stages"
        STAGE0["Stage 0: Clone & Fetch Repo Data"]:::pipeline
        STAGE1A["Stage 1.1: Fetch Commit Data"]:::pipeline
        STAGE1B["Stage 1.2a: Fetch CI Data from GitHub"]:::pipeline
        STAGE1C["Stage 1.2b: Fetch File Metadata from GitHub"]:::pipeline
        AUX1["Aux: get_ci_data.py"]:::pipeline
        AUX2["Aux: get_dependencies_for_each_commit.py"]:::pipeline
        AUX3["Aux: get_version_changing_commits.py"]:::pipeline
        STAGE2["Stage 2: Merge & Transform All Data"]:::pipeline
    end

    %% Dataset Builders
    subgraph "Dataset Builders"
        DATASET["create_dataset.py"]:::pipeline
        DATASET_BKP["create_dataset_bkp_circlei.py"]:::pipeline
    end

    %% Data Stores and Outputs
    RAW["Raw/Intermediate Files (JSON/CSV/Pickle)"]:::datastore
    FINAL_DS["Final Datasets Archive (datasets.zip)"]:::datastore
    ANALYSIS["Data Analysis Artifacts (data-analysis.zip)"]:::datastore
    MANUAL["Manual Analysis Artifacts (manual_analysis.zip)"]:::datastore
    APPENDIX["Online Appendix.pdf"]:::datastore

    %% Data Flow
    GH_API -->|fetch| BEGIN
    CI_API -->|fetch| BEGIN
    BEGIN --> WRAPPERS
    BEGIN --> COMMON
    WRAPPERS --> STAGE0
    COMMON --> STAGE0

    STAGE0 --> STAGE1A
    STAGE1A --> STAGE1B
    STAGE1A --> STAGE1C
    STAGE1B --> STAGE2
    STAGE1C --> STAGE2

    AUX1 --> STAGE2
    AUX2 --> STAGE2
    AUX3 --> STAGE2

    STAGE2 --> RAW
    RAW --> DATASET
    RAW --> DATASET_BKP
    DATASET --> FINAL_DS
    DATASET_BKP --> FINAL_DS

    FINAL_DS --> ANALYSIS
    FINAL_DS --> MANUAL
    FINAL_DS --> APPENDIX

    %% Click Events
    click WRAPPERS "https://github.com/kanaji2002/ci-skip/tree/main/data/replication/data-curation/data_downloader/script/wrappers"
    click COMMON "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/data_downloader/script/common/common.py"
    click BEGIN "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/data_downloader/script/begin_download.py"
    click STAGE0 "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/clone_and_get_data.py"
    click STAGE1A "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/1_1_pipeline_get_commit_data.py"
    click STAGE1B "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/1_2_pipeline_get_ci_data_from_gh.py"
    click STAGE1C "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/1_2_pipeline_get_files_data_from_gh.py"
    click AUX1 "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/get_ci_data.py"
    click AUX2 "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/get_dependencies_for_each_commit.py"
    click AUX3 "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/get_version_changing_commits.py"
    click STAGE2 "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/2_pipeline_merge_everything.py"
    click DATASET "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/create_dataset.py"
    click DATASET_BKP "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-curation/create_dataset_bkp_circlei.py"
    click FINAL_DS "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/datasets.zip"
    click ANALYSIS "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/data-analysis.zip"
    click MANUAL "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/manual_analysis.zip"
    click APPENDIX "https://github.com/kanaji2002/ci-skip/blob/main/data/replication/Online Appendix.pdf"

    %% Styles
    classDef external fill:#bbf,stroke:#333
    classDef pipeline fill:#bfb,stroke:#333
    classDef datastore fill:#fb3,stroke:#333,stroke-dasharray: 5 5
```
