# hayato-n8810/hayalab architecture

> Architecture diagram of the GitHub repository hayato-n8810/hayalab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hayato-n8810/hayalab
- Repository: https://github.com/hayato-n8810/hayalab

## Overview

##### Project Type & Purpose
- This is neither a classical web app nor a microservice product; it’s a research‐oriented data‐processing pipeline written in Python (with some JS/Notebook support).

##### High-Level File Structure
- Top level:
- Docker/Docker-Compose files – containerize various steps.
- src/ – core pipeline modules, subdivided by function (collect_repo, trace, abst, pattern, measure, grouping, MBquery_analysis, util, const).
- results/ – outputs of each stage.
- archive/ – historical snapshots of past research topics (Omori, Zenkoku).
- README documents the purpose and module layout.

##### Key Components & Responsibilities
- Data Ingestion (src/collect_repo): clones JS repos from GitHub, outputs raw repos.
- Trace (src/trace): walks commit history and diffs to extract code-change snippets.
- Abstraction (src/abst): transforms raw JS into abstract representations.
- Pattern Detection (src/pattern): identifies known slow code patterns from diffs.
- Microbenchmarking (src/measure): runs benchmarks on pattern variants to measure speed differences.
- Grouping & Analysis (src/grouping, src/MBquery_analysis): clusters benchmark results, matches patterns, prepares plots.
- Utilities & Constants (src/util, src/const): shared helper scripts and configuration constants.
- Results Storage (results/, plus each module’s sub-folders): CSV, JSON, plot images.

##### External Dependencies & Infrastructure
- Docker containers for isolation of clone, measure, and other stages (Dockerfiles under src/collect_repo and src/measure).
- Docker-Compose orchestrates multi-container workflows.
- Jupyter notebooks (in archive) for ad-hoc visualization.
- UV runner (“uv run python …”) for uniform module execution.
- NPM for jsparser.js.

##### Architectural Patterns & Design Principles
- Pipeline/ETL pattern: clearly staged transformations.
- Modular layering: each “src/<component>” is a self-contained service.
- Reproducibility via containers: Dockerfiles + docker-compose.
- Separation of concerns: data collection, trace, abstraction, detection, benchmarking, grouping.

##### Technologies & Frameworks to Call Out
- Python (main language), Jupyter notebooks (visualization), JavaScript parser (jsparser.js)
- Docker, Docker-Compose for environment control
- JSON/CSV for interchange formats
- UV (CLI tool) for running scripts

##### Detail Level
- Essential interactions—especially the reuse of util/const across modules.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Define styles
    classDef container stroke:#f66,stroke-dasharray: 5 5,fill:#fee;
    classDef service fill:#6f6,stroke:#363;
    classDef datastore fill:#66f,stroke:#336;
    classDef library fill:#fc6,stroke:#c63;
    classDef notebook fill:#ccf,stroke:#66c;

    %% External source
    GH["GitHub Repos"]:::datastore

    %% Docker Compose boundary
    subgraph "Dockerized Pipeline" 
        direction TB
        class DockerizedPipeline container

        subgraph "Collect_Repo Service"
            direction TB
            CR["Collect_Repo Service"]:::service
            CR_DF["Dockerfile"] 
            CR_GH["github_repo_collect.py"]
            CR_JS["js_repo_collect.py"]
        end

        subgraph "Trace Service"
            direction TB
            TR["Trace Service"]:::service
            TR_CH["collect_git_history.py"]
            TR_CD["filter_commit_diffs.py"]
            TR_TC["trace_code.py"]
        end

        subgraph "Abstraction Service"
            direction TB
            AB["Abstraction Service"]:::service
            AB_M["main.py"]
            AB_CA["code_abstract.py"]
            AB_P["parser.py"]
        end

        subgraph "Pattern Detection Service"
            direction TB
            PD["Pattern Detector"]:::service
            PD_SP["slow_pattern.py"]
        end

        subgraph "Measure Service"
            direction TB
            MS["Measure Service"]:::service
            MS_DF["Dockerfile"]
            MS_DC["docker-compose.yml"]
            MS_MM["measure_microbenchmark.py"]
            MS_BS["benchmark_speed_diff.py"]
            MS_CJ["codes.json"]
        end
    end

    %% Grouping & Analysis
    subgraph "Grouping & Analysis"
        direction TB
        GR["Grouping Service"]:::service
        GR_GP["grouping.py"]
        GR_GA["grouping_ast.py"]
        GR_GD["gumtree_diff.py"]
        GR_SL["search_label.py"]
        GR_SS["separate_setup.py"]
        GR_SSJS["separate_setup.js"]
        subgraph "Plotting"
            direction TB
            GR_PL1["plot_loop_combinations.py"]
            GR_PL2["plot_loop_combinations_by_group.py"]
        end
    end

    subgraph "MBquery Analysis Service"
        direction TB
        MB["MBquery Analysis"]:::service
        MB_CI["compare_ids.py"]
        MB_FP["find_matching_pattern.py"]
    end

    %% Data stores
    LR["Local Repo Storage"]:::datastore
    RCS["Raw Change Snippets"]:::datastore
    AAST["Abstract ASTs"]:::datastore
    ISP["Identified Slow Patterns"]:::datastore
    BR["Benchmark Results"]:::datastore
    RS["Results Storage (results/)"]:::datastore

    %% Common libraries
    UTIL["Utilities (src/util)"]:::library
    CONST["Constants (src/const)"]:::library

    %% Reference systems
    ARCH["Archive Snapshots"]:::notebook
    NB["Jupyter Notebooks"]:::notebook

    %% Data flow
    GH -->|clone repos| CR
    CR -->|raw repos| LR
    LR -->|repo history| TR
    TR -->|diff JSON| RCS
    RCS -->|snippets| AB
    AB -->|ASTs| AAST
    AAST -->|AST diffs| PD
    PD -->|patterns| ISP
    ISP -->|variants| MS
    MS -->|benchmarks| BR
    BR -->|data| GR
    GR -->|plots/CSV/JSON| RS
    GR -->|analysis| MB
    MB -->|matched results| RS

    %% Dependencies on libraries
    UTIL --> CR
    UTIL --> TR
    UTIL --> AB
    UTIL --> PD
    UTIL --> MS
    UTIL --> GR
    UTIL --> MB

    CONST --> CR
    CONST --> TR
    CONST --> AB
    CONST --> PD
    CONST --> MS
    CONST --> GR
    CONST --> MB

    %% Reference arrows
    ARCH --> NB
    ARCH --> AB
    RS --> NB

    %% Click events for component mapping
    click CR "https://github.com/hayato-n8810/hayalab/tree/main/src/collect_repo/"
    click CR_DF "https://github.com/hayato-n8810/hayalab/tree/main/src/collect_repo/Dockerfile"
    click CR_GH "https://github.com/hayato-n8810/hayalab/blob/main/src/collect_repo/github_repo_collect.py"
    click CR_JS "https://github.com/hayato-n8810/hayalab/blob/main/src/collect_repo/js_repo_collect.py"

    click TR "https://github.com/hayato-n8810/hayalab/tree/main/src/trace/"
    click TR_CH "https://github.com/hayato-n8810/hayalab/blob/main/src/trace/collect_git_history.py"
    click TR_CD "https://github.com/hayato-n8810/hayalab/blob/main/src/trace/filter_commit_diffs.py"
    click TR_TC "https://github.com/hayato-n8810/hayalab/blob/main/src/trace/trace_code.py"

    click AB "https://github.com/hayato-n8810/hayalab/tree/main/src/abst/"
    click AB_M "https://github.com/hayato-n8810/hayalab/blob/main/src/abst/main.py"
    click AB_CA "https://github.com/hayato-n8810/hayalab/blob/main/src/abst/code_abstract.py"
    click AB_P "https://github.com/hayato-n8810/hayalab/blob/main/src/abst/parser.py"

    click PD_SP "https://github.com/hayato-n8810/hayalab/blob/main/src/pattern/slow_pattern.py"

    click MS "https://github.com/hayato-n8810/hayalab/tree/main/src/measure/"
    click MS_DF "https://github.com/hayato-n8810/hayalab/tree/main/src/measure/Dockerfile"
    click MS_DC "https://github.com/hayato-n8810/hayalab/blob/main/src/measure/docker-compose.yml"
    click MS_MM "https://github.com/hayato-n8810/hayalab/blob/main/src/measure/measure_microbenchmark.py"
    click MS_BS "https://github.com/hayato-n8810/hayalab/blob/main/src/measure/benchmark_speed_diff.py"
    click MS_CJ "https://github.com/hayato-n8810/hayalab/blob/main/src/measure/codes.json"

    click GR "https://github.com/hayato-n8810/hayalab/tree/main/src/grouping/"
    click GR_GP "https://github.com/hayato-n8810/hayalab/blob/main/src/grouping/grouping.py"
    click GR_GA "https://github.com/hayato-n8810/hayalab/blob/main/src/grouping/grouping_ast.py"
    click GR_GD "https://github.com/hayato-n8810/hayalab/blob/main/src/grouping/gumtree_diff.py"
    click GR_SL "https://github.com/hayato-n8810/hayalab/blob/main/src/grouping/search_label.py"
    click GR_SS "https://github.com/hayato-n8810/hayalab/blob/main/src/grouping/separate_setup.py"
    click GR_SSJS "https://github.com/hayato-n8810/hayalab/blob/main/src/grouping/separate_setup.js"
    click GR_PL1 "https://github.com/hayato-n8810/hayalab/blob/main/src/grouping/plot/plot_loop_combinations.py"
    click GR_PL2 "https://github.com/hayato-n8810/hayalab/blob/main/src/grouping/plot/plot_loop_combinations_by_group.py"

    click MB "https://github.com/hayato-n8810/hayalab/tree/main/src/MBquery_analysis/"
    click MB_CI "https://github.com/hayato-n8810/hayalab/blob/main/src/MBquery_analysis/compare_ids.py"
    click MB_FP "https://github.com/hayato-n8810/hayalab/blob/main/src/MBquery_analysis/find_matching_pattern.py"

    click UTIL "https://github.com/hayato-n8810/hayalab/tree/main/src/util/"
    click UTIL "https://github.com/hayato-n8810/hayalab/blob/main/src/util/clone.py"
    click UTIL "https://github.com/hayato-n8810/hayalab/blob/main/src/util/jsparser.js"
    click UTIL "https://github.com/hayato-n8810/hayalab/blob/main/src/util/mb_to_jscode.py"

    click CONST "https://github.com/hayato-n8810/hayalab/tree/main/src/const/"
    click CONST "https://github.com/hayato-n8810/hayalab/blob/main/src/const/__init__.py"
    click CONST "https://github.com/hayato-n8810/hayalab/blob/main/src/const/hayalab_path.py"

    click RS "https://github.com/hayato-n8810/hayalab/tree/main/results/"

    click ARCH "https://github.com/hayato-n8810/hayalab/tree/main/archive/Omori/"
    click ARCH "https://github.com/hayato-n8810/hayalab/tree/main/archive/Zenkoku/"

    %% Class assignments
    class CR,TR,AB,PD,MS,GR,MB service
    class DockerizedPipeline container
    class LR,RCS,AAST,ISP,BR,RS datastore
    class UTIL,CONST library
    class ARCH,NB notebook
```
