# molefe-m/deepsage architecture

> Architecture diagram of the GitHub repository molefe-m/deepsage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/molefe-m/deepsage
- Repository: https://github.com/molefe-m/deepsage

## Overview

##### Project type and purpose
- This is a command-line, multi-stage machine-learning pipeline (“deepSAGE”) for road-type classification on OSM road-network graphs.
- It stages: (1) graph extraction, (2) categorical feature embedding, (3) deep autoencoder embedding, (4) graph representation learning (RankSAGE/GraphSAGE/GCN/GAT) plus evaluation.

##### File structure
- Top-level scripts drive each stage:
- 01_graph_extraction.py
- 02_categorical_feature_representation.py
- 03_deep_auto_encoder_embeddings.py
- 04_graph_representation_learning.py
- A shared “common/” library holds reusable modules:
- graph_data_loader.py, osm_graph_extraction.py (data ingestion)
- categorical_feature_representation.py, node_structural_features_extraction.py
- graph_dae_embedding.py, models.py (DAE training & modeling)
- aggregators.py, graph_representation_learning.py (GRL algorithms)
- load_save_data.py (I/O utilities)
- Artifacts (datasets, trained models, performance reports) are written to disk paths defined inside each script.
- requirements.txt enumerates dependencies (e.g., PyTorch, PyTorch Geometric, scikit-learn, OSMnx).

##### README for insights
- Confirms the two-stage embedding pipeline (DAE → RankSAGE) and details parameter search and performance reporting.

##### Detail
- I/O vs. feature engineering vs. model training vs. evaluation
- Role of common/utils (load_save_data) as a shared resource.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Source
    OSM["OpenStreetMap API\n(OSMnx)"]:::external

    %% Graph Extraction Stage
    subgraph "Graph Extraction Stage"
        direction TB
        GE_Script["Graph Extraction\n(01_graph_extraction.py)"]:::process
        GE_Lib1["common/osm_graph_extraction.py"]:::library
        GE_Lib2["common/graph_data_loader.py"]:::library
        GE_IO["common/load_save_data.py"]:::library
    end

    %% Storage after Graph Extraction
    RawGraphs["Raw Graph Files\n(storage)"]:::storage

    %% Categorical Feature Representation Stage
    subgraph "Categorical Feature Representation Stage"
        direction TB
        CF_Script["Categorical Feature Representation\n(02_categorical_feature_representation.py)"]:::process
        CF_Lib1["common/categorical_feature_representation.py"]:::library
        CF_Lib2["common/node_structural_features_extraction.py"]:::library
        CF_Lib3["common/raw_feature_extraction.py"]:::library
        CF_IO["common/load_save_data.py"]:::library
    end

    %% Storage after Categorical Features
    FeatureGraphs["Feature-Enhanced Graph Files\n(storage)"]:::storage

    %% Deep AutoEncoder Embedding Stage
    subgraph "Deep AutoEncoder Embedding Stage"
        direction TB
        DAE_Script["Deep AutoEncoder Embedding\n(03_deep_auto_encoder_embeddings.py)"]:::process
        DAE_Lib1["common/graph_dae_embedding.py"]:::library
        DAE_Lib2["common/models.py"]:::library
        DAE_IO["common/load_save_data.py"]:::library
    end

    %% Storage after DAE
    DAEEmbeddings["DAE Embeddings\n(storage)"]:::storage

    %% Graph Representation Learning & Ranking Stage
    subgraph "Graph Representation Learning & Ranking Stage"
        direction TB
        GRL_Script["GRL & Ranking\n(04_graph_representation_learning.py)"]:::process
        GRL_Lib1["common/graph_representation_learning.py"]:::library
        GRL_Lib2["common/aggregators.py"]:::library
        GRL_IO["common/load_save_data.py"]:::library
    end

    %% Storage after GRL & Evaluation
    Reports["Performance Reports & Models\n(storage)"]:::storage

    %% Data Flow Arrows
    OSM -->|"download graph data"| GE_Script
    GE_Script -->|"write raw graphs"| RawGraphs
    RawGraphs -->|"load raw graphs"| CF_Script
    CF_Script -->|"write feature graphs"| FeatureGraphs
    FeatureGraphs -->|"load features"| DAE_Script
    DAE_Script -->|"write DAE embeddings"| DAEEmbeddings
    DAEEmbeddings -->|"load embeddings"| GRL_Script
    GRL_Script -->|"write models & reports"| Reports

    %% Library Imports (dotted)
    GE_Lib1 -.-> GE_Script
    GE_Lib2 -.-> GE_Script
    GE_IO -.-> GE_Script

    CF_Lib1 -.-> CF_Script
    CF_Lib2 -.-> CF_Script
    CF_Lib3 -.-> CF_Script
    CF_IO -.-> CF_Script

    DAE_Lib1 -.-> DAE_Script
    DAE_Lib2 -.-> DAE_Script
    DAE_IO -.-> DAE_Script

    GRL_Lib1 -.-> GRL_Script
    GRL_Lib2 -.-> GRL_Script
    GRL_IO -.-> GRL_Script

    %% Click Events
    click GE_Lib1 "https://github.com/molefe-m/deepsage/blob/main/common/osm_graph_extraction.py"
    click GE_Lib2 "https://github.com/molefe-m/deepsage/blob/main/common/graph_data_loader.py"
    click GE_IO "https://github.com/molefe-m/deepsage/blob/main/common/load_save_data.py"
    click GE_Script "https://github.com/molefe-m/deepsage/blob/main/01_graph_extraction.py"

    click CF_Script "https://github.com/molefe-m/deepsage/blob/main/02_categorical_feature_representation.py"
    click CF_Lib1 "https://github.com/molefe-m/deepsage/blob/main/common/categorical_feature_representation.py"
    click CF_Lib2 "https://github.com/molefe-m/deepsage/blob/main/common/node_structural_features_extraction.py"
    click CF_Lib3 "https://github.com/molefe-m/deepsage/blob/main/common/raw_feature_extraction.py"
    click CF_IO "https://github.com/molefe-m/deepsage/blob/main/common/load_save_data.py"

    click DAE_Script "https://github.com/molefe-m/deepsage/blob/main/03_deep_auto_encoder_embeddings.py"
    click DAE_Lib1 "https://github.com/molefe-m/deepsage/blob/main/common/graph_dae_embedding.py"
    click DAE_Lib2 "https://github.com/molefe-m/deepsage/blob/main/common/models.py"
    click DAE_IO "https://github.com/molefe-m/deepsage/blob/main/common/load_save_data.py"

    click GRL_Script "https://github.com/molefe-m/deepsage/blob/main/04_graph_representation_learning.py"
    click GRL_Lib1 "https://github.com/molefe-m/deepsage/blob/main/common/graph_representation_learning.py"
    click GRL_Lib2 "https://github.com/molefe-m/deepsage/blob/main/common/aggregators.py"
    click GRL_IO "https://github.com/molefe-m/deepsage/blob/main/common/load_save_data.py"

    %% Styles
    classDef process fill:#FFD700,stroke:#333,stroke-width:1px
    classDef storage fill:#87CEFA,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef external fill:#FFE4E1,stroke:#333,stroke-width:1px
    classDef library fill:#98FB98,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2
```
