How aditya-grover/node2vec works
Project type and purpose
• This is an *open-source machine-learning tool* that implements the node2vec algorithm in two variants:
a. A Python CLI version (src/main.py + node2vec.py)
b. A distributed Scala/Spark version (node2vec_spark/)
• Purpose: consume an edgelist graph, generate biased random walks, train a skip-gram (Word2Vec) model, and emit node embeddings.
Main components
• CLI/Front-end
• Python entry point: src/main.py parses flags, invokes node2vec.py APIs
• Scala entry point: Main.scala reads arguments (via AbstractParams), sets up SparkContext
• Graph Loader & Processor
• Python: node2vec.py loads edgelist, builds in-memory graph
• Spark: GraphOps.scala loads edges into RDDs, constructs distributed graph representations
• Random Walk Generator
• Node2vec.scala (Scala) / functions in node2vec.py: implements biased second-order random walks
• Skip-Gram Trainer
• Word2vec.scala (Scala) / Word2vec module in Python: trains continuous embeddings on sequences
• Data Layer
• Input: graph/karate.edgelist
• Output: emb/karate.emb
• Configuration files: log4j2.properties, requirements.txt, pom.xml
• Build & Deployment
• Python: pip installs via requirements.txt
• Scala: Maven project (pom.xml) produces a JAR runnable on Spark cluster
• External Dependencies
• Python libraries: numpy, networkx (optional), gensim or pure Python
• Spark Core, Spark RDD APIs, log4j for logging
Relationships & Data Flows
• User invokes CLI → Parser → Graph Loader
• Graph Loader → Random Walk Generator (produces sequences of node IDs)
• Random Walks → Skip-Gram Trainer → Embedding vectors
• Trainer → persisting vectors to Output file
• In Spark path: data flows over RDD transformations between GraphOps → Node2vec → Word2vec
Architectural patterns & principles
• Modular design: clear separation between graph handling, sampling, and training
• Distributed data processing (Scala path): leverages RDD lineage, transformations, and actions
• CLI‐driven configurability: both implementations read flags/params for walk length, p/q, dimensions, etc.
• Extensibility points:
• Adding new sampling strategies in Node2vec.scala/node2vec.py
• Swapping in a different embedding trainer
Essential details
• Optionally number the steps 1–5 along the flow