How pahulmeet/sansa-python-wrapper works
Main Components
• User Interaction Layer: This includes the Jupyter Notebook environment (as found in the ml_notebook, rdf, and query example directories) where the user executes notebooks and directly calls the pysansa library.
• pysansa Package: This is the core of the project and is subdivided into several layers:
• RDF Layer (pysansa/rdf): Responsible for reading, parsing, and processing RDF data.
• Query Layer (pysansa/query): Provides methods to perform SPARQL queries on RDF datasets.
• ML Layer (pysansa/ml): Contains functionality for machine learning operations on the semantic data.
• myjars: This indicates the inclusion of external JAR files (Java dependencies) that interface with the SANSA-Stack.
• Data Sources: The ‘data’ folder holds RDF data files such as rdf.nt, alternative_bigger_rdf.nt, and movie.nt.
• External Dependencies/Infrastructure:
• Apache Spark: Used for distributed processing. The README details the Spark installation along with starting a Spark master and worker. This component should be shown as a cluster component.
• Hadoop: Configured as the underlying distributed file system and processing engine.
• Java / Maven / Scala: These underpin the Java-based SANSA-Stack libraries. Although they are part of the installation process, they are also part of the external tech stack that the pysansa jobs rely upon.
• SANSA JAR Library: Downloaded externally and used by the pysansa modules (especially the RDF layer) via spark-submit.
Relationships and Interactions
• The user (via a Jupyter notebook) sends commands, invoking methods from the pysansa package.
• The pysansa package, through its distinct layers, abstracts the complexities of parsing RDF data (RDF Layer), executing queries (Query Layer), and performing machine learning tasks (ML Layer).
• The underlying Spark cluster processes the requests initiated by pysansa (for example, running distributed SPARQL queries or ML algorithms).
• Hadoop acts as the file system for storing large RDF datasets and facilitating data exchange.
• External JAR libraries (from the ‘myjars’ directory) are called by the Python wrapper when interacting with the Java-based SANSA components.
Architectural Patterns and Design Principles
• Layered Architecture: The code is segmented into distinct logical layers corresponding to data ingestion (RDF), querying (Query), and data science (ML) operations.
• Wrapper/Facade Pattern: pysansa functions as a façade that simplifies the interaction with complex underlying systems (i.e., SANSA-Stack, Spark, and Hadoop).
• Separation of Concerns: The clear separation of notebooks, pysansa modules, and external configuration files indicates that the system design emphasizes modularity.