How dhanunjaya-elluri/narwhals works
Main Components
a) Core Components:
• Input Layer (Native DataFrames)
• Translation Layer
• Narwhals API Layer
• Output Layer (Native DataFrames)
b) Supported Backend Systems:
• Full API Support Systems (cuDF, Modin, pandas, Polars, PyArrow)
• Lazy-only Systems (Dask)
• Interchange Protocol Systems (DuckDB, Ibis, Vaex)
┌─────────────────────────────────────────────────────────┐
• Client Applications │
• ┘
• ↕
┌─────────────────────────────────────────────────────────┐
• Input Interface │
• (narwhals.from_native converter) │
• ┘
• ↕
┌─────────────────────────────────────────────────────────┐
• Narwhals Core Layer │
• ┬───────────────────────────────┤
• Expression Engine │ Type System │
• ┼───────────────────────────────┤
• DataFrame Operations │ Series Operations │
• ┴───────────────────────────────┘
• ↕
┌─────────────────────────────────────────────────────────┐
• Backend Implementation │
• ┬───────────┬────────────┬───────────────────┤
• ┴───────────┴────────────┴───────────────────┘
• ↕
┌─────────────────────────────────────────────────────────┐
• Output Interface │
• (narwhals.to_native converter) │
• ┘
Important Elements to Highlight
a) Core Features:
• Zero-dependency architecture
• Lazy and eager evaluation paths
• Expression-based operations
• Type system compatibility
b) Integration Points:
• DataFrame Interchange Protocol support
• Backend-specific optimizations
• API translation mechanisms
Data Flow Representation
How data moves through the system:
Implementation Details
• Relationship between _arrow, _dask, _polars, etc. implementations
• How the stable API versioning works
• Relationship between DataFrame, Series, and Expression components