How quansight-labs/ndindex works
Project type and purpose
• ndindex is an open-source Python library (no frontend / backend web layers) that provides a uniform API for representing and manipulating numpy array indices (slices, integers, ellipses, boolean/integer arrays, tuples).
• It focuses on correctness (full numpy semantics) and performance (via Cython modules), with supporting tooling for documentation (Sphinx), testing (pytest), and benchmarking (asv).
File structure
• Top‐level directories and files
• ndindex/ ← core Python modules and Cython .pyx files
• ndindex/tests/ ← pytest test suite
• benchmarks/ ← asv benchmarking scripts
• docs/ ← Sphinx documentation sources and static assets
• .github/workflows/ ← CI/CD (GitHub Actions) definitions
• setup.py, pyproject.toml, setup.cfg ← build & packaging
• versioneer.py ← version management
• Key patterns
• Modular design: each index type (slice, tuple, etc.) is a separate module
• Cython acceleration: _slice.pyx, _tuple.pyx, _crt.py compile to C extensions
• Conventional Python package layout with tests and docs in sibling dirs
• Tooling/config
• pytest.ini, conftest.py for tests
• asv.conf.json for benchmarks
• Sphinx conf.py, Makefile for docs
• GitHub Actions workflows for test, docs preview, package publish
README for additional insights
• Emphasizes uniform API, correct numpy semantics, transformation utilities
• Points to online docs site
• No runtime services or external APIs—pure library
Main components and interactions
a. Components
1. API Layer (ndindex/*.py)
2. Core Cython Layer (ndindex/*.pyx & _crt.py → compiled extension modules)
3. Test Suite (ndindex/tests + pytest config)
4. Benchmark Suite (benchmarks/ + asv)
5. Documentation (docs/, Sphinx)
6. Build & Packaging (setup.py, pyproject.toml, versioneer)
7. CI/CD (GitHub Actions workflows)
8. External Dependencies (numpy, C compiler, Python)
b. Relationships & data flows
• Developer commits → CI triggers tests, style checks, type checks
• Build step compiles Cython modules → generates .so extension
• Package step bundles Python code + compiled extensions → publish to PyPI
• Docs build step pulls code + doc sources → renders HTML and deploys to docs site
• Benchmark step runs asv against compiled library → records performance metrics
c. Architectural patterns & principles
• Layered architecture: high-level Python API over low-level Cython primitives
• Separation of concerns: pure code vs. tests vs. docs vs. benchmarks
• GitOps/CI-driven delivery (GitHub Actions)
d. Key technologies
• Python 3.x, Cython, numpy
• pytest, asv
• Sphinx (reStructuredText, Makefile)
• GitHub Actions for CI/CD
• versioneer for automatic versioning
Elements
• Clear labels for each component (e.g., “API Layer”, “Cython Extensions”, “Test Suite”)
• Code → Build
• Build → Package → Publish
• Code → Docs Builder → Docs Site
• Code → Test Runner → Test Reports
• Code → Benchmark Runner → Metrics Store
• Notes or annotations for critical details (e.g., which modules are .pyx vs. .py)
Detailed capture of essential elements
• “ndindex/*.py” (API Layer)
• “ndindex/*.pyx, _crt.py” (Cython Layer)
• “ndindex/tests/” (Test Suite)
• “benchmarks/” (Benchmark Suite)
• “docs/” (Documentation Sources)
• “setup.py & versioneer.py” (Build & Packaging)