How roche/pyreadstat works
Main Components and Architecture
The system should be represented as a layered architecture with the following main components:
a) Python Interface Layer:
• Entry point through pyreadstat's public functions (read_sas7bdat, read_xport, write_sav, etc.)
• Located in pyreadstat/__init__.py
• Handles high-level operations and parameter validation
b) Cython Bridge Layer:
• Files: _readstat_parser.pyx, _readstat_writer.pyx, pyreadstat.pyx
• Manages the interface between Python and C code
• Handles data type conversions and memory management
c) ReadStat C Core:
• Located in the src/ directory
• Organized into format-specific modules (sas/, spss/, stata/)
• Handles actual file parsing and writing
d) Utility Layer:
• Worker module for multiprocessing (worker.py)
• Python utility functions (pyfunctions.py)
Data Flow Representation
a) Reading Flow:
Python DataFrame/Metadata ← Python Interface ← Cython Bridge ← ReadStat Core ← Statistical Data File
b) Writing Flow:
Python DataFrame → Python Interface → Cython Bridge → ReadStat Core → Statistical Data File
Component Details
a) Python Interface:
• Public API functions
• Metadata handling
• DataFrame construction
• Multiprocessing coordination
b) Cython Bridge:
• Memory management
• Type conversions
• Error handling
• C function wrapping
c) ReadStat Core:
• File format parsers (SAS, SPSS, Stata)
• Compression handling
• Character encoding
• Value label processing
Special Features to Highlight
• Multiprocessing capability
• Chunk-based reading
• Value label handling
• Missing value processing
• Date/time conversions
External Dependencies
Connections to:
• Pandas (for DataFrame handling)
• Python standard library (multiprocessing)
• System libraries (iconv, zlib)