# roche/pyreadstat architecture

> Architecture diagram of the GitHub repository roche/pyreadstat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/roche/pyreadstat
- Repository: https://github.com/roche/pyreadstat

## Overview

##### Main Components and Architecture

The system should be represented as a layered architecture with the following main components:

a) Python Interface Layer:
- Entry point through pyreadstat's public functions (read_sas7bdat, read_xport, write_sav, etc.)
- Located in pyreadstat/__init__.py
- Handles high-level operations and parameter validation

b) Cython Bridge Layer:
- Files: _readstat_parser.pyx, _readstat_writer.pyx, pyreadstat.pyx
- Manages the interface between Python and C code
- Handles data type conversions and memory management

c) ReadStat C Core:
- Located in the src/ directory
- Organized into format-specific modules (sas/, spss/, stata/)
- Handles actual file parsing and writing

d) Utility Layer:
- Worker module for multiprocessing (worker.py)
- Python utility functions (pyfunctions.py)

##### Data Flow Representation

a) Reading Flow:
**Python DataFrame/Metadata ← Python Interface ← Cython Bridge ← ReadStat Core ← Statistical Data File**

b) Writing Flow:
**Python DataFrame → Python Interface → Cython Bridge → ReadStat Core → Statistical Data File**

##### Component Details

a) Python Interface:
- Public API functions
- Metadata handling
- DataFrame construction
- Multiprocessing coordination

b) Cython Bridge:
- Memory management
- Type conversions
- Error handling
- C function wrapping

c) ReadStat Core:
- File format parsers (SAS, SPSS, Stata)
- Compression handling
- Character encoding
- Value label processing

##### Special Features to Highlight

- Multiprocessing capability
- Chunk-based reading
- Value label handling
- Missing value processing
- Date/time conversions

##### External Dependencies

Connections to:
- Pandas (for DataFrame handling)
- Python standard library (multiprocessing)
- System libraries (iconv, zlib)

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef python fill:#3572A5,color:white
    classDef cython fill:#FED049,color:black
    classDef c fill:#90EE90,color:black
    classDef external fill:#D3D3D3,color:black
    classDef util fill:#FF69B4,color:white

    %% Legend
    subgraph Legend
        P[Python Component]:::python
        CY[Cython Component]:::cython
        C[C Component]:::c
        E[External Component]:::external
        U[Utility Component]:::util
    end

    %% Main Layers
    subgraph Python_Interface
        API[Public API Functions]:::python
        Meta[Metadata Handling]:::python
        DF[DataFrame Construction]:::python
        MP[Multiprocessing Coord]:::python
    end

    subgraph Cython_Bridge
        Parser[Parser Bridge]:::cython
        Writer[Writer Bridge]:::cython
        TypeConv[Type Conversions]:::cython
        MemMgmt[Memory Management]:::cython
    end

    subgraph ReadStat_Core
        CoreMain[ReadStat Main]:::c
        SAS[SAS Handler]:::c
        SPSS[SPSS Handler]:::c
        Stata[Stata Handler]:::c
        Convert[Data Conversion]:::c
        Error[Error Handling]:::c
    end

    subgraph Utilities
        Worker[Worker Module]:::util
        PyFunc[Python Functions]:::util
    end

    %% External Components
    Pandas[Pandas Library]:::external
    Files[Statistical Data Files]:::external
    SystemLibs[System Libraries]:::external

    %% Relationships
    Files --> CoreMain
    CoreMain --> SAS & SPSS & Stata
    SAS & SPSS & Stata --> Convert
    Convert --> Error
    CoreMain --> Parser
    Parser --> TypeConv
    TypeConv --> API
    API --> Meta
    Meta --> DF
    DF --> Pandas
    MP --> Worker
    Worker --> PyFunc
    API --> Writer
    Writer --> CoreMain
    SystemLibs --> CoreMain

    %% Click Events
    click API "https://github.com/Roche/pyreadstat/blob/master/pyreadstat/__init__.py"
    click Parser "https://github.com/Roche/pyreadstat/blob/master/pyreadstat/_readstat_parser.pyx"
    click Writer "https://github.com/Roche/pyreadstat/blob/master/pyreadstat/_readstat_writer.pyx"
    click CoreMain "https://github.com/Roche/pyreadstat/blob/master/src/readstat.h"
    click SAS "https://github.com/Roche/pyreadstat/blob/master/src/sas/readstat_sas.h"
    click SPSS "https://github.com/Roche/pyreadstat/blob/master/src/spss/readstat_spss.h"
    click Stata "https://github.com/Roche/pyreadstat/blob/master/src/stata/readstat_dta.h"
    click Worker "https://github.com/Roche/pyreadstat/blob/master/pyreadstat/worker.py"
    click PyFunc "https://github.com/Roche/pyreadstat/blob/master/pyreadstat/pyfunctions.py"
    click Error "https://github.com/Roche/pyreadstat/blob/master/src/readstat_error.c"
    click Convert "https://github.com/Roche/pyreadstat/blob/master/src/readstat_convert.c"
    click MemMgmt "https://github.com/Roche/pyreadstat/blob/master/src/readstat_malloc.h"
```
