# unsw-ceem/nempy architecture

> Architecture diagram of the GitHub repository unsw-ceem/nempy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/unsw-ceem/nempy
- Repository: https://github.com/unsw-ceem/nempy

## Overview

##### Project Type and Purpose
- Nempy is an open-source Python library for modelling the Australian National Electricity Market (NEM) dispatch procedure.
- It provides a flexible, layered framework where users can assemble and run market simulations of increasing complexity (e.g., adding ramp‐rate constraints, FCAS markets, interconnectors).

##### File Structure Analysis
- Top-level:
- .github/workflows/cicd.yml → CI pipeline (GitHub Actions)
- docs/ → Sphinx documentation source, build scripts, published PDFs
- benchmarking/ → performance analysis scripts and PDFs
- examples/ → standalone example scripts demonstrating API usage
- src/nempy/ → core library modules:
– help_functions/
– historical_inputs/ (AEMO data loaders, XML cache, MMS DB)
– spot_market_backend/ (constraint definitions, objective function, solver interface)
– markets.py & time_sequential.py (public API entry points)
- tests/ → pytest suites and test data
- build_test_data.py, pyproject.toml, pytest.ini → build and test configuration

##### Extracting Insights from README
- Public API: markets.SpotMarket + helper classes under nempy.historical_inputs
- Key workflows:
a. Load raw data (XML cache, SQLite)
b. Instantiate RawInputsLoader → UnitData, DemandData, ConstraintsData, InterconnectorData
c. Configure SpotMarket: set bids, constraints, interconnectors, demands
d. Dispatch runs via solver_interface → DataFrame outputs
e. Compare outputs to historical prices (benchmarking & examples)

##### Components
1. User/Client Code (examples, notebooks, CLI)
2. Public API Layer (markets.SpotMarket, TimeSequentialMarket)
3. Input Layer (nempy.historical_inputs):
– XMLCacheManager (downloads XML from AEMO)
– MMS DB Manager (SQLite interface)
– RawInputsLoader + data‐class wrappers (UnitData, DemandData, ConstraintData, InterconnectorData)
4. Core Modeling Layer (nempy.spot_market_backend):
– Constraint modules (unit, interconnector, FCAS, ramp‐rate, generic)
– Objective Function builder
– Dataframe Validator
– Variable ID manager
5. Solver Interface (solver_interface.py):
– Abstracts external MILP/QP solvers (e.g., PuLP, commercial solvers)
6. Data Persistence / Caching:
– XML cache on disk
– SQLite “market_management_system.db”
7. Output Layer:
– DataFrames: dispatch results, price results
– Benchmarking & analysis scripts
8. Documentation & CI:
– Sphinx docs, examples.rst, ReadTheDocs
– GitHub Actions pipeline → tests, lint

- User code calls Public API →
- Public API invokes RawInputsLoader →
- RawInputsLoader reads from XML cache & MMS DB →
- Builds pandas DataFrames for bids, constraints, demand →
- Public API passes these to Core Modeling Layer →
- Core Modeling Layer constructs MILP model using Solver Interface →
- Solver Interface calls external solver →
- Solver returns dispatch & price solution →
- API returns outputs to user code →
- Benchmarking scripts take outputs + historical DB data → analysis

c. Architectural Patterns & Principles
- Layered (Presentation → API → Domain/Model → Infrastructure)
- Separation of Concerns: data ingestion, model definition, solver abstraction, result analysis
- Plugin/Strategy Pattern: solver_interface allows swapping solvers; constraint modules follow common interface
- Modular design: historical_inputs modules decoupled from market engine

d. Key Technologies & Frameworks
- Python 3.x, pandas for tabular data
- SQLite (sqlite3) for historical MMS data
- XML parsing & local cache manager for AEMO data
- Sphinx (docs), pytest (tests), GitHub Actions (CI)
- Optional: pulp or other MILP solvers

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Examples Layer
    subgraph "Examples & Client Code"
        UC["User/Client Code (examples, notebooks, CLI)"]:::example
    end

    %% Public/API Layer
    subgraph "Public/API Layer"
        SM["SpotMarket API"]:::api
        TS["TimeSequentialMarket API"]:::api
    end

    %% Data Ingestion Layer
    subgraph "Input Layer (historical_inputs)"
        XML["XMLCacheManager"]:::data
        MMS["MMS DB Manager"]:::data
        RawLoader["RawInputsLoader"]:::data
        UnitData["UnitData Wrapper"]:::data
        DemandData["DemandData Wrapper"]:::data
        ConstraintsData["ConstraintsData Wrapper"]:::data
        InterconnectorData["InterconnectorData Wrapper"]:::data
        AemoMap["AEMO Name Mapping"]:::data
        RHS["RHS Calculator"]:::data
    end

    %% Core Modeling Layer
    subgraph "Core Modeling Layer (spot_market_backend)"
        UCst["Unit Constraint Module"]:::core
        ICst["Interconnector Constraint Module"]:::core
        FCAS["FCAS Constraint Module"]:::core
        Ramp["Ramp‐Rate Constraint Module"]:::core
        Elastic["Elastic/Generic Constraint Module"]:::core
        MktCst["Market‐Wide Constraints"]:::core
        Obj["Objective Function Builder"]:::core
        Validator["DataFrame Validator"]:::core
        VarID["Variable ID Manager"]:::core
    end

    %% Solver Interface Layer
    subgraph "Solver Interface"
        SolverIF["Solver Abstraction Layer"]:::infra
    end

    %% Infrastructure Layer
    subgraph "Data Persistence / Caching"
        DB["SQLite DB (market_management_system.db)"]:::infra
    end

    %% Output Layer
    subgraph "Output Layer"
        Benchmark["Benchmarking & Analysis Scripts"]:::example
        Cache["Example Outputs Cache"]:::example
    end

    %% Documentation & CI
    subgraph "Documentation & CI"
        DocsSrc["Sphinx Docs Source"]:::example
        PDFs["Built PDFs"]:::example
        CI["CI Pipeline"]:::example
    end

    %% Connections
    UC -->|"calls API"| SM
    UC --> SM
    UC --> TS
    SM -->|uses| RawLoader
    TS -->|uses| RawLoader
    RawLoader --> XML
    RawLoader --> MMS
    RawLoader --> UnitData
    RawLoader --> DemandData
    RawLoader --> ConstraintsData
    RawLoader --> InterconnectorData
    RawLoader --> AemoMap
    RawLoader --> RHS
    SM --> UCst
    SM --> ICst
    SM --> FCAS
    SM --> Ramp
    SM --> Elastic
    SM --> MktCst
    SM --> Obj
    SM --> Validator
    SM --> VarID
    TS --> UCst
    TS --> ICst
    TS --> FCAS
    TS --> Ramp
    TS --> Elastic
    TS --> MktCst
    TS --> Obj
    TS --> Validator
    TS --> VarID
    UCst --> SolverIF
    ICst --> SolverIF
    FCAS --> SolverIF
    Ramp --> SolverIF
    Elastic --> SolverIF
    MktCst --> SolverIF
    Obj --> SolverIF
    Validator --> SolverIF
    VarID --> SolverIF
    SolverIF -->|external solver calls| DB
    SolverIF -->|returns results| SM
    SM -->|outputs DataFrame| UC
    Benchmark --> SM
    Benchmark --> DB
    UC --> Cache
    DocsSrc --> PDFs
    CI -->|runs tests & lint| SM

    %% Click Events
    click UC "https://github.com/unsw-ceem/nempy/tree/master/examples/"
    click SM "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/markets.py"
    click TS "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/time_sequential.py"
    click XML "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/historical_inputs/xml_cache.py"
    click MMS "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/historical_inputs/mms_db.py"
    click RawLoader "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/historical_inputs/loaders.py"
    click UnitData "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/historical_inputs/units.py"
    click DemandData "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/historical_inputs/demand.py"
    click ConstraintsData "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/historical_inputs/constraints.py"
    click InterconnectorData "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/historical_inputs/interconnectors.py"
    click AemoMap "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/historical_inputs/aemo_to_nempy_name_mapping.py"
    click RHS "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/historical_inputs/rhs_calculator.py"
    click UCst "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/spot_market_backend/unit_constraints.py"
    click ICst "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/spot_market_backend/interconnectors.py"
    click FCAS "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/spot_market_backend/fcas_constraints.py"
    click Ramp "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/spot_market_backend/ramp_rate_processing.py"
    click Elastic "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/spot_market_backend/elastic_constraints.py"
    click MktCst "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/spot_market_backend/market_constraints.py"
    click Obj "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/spot_market_backend/objective_function.py"
    click Validator "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/spot_market_backend/dataframe_validator.py"
    click VarID "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/spot_market_backend/variable_ids.py"
    click SolverIF "https://github.com/unsw-ceem/nempy/blob/master/src/nempy/spot_market_backend/solver_interface.py"
    click DB "https://github.com/unsw-ceem/nempy/blob/master/market_management_system.db"
    click Benchmark "https://github.com/unsw-ceem/nempy/tree/master/benchmarking/"
    click Cache "https://github.com/unsw-ceem/nempy/tree/master/examples/nemde_cache/"
    click DocsSrc "https://github.com/unsw-ceem/nempy/tree/master/docs/source/"
    click PDFs "https://github.com/unsw-ceem/nempy/tree/master/docs/pdfs/"
    click CI "https://github.com/unsw-ceem/nempy/blob/master/.github/workflows/cicd.yml"

    %% Styles
    classDef api fill:#a8e6cf,stroke:#2e7d32,color:#000
    classDef data fill:#d0e1f9,stroke:#1565c0,color:#000
    classDef core fill:#fff59d,stroke:#fbc02d,color:#000
    classDef infra fill:#ffd180,stroke:#ef6c00,color:#000
    classDef example fill:#e0e0e0,stroke:#616161,color:#000
```
