How unsw-ceem/nempy works
Project Type and Purpose
• Nempy is an open-source Python library for modelling the Australian National Electricity Market (NEM) dispatch procedure.
• It provides a flexible, layered framework where users can assemble and run market simulations of increasing complexity (e.g., adding ramp‐rate constraints, FCAS markets, interconnectors).
File Structure Analysis
• Top-level:
• .github/workflows/cicd.yml → CI pipeline (GitHub Actions)
• docs/ → Sphinx documentation source, build scripts, published PDFs
• benchmarking/ → performance analysis scripts and PDFs
• examples/ → standalone example scripts demonstrating API usage
• src/nempy/ → core library modules:
– help_functions/
– historical_inputs/ (AEMO data loaders, XML cache, MMS DB)
– spot_market_backend/ (constraint definitions, objective function, solver interface)
– markets.py & time_sequential.py (public API entry points)
• tests/ → pytest suites and test data
• build_test_data.py, pyproject.toml, pytest.ini → build and test configuration
Extracting Insights from README
• Public API: markets.SpotMarket + helper classes under nempy.historical_inputs
• Key workflows:
a. Load raw data (XML cache, SQLite)
b. Instantiate RawInputsLoader → UnitData, DemandData, ConstraintsData, InterconnectorData
c. Configure SpotMarket: set bids, constraints, interconnectors, demands
d. Dispatch runs via solver_interface → DataFrame outputs
e. Compare outputs to historical prices (benchmarking & examples)
Components
1. User/Client Code (examples, notebooks, CLI)
2. Public API Layer (markets.SpotMarket, TimeSequentialMarket)
3. Input Layer (nempy.historical_inputs):
– XMLCacheManager (downloads XML from AEMO)
– MMS DB Manager (SQLite interface)
– RawInputsLoader + data‐class wrappers (UnitData, DemandData, ConstraintData, InterconnectorData)
4. Core Modeling Layer (nempy.spot_market_backend):
– Constraint modules (unit, interconnector, FCAS, ramp‐rate, generic)
– Objective Function builder
– Dataframe Validator
– Variable ID manager
5. Solver Interface (solver_interface.py):
– Abstracts external MILP/QP solvers (e.g., PuLP, commercial solvers)
6. Data Persistence / Caching:
– XML cache on disk
– SQLite “market_management_system.db”
7. Output Layer:
– DataFrames: dispatch results, price results
– Benchmarking & analysis scripts
8. Documentation & CI:
– Sphinx docs, examples.rst, ReadTheDocs
– GitHub Actions pipeline → tests, lint
• User code calls Public API →
• Public API invokes RawInputsLoader →
• RawInputsLoader reads from XML cache & MMS DB →
• Builds pandas DataFrames for bids, constraints, demand →
• Public API passes these to Core Modeling Layer →
• Core Modeling Layer constructs MILP model using Solver Interface →
• Solver Interface calls external solver →
• Solver returns dispatch & price solution →
• API returns outputs to user code →
• Benchmarking scripts take outputs + historical DB data → analysis
c. Architectural Patterns & Principles
• Layered (Presentation → API → Domain/Model → Infrastructure)
• Separation of Concerns: data ingestion, model definition, solver abstraction, result analysis
• Plugin/Strategy Pattern: solver_interface allows swapping solvers; constraint modules follow common interface
• Modular design: historical_inputs modules decoupled from market engine
d. Key Technologies & Frameworks
• Python 3.x, pandas for tabular data
• SQLite (sqlite3) for historical MMS data
• XML parsing & local cache manager for AEMO data
• Sphinx (docs), pytest (tests), GitHub Actions (CI)
• Optional: pulp or other MILP solvers