# swagnercarena/paltas architecture

> Architecture diagram of the GitHub repository swagnercarena/paltas, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/swagnercarena/paltas
- Repository: https://github.com/swagnercarena/paltas

## Overview

`paltas` is a Python package with two major architectural halves:

1. **Simulation / dataset generation**
2. **Analysis / simulation-based inference**

The repository is organized around a modular “configuration → sampling → physical components → image generation → saved dataset” pipeline, with a separate “dataset → neural network / posterior estimation → hierarchical inference” analysis pipeline.

#### 1) Core purpose and top-level flow

The package generates realistic strong-lensing image datasets, primarily for **HST-like observations**, using `lenstronomy` as the lensing/image-simulation engine. A user supplies a Python config file, and `paltas/generate.py` turns that config into a batch of synthetic lenses. Those outputs are then consumed either as standalone datasets or by the analysis code in `paltas/Analysis` for training posterior estimators and doing population-level inference.

The important architectural boundary is:

- **Configuration layer** decides what distributions and component models to use.
- **Physical component layer** builds the lensing system pieces.
- **Generation layer** combines those pieces into simulated outputs.
- **Analysis layer** treats generated datasets as training/inference inputs.

#### 2) Simulation architecture

##### A. Configuration subsystem
Located under `paltas/Configs`.

This is the user-facing definition layer for simulations. Config files are plain Python modules that describe which distributions and component classes to use, and with what parameters.

Key files and roles:
- `paltas/Configs/config_handler.py`: the central loader/dispatcher for config files.
- `paltas/Configs/Examples/*`: example configs for common simulation setups.
- `paltas/Configs/Horton/*`: configs tied to specific HST/COSMOS-style setups.
- `paltas/Configs/paper_2203_00690/*`: experiment-specific configs used for a published study.

Architecturally, configs are not just constants; they are the glue that binds together the model subsystems. A config defines which:
- source model to use
- main deflector model to use
- substructure model to use
- line-of-sight halo model to use
- point source model to use
- sampling distributions to draw parameter values from

This makes the system highly modular and lets the same generation pipeline produce very different lens populations.

##### B. Sampling subsystem
Located under `paltas/Sampling`.

This subsystem provides the distribution machinery used by configs and component classes.

Key files:
- `distributions.py`: parameter distribution definitions
- `sampler.py`: sampling orchestration

Its role is to translate config-specified priors/distributions into concrete parameter draws for one simulated lens realization. It is the stochastic boundary of the simulation stack: everything below it consumes sampled parameters, not abstract priors.

##### C. Physical component subsystems
These are the model families that represent pieces of a strong-lensing system.

###### 1. Main deflector
Located in `paltas/MainDeflector`.

Files:
- `main_deflector_base.py`
- `simple_deflectors.py`

This subsystem represents the primary lens galaxy mass model, which bends light from the background source. It abstracts the main lens into a base interface plus concrete deflector implementations.

###### 2. Source galaxies
Located in `paltas/Sources`.

Files:
- `source_base.py`
- `sersic.py`
- `cosmos.py`
- `cosmos_sersic.py`
- `galaxy_catalog.py`

This subsystem defines the background source light profile. It supports both analytic models, such as Sersic-based sources, and catalog-driven galaxy models based on COSMOS-like data.

Important data assets here:
- `bad_galaxies.csv`
- `val_galaxies.csv`

Those catalogs indicate that source selection is partly data-driven and that there is a curated catalog boundary between generic source modeling and real-galaxy image material.

###### 3. Point sources
Located in `paltas/PointSource`.

Files:
- `point_source_base.py`
- `single_point_source.py`

This subsystem models compact emission components, such as a single point source. It is structurally separate from extended galaxy sources because its rendering and parameterization differ.

###### 4. Substructure
Located in `paltas/Substructure`.

Files:
- `los_base.py`
- `los_dg19.py`
- `subhalos_base.py`
- `subhalos_dg19.py`
- `nfw_functions.py`

This subsystem adds small-scale structure to the lensing mass distribution:
- **subhalos** internal to the main lens halo
- **line-of-sight halos** between observer and source

The base/implementation split shows that the repo is built for multiple halo population models, with `dg19` variants providing one concrete prescription. `nfw_functions.py` contains shared NFW-related computations used by these models.

###### 5. Utility functions
Located in `paltas/Utils`.

Files:
- `cosmology_utils.py`
- `hubble_utils.py`
- `lenstronomy_utils.py`
- `power_law.py`

These are cross-cutting helpers. They are not standalone subsystems but support the physical models and the integration with `lenstronomy`. `lenstronomy_utils.py` is the strongest architectural hint that the package delegates core ray-tracing / lens rendering to the external `lenstronomy` engine.

##### D. Generation entry point
Located in `paltas/generate.py`.

This is the main execution path for dataset creation. The README’s primary workflow is:

- load a config file
- sample a lens system
- assemble source/lens/substructure/point-source components
- render the final image
- write outputs to disk

This is the central orchestrator for the simulation half of the repository.

Although the file tree does not show its internals, the surrounding structure implies that it coordinates the config handler, sampler, component models, and rendering utilities.

#### 3) Analysis architecture

The analysis half lives in `paltas/Analysis` and is separated cleanly from simulation. Its role is to take generated datasets and run machine-learning-based inference on them.

##### A. Dataset preparation
Files:
- `dataset_generation.py`

This module likely converts generated images and metadata into model-ready training datasets. It forms the bridge between simulator output and ML training input.

##### B. Neural network models
Files:
- `conv_models.py`
- `transformer_models.py`

This subsystem contains alternative posterior-estimation architectures:
- convolutional models
- transformer-based models

The presence of both suggests the analysis layer is designed as a model zoo, not a single network implementation.

##### C. Training and loss
Files:
- `train_model.py`
- `loss_functions.py`

These provide the training loop and objective functions for neural posterior estimation. The training code consumes simulated data and model definitions, then optimizes posterior estimators.

##### D. Posterior and PDF utilities
Files:
- `posterior_functions.py`
- `pdf_functions.py`

These are the inference-side utilities for turning model outputs into usable probability distributions and posteriors.

##### E. Hierarchical inference
File:
- `hierarchical_inference.py`

This is a higher-level inference subsystem that operates on populations of lenses rather than individual systems. It is the repository’s second major analysis boundary:
- first-level inference: infer parameters per lens
- second-level inference: infer population-level hyperparameters from collections of inferred lenses

##### F. Analysis configs
Located in `paltas/Analysis/AnalysisConfigs/*`

These are experiment-specific training configurations for the analysis pipeline, analogous to the simulation configs in `paltas/Configs`. They define how a particular network or inference experiment should be run.

The split between `Configs` and `AnalysisConfigs` is important:
- `Configs` define synthetic universe generation
- `AnalysisConfigs` define learning/inference experiments on those synthetic universes

#### 4) Data flow across the repository

A useful way to diagram the system is as two connected pipelines.

##### Simulation data flow
1. **User config file**
2. `config_handler` loads and resolves configuration
3. `Sampling` draws parameter values from configured distributions
4. Component subsystems build:
   - main deflector mass model
   - source light profile
   - substructure population
   - point sources
5. `Utils` and `lenstronomy` integration turn physical parameters into images
6. `generate.py` writes simulated data and metadata to disk

This flow is modular: each component can be swapped independently through configuration.

##### Analysis data flow
1. **Generated datasets or saved lens images**
2. `dataset_generation.py` prepares training/inference data
3. Neural model from `conv_models.py` or `transformer_models.py`
4. `train_model.py` optimizes the network with `loss_functions.py`
5. `posterior_functions.py` and `pdf_functions.py` convert outputs to distributions
6. `hierarchical_inference.py` aggregates inference across a population

This flow is a consumer of the simulation pipeline and does not directly generate the physical lens images itself.

#### 5) External dependencies that matter architecturally

##### `lenstronomy`
This is the critical simulation backend. `paltas` wraps and extends it rather than replacing it. That means the physics/rendering boundary is externalized: `paltas` primarily defines the lens population, component models, and sampling logic, while `lenstronomy` handles the underlying lensing computations and image generation.

##### `tensorflow`
The analysis module depends on TensorFlow for neural posterior estimation. This is not part of the simulation runtime by default, only the ML/inference runtime.

##### Data files
The repo includes real-world assets that materially affect the architecture:
- COSMOS-related FITS files in `datasets/hst_psf/`
- HST PSF files
- source catalogs and galaxy images under `test/test_data/cosmos/`
- fake training data under `test/test_data/fake_train/`

These indicate that the system is not purely synthetic in isolation; it relies on real observational artifacts and curated catalog data to produce more realistic simulations and to support tests.

#### 6) Packaging, validation, and documentation boundaries

##### Tests
`test/` is structured by subsystem:
- `analysis_tests.py`
- `config_tests.py`
- `generate_tests.py`
- `main_deflector_tests.py`
- `point_source_tests.py`
- `sampling_tests.py`
- `source_tests.py`
- `substructure_tests.py`
- `utils_tests.py`

This mirrors the architecture and confirms the package is intentionally decomposed by domain boundary. `test/test_data/` contains fixture data used to validate catalog handling, config parsing, and generated dataset behavior.

##### Documentation and notebooks
- `docs/` contains the Sphinx documentation structure.
- `notebooks/` provides tutorials for config creation, pipeline understanding, training, and population analysis.

These are not core runtime subsystems, but they reflect the two main user journeys:
- simulation authoring
- inference/training workflow

#### 7) Suggested architecture diagram framing

If you are drawing the architecture, the clearest layout is:

- **Left side: configuration and sampling**
  - config files
  - config handler
  - distributions/sampler

- **Center: simulation components**
  - main deflector
  - source models
  - point sources
  - substructure
  - utilities / lenstronomy integration

- **Output boundary**
  - generated images
  - metadata / catalogs / saved arrays

- **Right side: analysis pipeline**
  - dataset preparation
  - neural network models
  - training/loss
  - posterior/PDF utilities
  - hierarchical inference

- **Supporting layer**
  - data assets: PSFs, COSMOS catalogs, test fixtures
  - docs, notebooks, tests, CI

The key architectural idea is that `paltas` is a **modular lens-simulation framework with a separate ML inference stack**, connected through generated datasets and experiment-specific configs.

Main entry points: [`paltas/generate.py`](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/generate.py) (Generator), [`paltas/Configs/config_handler.py`](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Configs/config_handler.py) (Config loader), [`paltas/Analysis/train_model.py`](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Analysis/train_model.py) (Training), [`paltas/Analysis/dataset_generation.py`](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Analysis/dataset_generation.py) (Dataset prep)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph "Simulation"
  cfg_handler["Config loader<br/>config orchestration<br/>[config_handler.py]"]
  sim_configs["Sim configs<br/>simulation presets"]
  sampler["Sampler<br/>parameter sampling<br/>[sampler.py]"]
  distributions["Distributions<br/>priors<br/>[distributions.py]"]
  main_deflector[("Main deflector<br/>lens mass model")]
  sources[("Sources<br/>source light models")]
  point_source(("Point source<br/>compact source model"))
  substructure[("Substructure<br/>halo populations")]
  generate["Generator<br/>dataset generation<br/>[generate.py]"]
end

subgraph "Analysis"
  dataset_prep["Dataset prep<br/>data pipeline"]
  conv_models["Conv nets<br/>posterior models<br/>[conv_models.py]"]
  transformers["Transformers<br/>posterior models"]
  train_model["Training<br/>optimization loop<br/>[train_model.py]"]
  losses["Losses<br/>objective functions<br/>[loss_functions.py]"]
  posteriors["Posteriors<br/>inference utils"]
  pdfs["PDF utils<br/>density utils<br/>[pdf_functions.py]"]
  hierarchical["Hierarchical inf.<br/>population inference"]
  analysis_configs["Analysis configs<br/>experiment configs"]
end

subgraph "Support"
  utils["Utilities<br/>physics helpers"]
  lenstronomy["Lenstronomy<br/>rendering backend"]
  data_assets[("Data assets<br/>reference data")]
  docs_tests["Docs & tests<br/>validation/docs"]
end

sim_configs -->|"loaded by"| cfg_handler
cfg_handler -->|"binds priors"| distributions
cfg_handler -->|"selects"| main_deflector
cfg_handler -->|"selects"| sources
cfg_handler -->|"selects"| point_source
cfg_handler -->|"selects"| substructure
sampler -->|"draws from"| distributions
generate -->|"loads config"| cfg_handler
generate -->|"samples system"| sampler
generate -->|"assembles lens"| main_deflector
generate -->|"assembles source"| sources
generate -->|"assembles point light"| point_source
generate -->|"adds halos"| substructure
generate -->|"uses helpers"| utils
utils -->|"bridges to"| lenstronomy
generate -->|"reads refs"| data_assets
dataset_prep -->|"consumes outputs"| generate
dataset_prep -->|"driven by"| analysis_configs
train_model -->|"trains"| conv_models
train_model -->|"trains"| transformers
train_model -->|"optimizes with"| losses
posteriors -->|"formats as"| pdfs
hierarchical -->|"aggregates"| posteriors
hierarchical -->|"consumes data"| dataset_prep
docs_tests -.->|"documents"| sim_configs
docs_tests -.->|"documents"| analysis_configs

click cfg_handler "https://github.com/swagnercarena/paltas/blob/main/paltas/Configs/config_handler.py"
click sim_configs "https://github.com/swagnercarena/paltas/tree/main/paltas/Configs"
click sampler "https://github.com/swagnercarena/paltas/blob/main/paltas/Sampling/sampler.py"
click distributions "https://github.com/swagnercarena/paltas/blob/main/paltas/Sampling/distributions.py"
click main_deflector "https://github.com/swagnercarena/paltas/tree/main/paltas/MainDeflector"
click sources "https://github.com/swagnercarena/paltas/tree/main/paltas/Sources"
click point_source "https://github.com/swagnercarena/paltas/tree/main/paltas/PointSource"
click substructure "https://github.com/swagnercarena/paltas/tree/main/paltas/Substructure"
click utils "https://github.com/swagnercarena/paltas/tree/main/paltas/Utils"
click generate "https://github.com/swagnercarena/paltas/blob/main/paltas/generate.py"
click dataset_prep "https://github.com/swagnercarena/paltas/blob/main/paltas/Analysis/dataset_generation.py"
click conv_models "https://github.com/swagnercarena/paltas/blob/main/paltas/Analysis/conv_models.py"
click transformers "https://github.com/swagnercarena/paltas/blob/main/paltas/Analysis/transformer_models.py"
click train_model "https://github.com/swagnercarena/paltas/blob/main/paltas/Analysis/train_model.py"
click losses "https://github.com/swagnercarena/paltas/blob/main/paltas/Analysis/loss_functions.py"
click posteriors "https://github.com/swagnercarena/paltas/blob/main/paltas/Analysis/posterior_functions.py"
click pdfs "https://github.com/swagnercarena/paltas/blob/main/paltas/Analysis/pdf_functions.py"
click hierarchical "https://github.com/swagnercarena/paltas/blob/main/paltas/Analysis/hierarchical_inference.py"
click analysis_configs "https://github.com/swagnercarena/paltas/tree/main/paltas/Analysis/AnalysisConfigs"
click data_assets "https://github.com/swagnercarena/paltas/tree/main/datasets"
click docs_tests "https://github.com/swagnercarena/paltas/tree/main/docs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class cfg_handler,sim_configs,sampler,distributions,main_deflector,sources,point_source,substructure,generate toneBlue
class dataset_prep,conv_models,transformers,train_model,losses,posteriors,pdfs,hierarchical,analysis_configs toneAmber
class utils,lenstronomy,data_assets,docs_tests toneMint
```

## Components

### Simulation

Configuration, sampling, physical components, and dataset generation.

- [Config loader](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Configs/config_handler.py): config orchestration · Loads and resolves simulation config modules into a runnable setup. · `paltas/Configs/config_handler.py`
- [Sim configs](https://github.com/swagnercarena/paltas/tree/HEAD/paltas/Configs): simulation presets · Example, Horton, and paper-specific configuration families for synthetic lens generation. · `paltas/Configs`
- [Sampler](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Sampling/sampler.py): parameter sampling · Draws concrete parameter values from config-defined distributions. · `paltas/Sampling/sampler.py`
- [Distributions](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Sampling/distributions.py): priors · Distribution definitions used to parameterize lens population sampling. · `paltas/Sampling/distributions.py`
- [Main deflector](https://github.com/swagnercarena/paltas/tree/HEAD/paltas/MainDeflector): lens mass model · Primary lens galaxy mass model and concrete deflector implementations. · `paltas/MainDeflector`
- [Sources](https://github.com/swagnercarena/paltas/tree/HEAD/paltas/Sources): source light models · Extended background source models including analytic and catalog-driven galaxies. · `paltas/Sources`
- [Point source](https://github.com/swagnercarena/paltas/tree/HEAD/paltas/PointSource): compact source model · Models compact emission components such as single point sources. · `paltas/PointSource`
- [Substructure](https://github.com/swagnercarena/paltas/tree/HEAD/paltas/Substructure): halo populations · Subhalos and line-of-sight halo models added to the main lens mass distribution. · `paltas/Substructure`
- [Generator](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/generate.py): dataset generation · Main execution entry point that assembles components and writes simulated outputs. · `paltas/generate.py`

### Analysis

Neural posterior estimation and hierarchical inference on generated datasets.

- [Dataset prep](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Analysis/dataset_generation.py): data pipeline · Converts generated images and metadata into training-ready analysis inputs. · `paltas/Analysis/dataset_generation.py`
- [Conv nets](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Analysis/conv_models.py): posterior models · Convolutional neural network architectures for inference. · `paltas/Analysis/conv_models.py`
- [Transformers](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Analysis/transformer_models.py): posterior models · Transformer-based inference architectures. · `paltas/Analysis/transformer_models.py`
- [Training](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Analysis/train_model.py): optimization loop · Trains inference models on simulated datasets using loss functions. · `paltas/Analysis/train_model.py`
- [Losses](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Analysis/loss_functions.py): objective functions · Training objectives for neural posterior estimation. · `paltas/Analysis/loss_functions.py`
- [Posteriors](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Analysis/posterior_functions.py): inference utils · Utilities for converting model outputs into posterior estimates. · `paltas/Analysis/posterior_functions.py`
- [PDF utils](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Analysis/pdf_functions.py): density utils · Probability-density helpers used by the analysis pipeline. · `paltas/Analysis/pdf_functions.py`
- [Hierarchical inf.](https://github.com/swagnercarena/paltas/blob/HEAD/paltas/Analysis/hierarchical_inference.py): population inference · Aggregates per-lens inference into population-level hyperparameter inference. · `paltas/Analysis/hierarchical_inference.py`
- [Analysis configs](https://github.com/swagnercarena/paltas/tree/HEAD/paltas/Analysis/AnalysisConfigs): experiment configs · Training and inference experiment definitions for the analysis stack. · `paltas/Analysis/AnalysisConfigs`

### Support

Shared utilities, reference data, docs, notebooks, and tests.

- [Utilities](https://github.com/swagnercarena/paltas/tree/HEAD/paltas/Utils): physics helpers · Cross-cutting cosmology, Hubble, power-law, and lenstronomy integration helpers. · `paltas/Utils`
- **Lenstronomy**: rendering backend · External lensing and image-simulation engine used by the package.
- [Data assets](https://github.com/swagnercarena/paltas/tree/HEAD/datasets): reference data · PSFs, COSMOS catalogs, and fixture datasets used by simulation and tests. · `datasets`
- [Docs & tests](https://github.com/swagnercarena/paltas/tree/HEAD/docs): validation/docs · Documentation, notebooks, and subsystem-aligned tests covering the package architecture. · `docs`

## Connections

- Sim configs → Config loader: loaded by · Configuration modules are resolved through the config handler.
- Config loader → Distributions: binds priors · Config resolution connects model choices to sampling distributions.
- Config loader → Main deflector: selects · Configs choose the main lens mass model.
- Config loader → Sources: selects · Configs choose the source light model.
- Config loader → Point source: selects · Configs may include compact source components.
- Config loader → Substructure: selects · Configs choose halo and line-of-sight structure models.
- Sampler → Distributions: draws from · Sampler turns priors into concrete parameter values.
- Generator → Config loader: loads config · Generation begins by reading the user-supplied config.
- Generator → Sampler: samples system · Generator requests a parameter draw for each realization.
- Generator → Main deflector: assembles lens · Generator composes the primary lens model into the scene.
- Generator → Sources: assembles source · Generator composes the background source model into the scene.
- Generator → Point source: assembles point light · Generator adds compact emission where configured.
- Generator → Substructure: adds halos · Generator includes substructure and line-of-sight perturbations.
- Generator → Utilities: uses helpers · Generator relies on shared physics and integration utilities.
- Utilities → Lenstronomy: bridges to · Utilities adapt repository models to the external simulation backend.
- Generator → Data assets: reads refs · Simulation can depend on PSFs and catalog assets.
- Dataset prep → Generator: consumes outputs · Analysis preparation starts from saved simulated datasets.
- Dataset prep → Analysis configs: driven by · Analysis experiments are defined by separate config modules.
- Training → Conv nets: trains · Training loop can optimize convolutional inference models.
- Training → Transformers: trains · Training loop can optimize transformer inference models.
- Training → Losses: optimizes with · Training uses configured objective functions.
- Posteriors → PDF utils: formats as · Posterior utilities build density-like outputs.
- Hierarchical inf. → Posteriors: aggregates · Population inference consumes per-lens posterior products.
- Hierarchical inf. → Dataset prep: consumes data · Hierarchical inference operates on prepared datasets and inference outputs.
- Docs & tests → Sim configs: documents · Documentation and tests mirror the simulation configuration layer.
- Docs & tests → Analysis configs: documents · Documentation and tests mirror the analysis configuration layer.
