How abhibambhaniya/genz-llm-analyzer works
Purpose and Entry Points
GenZ is a Python LLM-inference performance and memory estimator. It combines model architecture, serving workload, hardware characteristics, datatype, and TP/PP parallelism to estimate latency, memory, communication, and power. Primary programmatic inference entry points are in GenZ/LLM_inference/llm_prefill.py, GenZ/LLM_inference/llm_decode.py, and GenZ/LLM_inference/llm_chunked.py. An interactive Streamlit frontend is rooted at Website/Home.py.
Model Construction
Model definitions encode transformer, MoE, and Mamba-style structures and named model families. GenZ/Models/get_language_model.py resolves a requested model into an operator-level representation; GenZ/Models/default_models.py and family catalogs such as GenZ/Models/Model_sets/meta.py supply architecture parameters. Layer/operator implementations reside in GenZ/Models/attention.py, GenZ/Models/ffn.py, and GenZ/Models/mamba.py.
Inference Workload Stages
Prefill and token-by-token decode are modeled separately because their sequence lengths, batching behavior, memory pressure, and bottlenecks differ. GenZ/LLM_inference/llm_prefill.py models prompt processing, while GenZ/LLM_inference/llm_decode.py models generation; GenZ/LLM_inference/llm_spec_decode.py adds speculative decoding. GenZ/LLM_inference/llm_chunked.py composes chunked-prefill behavior across these stages.
Operator Analysis and Cost Model
The model graph is lowered to compute/memory operators, then evaluated against system limits. GenZ/operators.py and GenZ/operator_base.py define operator abstractions; GenZ/analyse_model.py aggregates their execution and memory analysis. GenZ/db.py, GenZ/unit.py, and GenZ/power.py provide performance data handling, unit conversion, and power-related calculations. This boundary separates model/workload semantics from hardware cost estimation.
Hardware and Parallel Execution
Hardware platforms are represented by system definitions and can be partitioned across tensor and pipeline parallel dimensions. GenZ/system.py defines system-level characteristics, Systems/system_configs.py provides platform configurations, and GenZ/parallelism.py applies device allocation and parallel execution logic. GenZ/collective_times.py models TP ring-allreduce and PP point-to-point communication, making interconnect behavior part of latency estimation.
Optimization and Planning
The estimator accepts serving-level choices such as datatype, batching, beam count, and parallelism. GenZ/LLM_inference/best_parallelization.py searches/selects favorable parallelization arrangements, while GenZ/LLM_inference/platform_size.py estimates required platform scale. GenZ/LLM_inference/utils.py acts as shared orchestration support across inference modes.
External Simulation and Delivery Boundaries
Analytic estimation can be complemented by simulator adapters: GenZ/Scale_Sim/get_scale_sim_time.py targets Scale-Sim, and GenZ/Astra_sim/get_astra_sim_time.py plus GenZ/Astra_sim/run.sh invoke Astra-sim using network/memory/system inputs. Astra-sim depends on the GenZ/aiconfigurator submodule. The packaged library is configured through pyproject.toml; the Streamlit UI (Website/Home.py and Website/pages/) consumes the same analysis capabilities rather than defining a separate backend.
Main entry points: llm_decode.py Decode estimator · analyse_model.py Execution and memory analysis · operators.py Operator abstractions · llm_prefill.py Prefill estimator
Components 19
Model construction
- Model family catalogs · Named families and default architecture parameters.
GenZ/Models/default_models.py - Model resolver · Resolves requested models into operator-level representations.
GenZ/Models/get_language_model.py - Transformer and Mamba layers
GenZ/Models/attention.py
Inference planning
- Prefill estimator · Models prompt processing.
GenZ/LLM_inference/llm_prefill.py - Decode estimator · Models token-by-token generation.
GenZ/LLM_inference/llm_decode.py - Speculative decode
GenZ/LLM_inference/llm_spec_decode.py - Chunked prefill · Composes chunked-prefill behavior across stages.
GenZ/LLM_inference/llm_chunked.py - Serving planner · Selects parallelism and estimates platform scale.
GenZ/LLM_inference/best_parallelization.py
Cost analysis
- Operator abstractions · Compute and memory operator definitions.
GenZ/operators.py - Execution and memory analysis · Aggregates operator execution and memory estimates.
GenZ/analyse_model.py - Performance, units and power
GenZ/db.py
Hardware execution
- System model · System-level hardware characteristics.
GenZ/system.py - Platform configurations
Systems/system_configs.py - Parallel execution · Applies device allocation and tensor/pipeline parallelism.
GenZ/parallelism.py - Collective communication · Models TP ring-allreduce and PP point-to-point time.
GenZ/collective_times.py
Delivery and simulation
- Streamlit UI · Consumes the shared analysis capabilities.
Website/Home.py - Scale-Sim adapter
GenZ/Scale_Sim/get_scale_sim_time.py - Astra-sim adapter · Invokes Astra-sim with network, memory, and system inputs.
GenZ/Astra_sim/get_astra_sim_time.py - AIConfigurator submodule
GenZ/aiconfigurator
Connections 27
- Model family catalogs → Model resolver: architecture parameters
- Model resolver → Transformer and Mamba layers: builds from
- Transformer and Mamba layers → Operator abstractions: lowers to
- Model resolver → Prefill estimator: model representation
- Model resolver → Decode estimator: model representation
- Prefill estimator → Chunked prefill: prompt stage
- Decode estimator → Chunked prefill: generation stage
- Decode estimator → Speculative decode: base generation model
- Prefill estimator → Operator abstractions: workload operators
- Decode estimator → Operator abstractions: workload operators
- Speculative decode → Operator abstractions: workload operators
- Chunked prefill → Operator abstractions: workload operators
- Operator abstractions → Execution and memory analysis: execution and memory costs
- Performance, units and power → Execution and memory analysis: performance and power data
- Platform configurations → System model: platform definitions
- System model → Parallel execution: hardware limits
- Parallel execution → Collective communication: TP/PP communication
- Parallel execution → Execution and memory analysis: parallel execution constraints
- Collective communication → Execution and memory analysis: communication latency
- Serving planner → Parallel execution: searches arrangements
- Prefill estimator → Serving planner: serving choices
- Decode estimator → Serving planner: serving choices
- Streamlit UI → Prefill estimator: invokes
- Streamlit UI → Decode estimator: invokes
- Execution and memory analysis → Scale-Sim adapter: optional validation
- Execution and memory analysis → Astra-sim adapter: optional validation
- AIConfigurator submodule → Astra-sim adapter: dependency