How hcb06/pyerualjetwork works
Project Type and Purpose
• This is a Python-based machine-learning library (formerly “anaplan”, now “pyerualjetwork”) designed to offer multiple training algorithms (PLAN, MLP/PTNN, ENE) with CPU and GPU (CUDA) backends.
• It provides data preprocessing, model construction, training, evaluation, visualization, serialization, and memory management utilities.
File Structure Analysis
• Top-level packaging under “pyerualjetwork-X.YY”:
• pyerualjetwork/nn.py, ene.py, model_ops.py, memory_ops.py, help.py, issue_solver.py, fitness_functions.py
• Subpackages cpu/ (activation_functions.py, data_ops.py, loss_functions.py, metrics.py, visualizations.py)
• Subpackages cuda/ (same modules as cpu/)
• Demo code under Welcome_to_* directories is illustrative only and not part of the shipped library.
• setup.py, setup.cfg, PKG-INFO files define packaging and dependencies.
Key Components (to represent
A. User Application Layer
• User’s script / Jupyter notebook importing “pyerualjetwork” modules.
B. API Layer (Core Modules)
• nn module: building and training neural-network architectures (PLAN, MLP, PTNN functions)
• ene module: genetic / neuroevolution routines
• model_ops module: save/load models, checkpointing, serialization
• memory_ops module: memory allocation, GPU/CPU memory management
• issue_solver module: compatibility checks and version-issue resolution
• help, fitness_functions modules: utilities and objective functions
C. Backend Layer
• cpu.data_ops, cpu.activation_functions, cpu.loss_functions, cpu.metrics, cpu.visualizations
• cuda.data_ops, cuda.activation_functions, cuda.loss_functions, cuda.metrics, cuda.visualizations
–– Each backend exports the same interface but routes compute to NumPy/SciPy (CPU) or CuPy (CUDA).
D. External Dependencies
• NumPy, pandas, networkx, scipy, tqdm, matplotlib, seaborn (optional), cupy-cuda, psutil
E. Hardware Layer
• CPU cores and GPU devices
Relationships and Data Flows
• User code calls high-level API (nn.fit, ene.run, etc.) → Core Module
• Core Module delegates tensor operations & graph construction to selected backend (CPU vs. CUDA) based on user choice or default.
• Backend layer executes numerical kernels via NumPy or CuPy and returns results.
• model_ops writes to disk (pickle, numpy .npy) and fetches models.
• memory_ops mediates memory allocation/frees on CPU/GPU.
• Visualizations modules extract metrics/history from core modules and render via matplotlib/networkx.
Architectural Patterns and Design Principles
• Layered architecture: clear separation between API, backend, and hardware.
• Strategy pattern for backend selection (CPU vs. CUDA).
• Modular design: each concern (data, metrics, activations, loss) lives in its own module/subpackage.
• Dependency injection: core modules accept data arrays and loss/activation functions interchangeably.
• Singleton or factory pattern for issue_solver to manage global compatibility checks.
Technologies and Frameworks
• Python packaging (setuptools) via setup.py / setup.cfg
• NumPy, pandas, SciPy for CPU compute
• CuPy for GPU compute
• Matplotlib, seaborn, networkx for visualizations
• pickle / .npy for model serialization
• tqdm for progress bars
• psutil for resource monitoring
Essential Elements to Capture
• Module boundaries: API vs. backend
• Data flow: training data → pipeline → compute → model artifacts → visualization
• Dependency graph: core modules depend on backend modules; backend modules depend on external libraries; core modules depend on model_ops/memory_ops.
• Support for eager vs. static execution (annotate plan_fit vs. learn functions).