How jamalsz/qepubunts works
Project Type and Purpose
• This is a research‐oriented Python codebase implementing experiments from the SIGKDD 2024 paper “Quantifying and Estimating the Predictability Upper Bound of Univariate Numeric Time Series.”
• It is not a typical client‐server or web application but a data‐processing and experimentation pipeline.
• Key goals:
• Generate synthetic Markov time series
• Estimate entropy and predictability upper bounds (PIMAX) using various algorithms (NLZ1, NLZ2, exact)
• Run forecasting models (AR1, ARIMA, LSTM, CNN‐LSTM) on real and synthetic data
• Produce figures for challenges (Figure 2) and Experiments 1–3
File Structure Analysis
• Top Level
• README.md, requirements.txt
• Datasets/ – raw and synthetic data sources
• Utils/ – core algorithms (entropy estimators, PIMAX computation, data generator)
• Challenges_section/ – code to produce Figure 2 for the “Challenges” section
• Experiments/ – three subfolders Exp1, Exp2, Exp3, each containing scripts, model definitions, plotting routines, and Results/
• Patterns
• Reusable utility modules in Utils
• Data flow: Datasets → Utils (generation/estimation) → Experiments (modeling) → Results/
• Each experiment subfolder has “execute scripts” (e.g., pimax.py, ar1.py, markov_predict.py) followed by plot_*.py and a Results directory
Insights from README
• High-level workflow descriptions for Figure 2, Experiment 1 (pipelines: PIMAX → AR1 → Markov → plot), Experiment 2 (two figure scripts + combined plot), and Experiment 3 (per-dataset loops of PIMAX, ARIMA, LSTM, CNN-LSTM + plotting).
• Dependencies: Python + libraries in requirements.txt.
Components
a. Data Sources
• Real-world CSVs (temperature.csv, ETTh1.csv, Stock_Open.csv)
• Synthetic Markov CSVs (Datasets/Markov/Exp1 & Exp2)
b. Utility Layer
• gen_data.py (synthetic generator)
• computeH_exact.py (exact entropy)
• compute_PIMax.py (predictability bound)
• LZ1.py, LZ2.py (NLZ entropy estimators)
c. Experiment Pipelines
• Challenges_section/fig2.py → Challenges_section/Results
• Experiments/Exp1
• pimax.py → ar1.py → markov_predict.py → plot_exp1.py → Exp1/Results
• Experiments/Exp2
• exp2_fig4a.py & exp2_fig4b.py → plot_exp2_fig4a_4b.py → Exp2/Results
• Experiments/Exp3
• For each dataset: temp.py/etth1.py/stock.py (choosing model + estimator) → respective Results subfolder → plot_*.py
d. Visualization/Results
• CSVs and TXT results stored under each Results/ directory
• Python plotting scripts generate final figures
Architectural Patterns & Principles
• Layered architecture:
• Data Layer (Datasets)
• Logic/Computation Layer (Utils)
• Orchestration Layer (Experiment scripts)
• Presentation Layer (plot_*.py → figures)
• Reusability of Utils modules across all experiments
• Separation of concerns: data generation, entropy estimation, forecasting, and plotting each in distinct modules
Key Elements to Capture
• All four main pipelines: Challenges, Exp1, Exp2, Exp3
• Shared Utils layer and Datasets layer
• Sequential execution within each experiment
• Final Results destinations under each Results/ folder