habibaaymannn/autocheck

1. What this repository is

- AutoCheck is a Python-based checkpointing framework that wraps long-running training or HPC jobs and persists their state to the local filesystem.

- The core design goal is “zero code modification”: the training job keeps running, while AutoCheck observes it from the outside and saves/restores state.

2. Core runtime flow

- cli.py and the Controller module are the main runtime entry points for starting a checkpointing session.

- A controller/orchestrator coordinates three things: reading config, watching execution state, and invoking the checkpoint manager when a policy threshold is hit.

- The persistence target is a structured checkpoints/ tree with metadata files such as session_info.json.

3. Configuration subsystem

- config/ConfigManager.py loads YAML configuration from files like config.yaml, config_hpc*.yaml, config_nlp.yaml, config_sklearn.yaml, and config_torch.yaml.

- config/YamlOBJ/* contains typed config objects for different concerns: checkpoint settings, HPC settings/state, ML settings, notifications, and system-level options.

- This is the main boundary where policy is defined: intervals, session limits, framework choice, storage paths, and environment-specific settings.

4. Checkpoint management subsystem

- checkpointManager/CheckpointManager.py and GenericCheckpointManager.py define the abstract save/load contract.

- Framework-specific implementations exist for:

- KerasCheckpointManager.py

- PyTorchCheckpointManager.py

- SKLearnCheckpointManager.py

- serializers.py suggests the framework-agnostic portion is delegated to serializers for model state and metadata packaging.

5. State tracking subsystem

- stateTracker/ holds the live execution state used by the controller.

- HPCStateTracker.py and MLStateTracker.py separate job-level state from ML-training state.

- The tracked state likely includes progress markers such as epoch, batch, timing, loss/accuracy, and job/session status, which the controller uses to decide when to checkpoint.

6. Execution layers / instrumentation boundaries

- layers/ contains the abstraction for interacting with different execution environments.

- BaseLayer.py, BaseDebuggerLayer.py, TraceLayer.py, and SignalLayer.py indicate generic monitoring and interruption handling.

- CLayer.py, CppLayer.py, and JavaLayer.py show explicit support for native/HPC-style jobs in addition to Python workflows.

- Architecturally, this is the compatibility layer between AutoCheck and the job being monitored.

7. Provider and runner coordination

- provider/Provider.py likely acts as the access point for shared runtime services or environment-specific wiring.

- runnerscript/RunnerScript.py is the bridge between user code and the checkpointing runtime, matching the README’s “run the training loop normally, but let AutoCheck orchestrate around it” model.

- In the architecture diagram, this is the component that calls into the model/training loop while feeding state updates to the tracker.

8. Notifications, logging, and error handling

- notifications/NotificationService.py suggests optional user-facing alerts around checkpoint events, failures, or termination.

- Utilites/logger.py, Utilites/error.py, and Utilites/enums.py provide shared logging, exception, and enum definitions used across the system.

- The presence of per-component log files in Testing_phase/keras_testing_done_on/logs/ shows observability is part of the design, not just training logic.

9. Framework validation and experimental harness

- Testing_phase/ is a large validation sandbox covering multiple domains:

- TensorFlow/Keras (keras_testing_done_on/)

- PyTorch (pytorch_testing_done_on/)

- scikit-learn (sklearn_testing_done_on/)

- classic ML and regression models (SVM.py, SVR.py, Kmeans.py, logisticReg.py, etc.)

- HPC jobs in Java, C, and C++ (HPCJob.java, train_sim.c, traim_sim_cpp.cpp)

- This folder is best understood as a test/integration lab proving that the checkpointing architecture can span frameworks and languages.

10. Persistence model and artifacts

- Checkpoints are stored as versioned directories/files under checkpoints/, test_checkpoints/, and framework-specific subtrees.

- The saved state includes model weights or serialized models plus metadata/session info; examples include TensorFlow weight shards, PyTorch/Torch-style artifacts, and scikit-learn .joblib models.

- There is no database or remote storage layer in the repo; the storage boundary is the filesystem.

11. Packaging and generated outputs

- dist/ contains packaged artifacts and copied runtime assets, including a built AutoCheck.exe and model/checkpoint bundles.

- These are build outputs rather than source architecture, but they confirm the project is intended to be distributable and runnable outside the source tree.

12. What to draw as the main architecture boundaries

- Draw a clear separation between:

- configuration ingestion

- runtime orchestration

- state tracking

- framework/language adaptation layers

- checkpoint serialization/persistence

- logging/notification side services

- The central flow is: config → controller/runner → state tracker + layers → checkpoint manager → filesystem checkpoints, with notifications/logging as side channels and testing_phase as validation inputs.