How devadripta/agentic-training-copilot works
This repo is a single-file, CLI-based observability tool for deep learning training runs. It does not run as a service and has no UI; it attaches to training log files, interprets them via a deterministic state machine, predicts ETA, and emits narrative alerts to Console and optionally Telegram.
From the file tree + README, the core “system” is essentially:
• A local Python runtime (python copilot.py ...)
• Reading from the filesystem (log discovery + tailing)
• Optional outbound calls to Telegram Bot API via requests
What the file tree implies architecturally
Top-level:
• copilot.py — the primary executable (v2.1 per README)
• README.md, QUICKSTART.md — documentation, usage contracts (CLI flags / env vars)
Main components to show (decompose aggressively)
Use the README’s architecture block as your baseline, then expand it into explicit subcomponents:
Entry & Configuration
• CLI Entrypoint: python copilot.py
• Argument parsing: --auto, --log, --wait, --base-dir, --from-start, --token, --chat
• Config sources:
• CLI flags
• Environment variables (TELEGRAM_TOKEN, TELEGRAM_CHAT)
Log Discovery & Attachment Layer (platform-agnostic)
• LogDiscovery
• Builds search_paths (current dir + common dirs + optional platform hints)
• Recursive glob search (**/*.log)
• Filters by heuristics:
• size > 1KB
• contains training indicators (“epoch”, “loss”, “train”)
• Picks “best candidate” by recency/fitness
• Wait Mechanism
• If no log exists yet, sleeps and retries up to --wait seconds
Log Reader / Tailer
• File tailing loop
• Follow appended lines (like tail -f)
• Optional “read from beginning” (--from-start)
• Produces a stream of new log lines/events into the pipeline
Parser (Extract structured signals)
• MMEngine Log Parser
• Extracts metrics: loss, lr, val mAP, epoch/iter counters, timestamps
• Detects anomalies: NaN/Inf, exploding loss patterns (for DIVERGING)
Training State Machine (Interpretation)
• Phase/state classifier
• States listed in README: INITIALIZING, WARMUP, LEARNING_FAST, STABLE_CONVERGENCE, PLATEAU, OVERFITTING_RISK, DIVERGING, CONVERGED
• Transition logic
• Deterministic thresholds (improvement per epoch, divergence between train/val, etc.)
• Emits “phase transition” events with reason strings
ETA Engine (Predictive)
• ETA prediction
• EMA-smoothed progress rate
• Confidence band / qualitative confidence
• “Plateau-aware” ETA adjustments (per README narrative)
Decision / Recommendation Engine
• Early termination recommendation
• Probability-of-gain heuristics
• Savings estimate in GPU-hours (needs runtime assumptions or inferred epoch time)
Notifier / Output Adapters
• Intelligent Notifier
• Cooldown / spam suppression
• Event formatting: narrative messages (phase transitions, early stop, divergence)
• Sinks
• Console sink (stdout)
• Telegram sink (Telegram Bot API via requests)
External Systems (must be explicit
• Training job / framework (MMEngine) producing logs
• Filesystem where logs are written
• Telegram Bot API (external SaaS endpoint)
CLI/Config decides mode
• --log explicit path OR --auto discovery
Control plane (configuration & gating)
• CLI flags/env vars configure:
• search locations (--base-dir)
• waiting (--wait)
• reading mode (--from-start)
• notification endpoints (--token, --chat)
• cooldown behavior (if implemented)
Architectural patterns to highlight (what makes this project “itself”)
• Single-file deployable monolith with internal modular boundaries
• Pipeline architecture (Reader → Parser → Interpreters → Notifier)
• State machine as the interpretability primitive (key innovation)
• Adapter pattern for notification sinks (Console, Telegram)
• Platform-agnostic filesystem discovery (no hard dependency on Kaggle/Colab APIs)
• User
→ (runs) Copilot CLI (copilot.py)
• Config (CLI/env)
• LogDiscovery (+ Wait loop)
• Tailer (follow file)
• Parser (MMEngine patterns)
• State Machine (8 phases)
• ETA Engine (EMA + confidence)
• Decision Engine (early stop/overfit/diverge)
• Notifier (cooldown + formatting)
• Console sink
• Telegram sink (requests)
↔ Filesystem (log file)
• Training job (MMEngine) → Filesystem
• Copilot → Telegram Bot API → User phone