How asavinov/intelligent-trading-bot works
<explanation>
1. Project type and purpose
- This is a Python‑based, data‑pipeline + ML‑driven trading bot. It consists of two major “subsystems”:
a. An offline batch pipeline (scripts under scripts/, backed by utility code in common/) that downloads market data, merges it, generates features and labels, trains ML models, aggregates scores, and optimizes signal parameters.
b. An online service (code under service/, with connectors in inputs/ and outputs/) that periodically pulls live data, computes features, applies pre‑trained models, aggregates scores, issues buy/sell signals, and optionally places real trades and sends Telegram notifications.
2. File‑structure analysis
- Top‑level directories:
• common/ – shared routines: backtesting, feature/label generators, signal aggregation, model‑store definitions.
• scripts/ – offline batch steps (download_…, merge, features, labels, train, signals, train_signals, plus helpers for Yahoo/rolling prediction).
• inputs/ – live data collectors (e.g., collector_binance.py).
• outputs/ – notification/trading adapters (e.g., notifier_*.py, trader_binance.py).
• service/ – the server.py entrypoint, App.py config, real‑time analyzer.
• configs/ – sample JSONC config files.
• docs/, tests/, requirements.txt, etc.
3. Key README insights
- Offline steps produce a model bundle in MODELS/.
- Online service (python -m service.server -c config.json) requires those models.
- External integrations: Binance REST/WebSocket API, Telegram Bot API.
- ML frameworks: scikit‑learn (via model_store.py), TA‑lib, tsfresh, custom algorithms.
4. Main components for the diagram
A. Offline Batch Pipeline (left side)
1. Data Sources
– Binance API (scripts/download_binance)
– Yahoo API (scripts/download_yahoo)
2. Merge & Rasterize (scripts/merge)
3. Feature Generation (scripts/features + common/gen_features)
4. Label Generation (scripts/labels + common/gen_labels_*)
5. Model Training (scripts/train + common/model_store)
6. Score Aggregation (scripts/signals + scripts/train_signals)
7. Artifact Store
– Time‑series CSV/Parquet files (under data_folder)
– Trained model files (under MODELS/)
B. Online Service (right side)
1. Live Data Collector (inputs/collector_binance)
2. In‑Memory Data Window (config‑driven history buffer)
3. Feature Computation (common/gen_features rolling)
4. Prediction Engine (service/analyzer applies models)
5. Score Aggregator (common/gen_signals)
6. Signal Orchestrator (service/server + trade_model rules)
7. Notification Service (outputs/notifier_*.py → Telegram)
8. Trader Service (outputs/trader_binance → Binance execution)
C. Shared Libraries
– common/ utilities used in both offline and online paths.
5. Relationships and data flows
- Draw directional arrows:
* Data sources → Merge → Features → Labels → Training → Model Store
* Model Store + Live Data → Prediction Engine → Aggregation → Signals/Trades → External APIs
- Show feedback loop: best signal thresholds derived offline feed back to online Signal Orchestrator.
6. Architectural patterns and design principles
- Separation of Concerns: clear offline vs. online pipelines
- Modularity: common/ holds reusable logic; scripts/ form a linear ETL chain; service/ is a long‑running daemon
- Config‑driven: everything—data sources, feature sets, labels, algorithms, thresholds—is defined in JSONC
- Extensibility: new feature generators (in common/), new label generators, new output hooks
7. Diagramming guidelines
- Use containers or swimlanes for “Offline Batch” vs. “Online Service.”
- Represent filesystems/data stores as cylinders (data folders, model store).
- Use rectangles for compute steps (Merge, Feature Gen, Train, Prediction, Aggregation).
- Use parallelograms or cloud icons for external systems (Binance API, Telegram).
- Color‑code:
• Blue = batch steps
• Green = online steps
• Orange = external services
• Gray = shared library (common/)
- Label each component with its module/file path (e.g., “scripts/download_binance.py”).
- Draw arrowheads on flows to show direction (e.g., “raw-data.csv → merge → merged.csv”).
8. Tailoring to this project
- Emphasize the ETL nature of the offline pipeline.
- Highlight the real‑time loop in the service: fetch data, compute features, predict, send signals.
- Show how models trained offline are consumed online.
- Call out configuration inputs (configs/*.jsonc) that parameterize both pipelines.
9. Essential details to capture
- The data‑artifact lifecycle from raw download to trained model.
- The shared code in common/ that powers both scripts and service.
- The interaction between Signal Orchestrator and Notification/Trader adapters.
- The external touchpoints: Binance and Telegram.
By following this breakdown, the principal engineer can draft a layered system‑architecture diagram that clearly delineates offline vs. online components, data flows, dependencies, and external integrations—ensuring no critical element is overlooked.
</explanation>