How dmidlo/histdata.com-tools works
Purpose and entry points
histdatacom is a Python 3.10+ CLI and API package for acquiring HistData.com ASCII FX tick archives, extracting them, building canonical Polars .data caches, optionally importing to InfluxDB, and assessing local data quality. Primary entry points are the console CLI and histdatacom(options) API:
• src/histdatacom/__main__.py
• src/histdatacom/cli.py
• src/histdatacom/api.py
Request resolution and public API boundary
CLI flags/YAML configuration are resolved into Options, then submitted as orchestration requests. The public automation boundary is Options, main, and orchestration RunRequest; callers should not directly invoke legacy scraper/cache/import side effects.
• src/histdatacom/options.py
• src/histdatacom/config.py
• src/histdatacom/orchestration/contracts.py
Durable Temporal orchestration
Temporal is the default execution engine; there is no foreground fallback. Runtime lifecycle code provisions/locates the pinned Temporal executable, starts a local service and worker lanes, persists status, and routes work through workflows and activities. CLI/API callers may wait for results or submit-only; job commands inspect, cancel, resume, and retrieve artifacts.
• src/histdatacom/orchestration/runtime.py
• src/histdatacom/orchestration/workflows.py
• src/histdatacom/orchestration/worker.py
The runtime is local infrastructure backed by Temporal process state, SQLite history/status, logs, task queues, and packaged runtime metadata. First-run executable provisioning can use an operator override, offline bundle, verified cache, or verified download.
• src/histdatacom/orchestration/resources.py
• src/histdatacom/orchestration/assets/temporal-runtime-index.json
• src/histdatacom/orchestration/client.py
ETL execution stages and storage boundary
Activities execute the staged pipeline: resolve repository inventory and URLs, validate/download ZIP archives, extract CSV, build Polars caches, and optionally import. Data artifacts live under the selected data directory; canonical cache mode removes transient ZIP/CSV inputs after successful cache creation. Operational record metadata is manifest-backed rather than new adjacent .meta files.
• src/histdatacom/activity_stages.py
• src/histdatacom/scraper/scraper.py
• src/histdatacom/csvs.py
Repository inventory and artifact status are separate persistence concerns: the local .repo supports available-data and bounded quality summaries, while manifest storage tracks run/artifact lineage.
• src/histdatacom/scraper/repo.py
• src/histdatacom/manifest_store.py
• src/histdatacom/records.py
Data-quality engine
Offline quality workflows discover ZIP, CSV, and .data targets, ingest/normalize them, apply inventory, ingestion, time, tick, domain, modeling, provenance, and fingerprint rules, then serialize deterministic JSON reports and enforce configurable exit policy. This path does not require HistData.com or InfluxDB; provenance is conditional on local manifest state.
• src/histdatacom/data_quality/engine.py
• src/histdatacom/data_quality/discovery.py
• src/histdatacom/data_quality/reporting.py
Profiles, report contracts, bounded payloads, fingerprints, and preflight benchmarking form the reporting/automation boundary for CI and downstream consumers.
• src/histdatacom/data_quality/profiles.py
• src/histdatacom/data_quality/fingerprint_contracts.py
• src/histdatacom/data_quality/preflight.py
Operational control, telemetry, and cleanup
Job control manages schedule identity/no-overlap guards, workflow metadata, progress, cancellation, and artifact polling. Rich rendering is an interactive CLI presentation layer over telemetry; JSON remains the automation surface. Cleanup preserves .data caches while removing transient source artifacts.
• src/histdatacom/orchestration/control.py
• src/histdatacom/orchestration/telemetry.py
• src/histdatacom/source_cleanup.py
Optional integrations and analytics
InfluxDB is an optional external service: tick timestamps are transformed from HistData fixed EST-no-DST semantics to UTC epoch timestamps before batched writes. Optional pandas and Arrow API formats materialize completed cache artifacts. Offline feed-regime analytics is intentionally separate from pass/fail quality assessment.
• src/histdatacom/influx.py
• src/histdatacom/data_analytics/feed_regimes.py
• src/histdatacom/data_analytics/cli.py
Main entry points: activity_stages.py ETL activities · runtime.py Runtime lifecycle · csvs.py Data artifacts · engine.py Quality engine
Components 20
Public Interface
- CLI
src/histdatacom/cli.py - Python API
src/histdatacom/api.py - Options & config · Resolves CLI flags and YAML into Options.
src/histdatacom/options.py - RunRequest · Public submission boundary for work requests.
src/histdatacom/orchestration/contracts.py
Temporal Runtime
- Runtime lifecycle · Provisions and starts the local Temporal service and worker lanes.
src/histdatacom/orchestration/runtime.py - Runtime resources · Selects override, offline bundle, verified cache, or download.
src/histdatacom/orchestration/resources.py - Local Temporal · Durable workflow execution, queues, and history.
- Worker lanes
src/histdatacom/orchestration/worker.py - Workflows · Coordinates durable staged ETL execution.
src/histdatacom/orchestration/workflows.py - Job control & telemetry · Manages status, progress, cancellation, resume, and artifacts.
src/histdatacom/orchestration/control.py
Archive Pipeline
- ETL activities · Resolves inventory, downloads, extracts, builds caches, and imports.
src/histdatacom/activity_stages.py - HistData.com · External ASCII FX tick archive source.
src/histdatacom/scraper/scraper.py - Data artifacts · ZIP and CSV inputs plus canonical Polars .data caches.
src/histdatacom/csvs.py - Repository inventory · Available-data inventory and bounded quality summaries.
src/histdatacom/scraper/repo.py - Manifest store · Tracks run and artifact lineage.
src/histdatacom/manifest_store.py - InfluxDB · Receives UTC-normalized tick batches.
src/histdatacom/influx.py - Source cleanup · Removes transient sources while preserving .data caches.
src/histdatacom/source_cleanup.py
Quality & Analytics
- Quality engine · Discovers local targets and applies deterministic validation rules.
src/histdatacom/data_quality/engine.py - Quality reports · Profiles, fingerprints, preflight, and deterministic JSON reports.
src/histdatacom/data_quality/reporting.py - Feed-regime analytics · Separate from pass/fail quality assessment.
src/histdatacom/data_analytics/feed_regimes.py
Connections 22
- CLI → Options & config: resolves
- Python API → RunRequest: submits
- Options & config → RunRequest: creates
- RunRequest → Runtime lifecycle: starts or connects
- Runtime lifecycle → Runtime resources: provisions from
- Runtime lifecycle → Local Temporal: runs
- Runtime lifecycle → Worker lanes: starts
- Local Temporal → Workflows: executes
- Worker lanes → ETL activities: runs activities
- Workflows → ETL activities: orchestrates
- Job control & telemetry → Local Temporal: inspects and controls
- ETL activities → HistData.com: fetches archives
- ETL activities → Repository inventory: resolves inventory
- ETL activities → Data artifacts: writes caches
- ETL activities → Manifest store: records lineage
- ETL activities → InfluxDB: optional import
- Data artifacts → Source cleanup: transient inputs
- Data artifacts → Quality engine: offline targets
- Repository inventory → Quality engine: inventory rules
- Manifest store → Quality engine: conditional provenance
- Quality engine → Quality reports: serializes
- Data artifacts → Feed-regime analytics: completed caches