# dmidlo/histdata.com-tools architecture

> Architecture diagram of the GitHub repository dmidlo/histdata.com-tools, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dmidlo/histdata.com-tools
- Repository: https://github.com/dmidlo/histdata.com-tools

## Overview

#### Purpose and entry points
`histdatacom` is a Python 3.10+ CLI and API package for acquiring HistData.com ASCII FX tick archives, extracting them, building canonical Polars `.data` caches, optionally importing to InfluxDB, and assessing local data quality. Primary entry points are the console CLI and `histdatacom(options)` API:
- `src/histdatacom/__main__.py`
- `src/histdatacom/cli.py`
- `src/histdatacom/api.py`

#### Request resolution and public API boundary
CLI flags/YAML configuration are resolved into `Options`, then submitted as orchestration requests. The public automation boundary is `Options`, `main`, and orchestration `RunRequest`; callers should not directly invoke legacy scraper/cache/import side effects.
- `src/histdatacom/options.py`
- `src/histdatacom/config.py`
- `src/histdatacom/orchestration/contracts.py`

#### Durable Temporal orchestration
Temporal is the default execution engine; there is no foreground fallback. Runtime lifecycle code provisions/locates the pinned Temporal executable, starts a local service and worker lanes, persists status, and routes work through workflows and activities. CLI/API callers may wait for results or submit-only; job commands inspect, cancel, resume, and retrieve artifacts.
- `src/histdatacom/orchestration/runtime.py`
- `src/histdatacom/orchestration/workflows.py`
- `src/histdatacom/orchestration/worker.py`

The runtime is local infrastructure backed by Temporal process state, SQLite history/status, logs, task queues, and packaged runtime metadata. First-run executable provisioning can use an operator override, offline bundle, verified cache, or verified download.
- `src/histdatacom/orchestration/resources.py`
- `src/histdatacom/orchestration/assets/temporal-runtime-index.json`
- `src/histdatacom/orchestration/client.py`

#### ETL execution stages and storage boundary
Activities execute the staged pipeline: resolve repository inventory and URLs, validate/download ZIP archives, extract CSV, build Polars caches, and optionally import. Data artifacts live under the selected data directory; canonical cache mode removes transient ZIP/CSV inputs after successful cache creation. Operational record metadata is manifest-backed rather than new adjacent `.meta` files.
- `src/histdatacom/activity_stages.py`
- `src/histdatacom/scraper/scraper.py`
- `src/histdatacom/csvs.py`

Repository inventory and artifact status are separate persistence concerns: the local `.repo` supports available-data and bounded quality summaries, while manifest storage tracks run/artifact lineage.
- `src/histdatacom/scraper/repo.py`
- `src/histdatacom/manifest_store.py`
- `src/histdatacom/records.py`

#### Data-quality engine
Offline quality workflows discover ZIP, CSV, and `.data` targets, ingest/normalize them, apply inventory, ingestion, time, tick, domain, modeling, provenance, and fingerprint rules, then serialize deterministic JSON reports and enforce configurable exit policy. This path does not require HistData.com or InfluxDB; provenance is conditional on local manifest state.
- `src/histdatacom/data_quality/engine.py`
- `src/histdatacom/data_quality/discovery.py`
- `src/histdatacom/data_quality/reporting.py`

Profiles, report contracts, bounded payloads, fingerprints, and preflight benchmarking form the reporting/automation boundary for CI and downstream consumers.
- `src/histdatacom/data_quality/profiles.py`
- `src/histdatacom/data_quality/fingerprint_contracts.py`
- `src/histdatacom/data_quality/preflight.py`

#### Operational control, telemetry, and cleanup
Job control manages schedule identity/no-overlap guards, workflow metadata, progress, cancellation, and artifact polling. Rich rendering is an interactive CLI presentation layer over telemetry; JSON remains the automation surface. Cleanup preserves `.data` caches while removing transient source artifacts.
- `src/histdatacom/orchestration/control.py`
- `src/histdatacom/orchestration/telemetry.py`
- `src/histdatacom/source_cleanup.py`

#### Optional integrations and analytics
InfluxDB is an optional external service: tick timestamps are transformed from HistData fixed EST-no-DST semantics to UTC epoch timestamps before batched writes. Optional pandas and Arrow API formats materialize completed cache artifacts. Offline feed-regime analytics is intentionally separate from pass/fail quality assessment.
- `src/histdatacom/influx.py`
- `src/histdatacom/data_analytics/feed_regimes.py`
- `src/histdatacom/data_analytics/cli.py`

Main entry points: [`src/histdatacom/activity_stages.py`](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/activity_stages.py) (ETL activities), [`src/histdatacom/orchestration/runtime.py`](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/runtime.py) (Runtime lifecycle), [`src/histdatacom/csvs.py`](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/csvs.py) (Data artifacts), [`src/histdatacom/data_quality/engine.py`](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/data_quality/engine.py) (Quality engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_public["Public Interface"]
  node_cli["CLI<br/>entry point<br/>[cli.py]"]
  node_api["Python API<br/>entry point<br/>[api.py]"]
  node_options["Options &amp; config<br/>request resolution<br/>[options.py]"]
  node_request["RunRequest<br/>orchestration contract<br/>[contracts.py]"]
end

subgraph group_orchestration["Temporal Runtime"]
  node_runtime{{"Runtime lifecycle<br/>Temporal bootstrap<br/>[runtime.py]"}}
  node_resources["Runtime resources<br/>executable provisioning<br/>[resources.py]"]
  node_temporal["Local Temporal<br/>workflow service"]
  node_worker["Worker lanes<br/>Temporal workers<br/>[worker.py]"]
  node_workflows["Workflows<br/>pipeline orchestration<br/>[workflows.py]"]
  node_control["Job control &amp; telemetry<br/>operations API<br/>[control.py]"]
end

subgraph group_pipeline["Archive Pipeline"]
  node_stages["ETL activities<br/>activity stages<br/>[activity_stages.py]"]
  node_histdata[("HistData.com<br/>archive source<br/>[scraper.py]")]
  node_artifacts[("Data artifacts<br/>local data directory<br/>[csvs.py]")]
  node_repository[("Repository inventory<br/>local .repo store<br/>[repo.py]")]
  node_manifest[("Manifest store<br/>lineage storage<br/>[manifest_store.py]")]
  node_influx[("InfluxDB<br/>optional integration<br/>[influx.py]")]
  node_cleanup["Source cleanup<br/>artifact lifecycle<br/>[source_cleanup.py]"]
end

subgraph group_quality["Quality &amp; Analytics"]
  node_quality["Quality engine<br/>offline assessment<br/>[engine.py]"]
  node_reports["Quality reports<br/>automation contract<br/>[reporting.py]"]
  node_analytics["Feed-regime analytics<br/>offline analytics<br/>[feed_regimes.py]"]
end

node_cli -->|"resolves"| node_options
node_api -->|"submits"| node_request
node_options -->|"creates"| node_request
node_request -->|"starts or connects"| node_runtime
node_runtime -->|"provisions from"| node_resources
node_runtime -->|"runs"| node_temporal
node_runtime -->|"starts"| node_worker
node_temporal -->|"executes"| node_workflows
node_worker -->|"runs activities"| node_stages
node_workflows -->|"orchestrates"| node_stages
node_control -->|"inspects and controls"| node_temporal
node_stages -->|"fetches archives"| node_histdata
node_stages -->|"resolves inventory"| node_repository
node_stages -->|"writes caches"| node_artifacts
node_stages -->|"records lineage"| node_manifest
node_stages -.->|"optional import"| node_influx
node_artifacts -->|"transient inputs"| node_cleanup
node_artifacts -->|"offline targets"| node_quality
node_repository -->|"inventory rules"| node_quality
node_manifest -.->|"conditional provenance"| node_quality
node_quality -->|"serializes"| node_reports
node_artifacts -->|"completed caches"| node_analytics

click node_cli "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/cli.py"
click node_api "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/api.py"
click node_options "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/options.py"
click node_request "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/contracts.py"
click node_runtime "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/runtime.py"
click node_resources "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/resources.py"
click node_worker "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/worker.py"
click node_workflows "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/workflows.py"
click node_control "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/control.py"
click node_stages "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/activity_stages.py"
click node_histdata "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/scraper/scraper.py"
click node_artifacts "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/csvs.py"
click node_repository "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/scraper/repo.py"
click node_manifest "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/manifest_store.py"
click node_influx "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/influx.py"
click node_quality "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/data_quality/engine.py"
click node_reports "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/data_quality/reporting.py"
click node_analytics "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/data_analytics/feed_regimes.py"
click node_cleanup "https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/source_cleanup.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_api,node_options,node_request toneBlue
class node_runtime,node_resources,node_temporal,node_worker,node_workflows,node_control toneAmber
class node_stages,node_histdata,node_artifacts,node_repository,node_manifest,node_influx,node_cleanup toneMint
class node_quality,node_reports,node_analytics toneRose
```

## Components

### Public Interface

- [CLI](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/cli.py): entry point · `src/histdatacom/cli.py`
- [Python API](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/api.py): entry point · `src/histdatacom/api.py`
- [Options & config](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/options.py): request resolution · Resolves CLI flags and YAML into Options. · `src/histdatacom/options.py`
- [RunRequest](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/contracts.py): orchestration contract · Public submission boundary for work requests. · `src/histdatacom/orchestration/contracts.py`

### Temporal Runtime

- [Runtime lifecycle](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/runtime.py): Temporal bootstrap · Provisions and starts the local Temporal service and worker lanes. · `src/histdatacom/orchestration/runtime.py`
- [Runtime resources](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/resources.py): executable provisioning · Selects override, offline bundle, verified cache, or download. · `src/histdatacom/orchestration/resources.py`
- **Local Temporal**: workflow service · Durable workflow execution, queues, and history.
- [Worker lanes](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/worker.py): Temporal workers · `src/histdatacom/orchestration/worker.py`
- [Workflows](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/workflows.py): pipeline orchestration · Coordinates durable staged ETL execution. · `src/histdatacom/orchestration/workflows.py`
- [Job control & telemetry](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/orchestration/control.py): operations API · Manages status, progress, cancellation, resume, and artifacts. · `src/histdatacom/orchestration/control.py`

### Archive Pipeline

- [ETL activities](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/activity_stages.py): activity stages · Resolves inventory, downloads, extracts, builds caches, and imports. · `src/histdatacom/activity_stages.py`
- [HistData.com](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/scraper/scraper.py): archive source · External ASCII FX tick archive source. · `src/histdatacom/scraper/scraper.py`
- [Data artifacts](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/csvs.py): local data directory · ZIP and CSV inputs plus canonical Polars .data caches. · `src/histdatacom/csvs.py`
- [Repository inventory](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/scraper/repo.py): local .repo store · Available-data inventory and bounded quality summaries. · `src/histdatacom/scraper/repo.py`
- [Manifest store](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/manifest_store.py): lineage storage · Tracks run and artifact lineage. · `src/histdatacom/manifest_store.py`
- [InfluxDB](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/influx.py): optional integration · Receives UTC-normalized tick batches. · `src/histdatacom/influx.py`
- [Source cleanup](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/source_cleanup.py): artifact lifecycle · Removes transient sources while preserving .data caches. · `src/histdatacom/source_cleanup.py`

### Quality & Analytics

- [Quality engine](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/data_quality/engine.py): offline assessment · Discovers local targets and applies deterministic validation rules. · `src/histdatacom/data_quality/engine.py`
- [Quality reports](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/data_quality/reporting.py): automation contract · Profiles, fingerprints, preflight, and deterministic JSON reports. · `src/histdatacom/data_quality/reporting.py`
- [Feed-regime analytics](https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/data_analytics/feed_regimes.py): offline analytics · Separate from pass/fail quality assessment. · `src/histdatacom/data_analytics/feed_regimes.py`

## Connections

- CLI → Options & config: resolves
- Python API → RunRequest: submits
- Options & config → RunRequest: creates
- RunRequest → Runtime lifecycle: starts or connects
- Runtime lifecycle → Runtime resources: provisions from
- Runtime lifecycle → Local Temporal: runs
- Runtime lifecycle → Worker lanes: starts
- Local Temporal → Workflows: executes
- Worker lanes → ETL activities: runs activities
- Workflows → ETL activities: orchestrates
- Job control & telemetry → Local Temporal: inspects and controls
- ETL activities → HistData.com: fetches archives
- ETL activities → Repository inventory: resolves inventory
- ETL activities → Data artifacts: writes caches
- ETL activities → Manifest store: records lineage
- ETL activities → InfluxDB: optional import
- Data artifacts → Source cleanup: transient inputs
- Data artifacts → Quality engine: offline targets
- Repository inventory → Quality engine: inventory rules
- Manifest store → Quality engine: conditional provenance
- Quality engine → Quality reports: serializes
- Data artifacts → Feed-regime analytics: completed caches
