# sameerbandi/tenx-data-engineer-starter architecture

> Architecture diagram of the GitHub repository sameerbandi/tenx-data-engineer-starter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sameerbandi/tenx-data-engineer-starter
- Repository: https://github.com/sameerbandi/tenx-data-engineer-starter

## Overview

This repository derives August 2026 month-end advisor positions from legacy files, compares them with migrated-platform positions, and writes a reconciliation report. The user starts the workflow with `python run.py`; the entry point configures and runs the DuckDB pipeline, then validates the report. The main stages are ingestion, cleaning and valuation, position derivation, reconciliation, and report output. The graph reflects documented inputs and shown calls; some SQL details and report-writing implementation were not excerpted.

Main entry points: [`pipeline.py`](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py) (Pipeline coordinator), [`run.py`](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/run.py) (Command runner), [`data/migrated`](https://github.com/sameerbandi/tenx-data-engineer-starter/tree/main/data/migrated) (Migrated positions), [`data/legacy`](https://github.com/sameerbandi/tenx-data-engineer-starter/tree/main/data/legacy) (Legacy CSV data)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry point"]
  node_runner["Command runner<br/>[run.py]"]
end

subgraph group_processing["Position processing"]
  node_pipeline["Pipeline coordinator<br/>[pipeline.py]"]
  node_ingest["Raw ingestion<br/>[pipeline.py]"]
  node_clean["Cleaning and pricing<br/>[pipeline.py]"]
  node_derive["Position derivation<br/>[pipeline.py]"]
end

subgraph group_reconciliation["Reconciliation"]
  node_reconcile["Position reconciliation<br/>[pipeline.py]"]
  node_write["Report writer<br/>[pipeline.py]"]
end

subgraph group_artifacts["Inputs and output"]
  node_legacy["Legacy CSV data"]
  node_migrated["Migrated positions"]
  node_report["Reconciliation report<br/>[recon_report.json]"]
  node_schema["Report schema"]
end

node_user(("Operator"))

node_user -->|"runs command"| node_runner
node_runner -->|"starts pipeline"| node_pipeline
node_pipeline -->|"calls ingestion"| node_ingest
node_pipeline -->|"calls cleaning"| node_clean
node_pipeline -->|"calls derivation"| node_derive
node_pipeline -->|"calls reconciliation"| node_reconcile
node_pipeline -->|"calls writer"| node_write
node_ingest -->|"reads CSVs"| node_legacy
node_ingest -->|"reads positions"| node_migrated
node_clean -->|"provides cleaned data"| node_derive
node_derive -->|"provides positions"| node_reconcile
node_reconcile -->|"compares against"| node_migrated
node_write -->|"writes report"| node_report
node_runner -->|"validates report"| node_schema

click node_runner "https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/run.py"
click node_pipeline "https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py"
click node_ingest "https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py"
click node_clean "https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py"
click node_derive "https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py"
click node_reconcile "https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py"
click node_write "https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py"
click node_legacy "https://github.com/sameerbandi/tenx-data-engineer-starter/tree/main/data/legacy"
click node_migrated "https://github.com/sameerbandi/tenx-data-engineer-starter/tree/main/data/migrated"
click node_report "https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/out/recon_report.json"
click node_schema "https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/schema/recon_report.schema.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner toneBlue
class node_pipeline,node_ingest,node_clean,node_derive toneAmber
class node_reconcile,node_write toneMint
class node_legacy,node_migrated,node_report,node_schema toneRose
class node_user toneIndigo
```

## Components

### Entry point

- [Command runner](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/run.py): `run.py`

### Position processing

- [Pipeline coordinator](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py): `pipeline.py`
- [Raw ingestion](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py): `pipeline.py`
- [Cleaning and pricing](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py): `pipeline.py`
- [Position derivation](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py): `pipeline.py`

### Reconciliation

- [Position reconciliation](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py): `pipeline.py`
- [Report writer](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/pipeline.py): `pipeline.py`

### Inputs and output

- [Legacy CSV data](https://github.com/sameerbandi/tenx-data-engineer-starter/tree/main/data/legacy): `data/legacy`
- [Migrated positions](https://github.com/sameerbandi/tenx-data-engineer-starter/tree/main/data/migrated): `data/migrated`
- [Reconciliation report](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/out/recon_report.json): `out/recon_report.json`
- [Report schema](https://github.com/sameerbandi/tenx-data-engineer-starter/blob/main/schema/recon_report.schema.json): `schema/recon_report.schema.json`

### Other components

- **Operator**

## Connections

- Operator → Command runner: runs command (evidence: `README.md`)
- Command runner → Pipeline coordinator: starts pipeline (evidence: `run.py`)
- Pipeline coordinator → Raw ingestion: calls ingestion (evidence: `pipeline.py`)
- Pipeline coordinator → Cleaning and pricing: calls cleaning (evidence: `pipeline.py`)
- Pipeline coordinator → Position derivation: calls derivation (evidence: `pipeline.py`)
- Pipeline coordinator → Position reconciliation: calls reconciliation (evidence: `pipeline.py`)
- Pipeline coordinator → Report writer: calls writer (evidence: `pipeline.py`)
- Raw ingestion → Legacy CSV data: reads CSVs (evidence: `pipeline.py`)
- Raw ingestion → Migrated positions: reads positions (evidence: `pipeline.py`)
- Cleaning and pricing → Position derivation: provides cleaned data (evidence: `pipeline.py`)
- Position derivation → Position reconciliation: provides positions (evidence: `pipeline.py`)
- Position reconciliation → Migrated positions: compares against (evidence: `README.md`)
- Report writer → Reconciliation report: writes report (evidence: `README.md`)
- Command runner → Report schema: validates report (evidence: `run.py`)