# pierregr7/data_factory_x architecture

> Architecture diagram of the GitHub repository pierregr7/data_factory_x, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pierregr7/data_factory_x
- Repository: https://github.com/pierregr7/data_factory_x

## Overview

Data_Factory is a CSV profiling and dimensional-modeling tool. Its primary workflow loads CSV tables, profiles them, discovers likely joins, chooses a fact-and-dimension schema, and materializes an enriched DuckDB table with run summaries. The `first_py/` scripts provide separate, simpler profiling, cleaning, join-discovery, and schema-proposal utilities; the available excerpts do not show them integrated with the main runner. Edges for the main workflow are grounded in `dfx_run.py`; standalone utility relationships are omitted where no call is shown.

Main entry points: [`dfx_run.py`](https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py) (Pipeline runner), [`outputs`](https://github.com/pierregr7/data_factory_x/tree/main/outputs) (DuckDB model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["Main pipeline"]
  node_runner["Pipeline runner<br/>[dfx_run.py]"]
  node_loader["CSV loader<br/>[dfx_run.py]"]
  node_profiler["Table profiler<br/>[dfx_run.py]"]
end

subgraph group_modeling["Schema modeling"]
  node_join_discovery["Join discovery<br/>[dfx_run.py]"]
  node_schema_choice["Schema selection<br/>[dfx_run.py]"]
  node_fact_dim["Fact-dimension scoring<br/>[dfx_run.py]"]
  node_duckdb_builder["DuckDB builder<br/>[dfx_run.py]"]
end

subgraph group_utilities["Standalone utilities"]
  node_simple_profile["CSV profiling<br/>[profiling.py]"]
  node_multi_profile["Multi-CSV profiling"]
  node_clean_proposal["Cleaning suggestions"]
  node_clean_apply["Cleaning application<br/>[apply_cleaning.py]"]
  node_join_utility["Join utility<br/>[find_joins.py]"]
  node_fact_dim_utility["Fact-dim utility<br/>[fact_dim_guess.py]"]
  node_schema_utility["Schema proposal"]
  node_csv_reader["CSV display<br/>[read_csv.py]"]
end

subgraph group_artifacts["Run artifacts"]
  node_run_dir["Run directory<br/>[dfx_run.py]"]
  node_model_store[("DuckDB model")]
  node_summary["Run summary"]
end

node_analyst(("Operator"))
node_csv["CSV inputs"]

node_analyst -->|"runs"| node_runner
node_runner -->|"creates"| node_run_dir
node_runner -->|"calls"| node_loader
node_loader -->|"reads"| node_csv
node_runner -->|"profiles"| node_profiler
node_runner -->|"discovers joins"| node_join_discovery
node_runner -->|"selects schema"| node_schema_choice
node_schema_choice -->|"scores tables"| node_fact_dim
node_runner -->|"builds model"| node_duckdb_builder
node_duckdb_builder -->|"writes"| node_model_store
node_runner -->|"writes"| node_summary
node_run_dir -.->|"contains"| node_model_store
node_run_dir -.->|"contains"| node_summary

click node_runner "https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py"
click node_run_dir "https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py"
click node_loader "https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py"
click node_profiler "https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py"
click node_join_discovery "https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py"
click node_schema_choice "https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py"
click node_fact_dim "https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py"
click node_duckdb_builder "https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py"
click node_model_store "https://github.com/pierregr7/data_factory_x/tree/main/outputs"
click node_summary "https://github.com/pierregr7/data_factory_x/tree/main/outputs"
click node_simple_profile "https://github.com/pierregr7/data_factory_x/blob/main/first_py/profiling.py"
click node_multi_profile "https://github.com/pierregr7/data_factory_x/blob/main/first_py/multi_csv_profiling.py"
click node_clean_proposal "https://github.com/pierregr7/data_factory_x/blob/main/first_py/propose_cleaning.py"
click node_clean_apply "https://github.com/pierregr7/data_factory_x/blob/main/first_py/apply_cleaning.py"
click node_join_utility "https://github.com/pierregr7/data_factory_x/blob/main/first_py/find_joins.py"
click node_fact_dim_utility "https://github.com/pierregr7/data_factory_x/blob/main/first_py/fact_dim_guess.py"
click node_schema_utility "https://github.com/pierregr7/data_factory_x/blob/main/first_py/star_schema_proposal.py"
click node_csv_reader "https://github.com/pierregr7/data_factory_x/blob/main/first_py/read_csv.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_loader,node_profiler toneBlue
class node_join_discovery,node_schema_choice,node_fact_dim,node_duckdb_builder toneAmber
class node_simple_profile,node_multi_profile,node_clean_proposal,node_clean_apply,node_join_utility,node_fact_dim_utility,node_schema_utility,node_csv_reader toneMint
class node_run_dir,node_model_store,node_summary toneRose
class node_analyst,node_csv toneIndigo
```

## Components

### Main pipeline

- [Pipeline runner](https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py): `dfx_run.py`
- [CSV loader](https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py): `dfx_run.py`
- [Table profiler](https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py): `dfx_run.py`

### Schema modeling

- [Join discovery](https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py): `dfx_run.py`
- [Schema selection](https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py): `dfx_run.py`
- [Fact-dimension scoring](https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py): `dfx_run.py`
- [DuckDB builder](https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py): `dfx_run.py`

### Standalone utilities

- [CSV profiling](https://github.com/pierregr7/data_factory_x/blob/main/first_py/profiling.py): `first_py/profiling.py`
- [Multi-CSV profiling](https://github.com/pierregr7/data_factory_x/blob/main/first_py/multi_csv_profiling.py): `first_py/multi_csv_profiling.py`
- [Cleaning suggestions](https://github.com/pierregr7/data_factory_x/blob/main/first_py/propose_cleaning.py): `first_py/propose_cleaning.py`
- [Cleaning application](https://github.com/pierregr7/data_factory_x/blob/main/first_py/apply_cleaning.py): `first_py/apply_cleaning.py`
- [Join utility](https://github.com/pierregr7/data_factory_x/blob/main/first_py/find_joins.py): `first_py/find_joins.py`
- [Fact-dim utility](https://github.com/pierregr7/data_factory_x/blob/main/first_py/fact_dim_guess.py): `first_py/fact_dim_guess.py`
- [Schema proposal](https://github.com/pierregr7/data_factory_x/blob/main/first_py/star_schema_proposal.py): `first_py/star_schema_proposal.py`
- [CSV display](https://github.com/pierregr7/data_factory_x/blob/main/first_py/read_csv.py): `first_py/read_csv.py`

### Run artifacts

- [Run directory](https://github.com/pierregr7/data_factory_x/blob/main/dfx_run.py): `dfx_run.py`
- [DuckDB model](https://github.com/pierregr7/data_factory_x/tree/main/outputs): `outputs`
- [Run summary](https://github.com/pierregr7/data_factory_x/tree/main/outputs): `outputs`

### Other components

- **Operator**
- **CSV inputs**

## Connections

- Operator → Pipeline runner: runs (evidence: `dfx_run.py`)
- Pipeline runner → Run directory: creates (evidence: `dfx_run.py`)
- Pipeline runner → CSV loader: calls (evidence: `dfx_run.py`)
- CSV loader → CSV inputs: reads (evidence: `dfx_run.py`)
- Pipeline runner → Table profiler: profiles (evidence: `dfx_run.py`)
- Pipeline runner → Join discovery: discovers joins (evidence: `dfx_run.py`)
- Pipeline runner → Schema selection: selects schema (evidence: `dfx_run.py`)
- Schema selection → Fact-dimension scoring: scores tables (evidence: `dfx_run.py`)
- Pipeline runner → DuckDB builder: builds model (evidence: `dfx_run.py`)
- DuckDB builder → DuckDB model: writes (evidence: `dfx_run.py`)
- Pipeline runner → Run summary: writes (evidence: `dfx_run.py`)
- Run directory → DuckDB model: contains
- Run directory → Run summary: contains
