# adm-l/data_profiler architecture

> Architecture diagram of the GitHub repository adm-l/data_profiler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/adm-l/data_profiler
- Repository: https://github.com/adm-l/data_profiler

## Overview

Go Data Profiler is an API-first service that accepts profiling jobs, runs them asynchronously against PostgreSQL, MySQL, or SQL Server sources, and persists results in a separate PostgreSQL metadata store. The principal workflow is request submission, queued profiling, quality and sensitive-data analysis, drift comparison, and profile retrieval/history. The graph includes documented profiling capabilities, but some internal calls are omitted where excerpts do not show their invocation; unsampled implementation details are not assumed.

Main entry points: [`internal/worker/manager.go`](https://github.com/adm-l/data_profiler/blob/master/internal/worker/manager.go) (Job Manager), [`internal/store/store.go`](https://github.com/adm-l/data_profiler/blob/master/internal/store/store.go) (Metadata Store), [`internal/api/router.go`](https://github.com/adm-l/data_profiler/blob/master/internal/api/router.go) (HTTP API), [`cmd/server/main.go`](https://github.com/adm-l/data_profiler/blob/master/cmd/server/main.go) (Server Startup)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_service["Service API"]
  node_server["Server Startup<br/>[main.go]"]
  node_config["Configuration<br/>[config.go]"]
  node_api["HTTP API<br/>[router.go]"]
  node_jobmanager["Job Manager<br/>[manager.go]"]
  node_domain["Profile Models<br/>[models.go]"]
end

subgraph group_profiling["Profiling"]
  node_adapter["Source Adapters<br/>[adapter.go]"]
  node_profiler["Table Profiler<br/>[profile.go]"]
end

subgraph group_analysis["Quality And History"]
  node_quality["Quality Rules<br/>[quality.go]"]
  node_pii["PII Detection<br/>[detect.go]"]
  node_drift["Drift Comparison<br/>[drift.go]"]
end

subgraph group_persistence["Persistence"]
  node_store[("Metadata Store<br/>[store.go]")]
  node_result["Profile Results<br/>[models.go]"]
end

node_client(("Client"))
node_source[("Source Databases")]
node_postgres[("Metadata PostgreSQL")]

node_server -->|"loads settings"| node_config
node_server -->|"creates store"| node_store
node_server -->|"starts workers"| node_jobmanager
node_server -->|"serves routes"| node_api
node_client -->|"submits job"| node_api
node_api -->|"creates job"| node_jobmanager
node_api -->|"reads job and profiles"| node_store
node_jobmanager -->|"manages job state"| node_store
node_jobmanager -->|"runs profiling"| node_adapter
node_adapter -->|"connects and discovers"| node_source
node_profiler -->|"queries table metrics"| node_source
node_adapter -->|"profiles table"| node_profiler
node_jobmanager -->|"evaluates profile"| node_quality
node_jobmanager -->|"detects sensitive data"| node_pii
node_jobmanager -->|"compares snapshots"| node_drift
node_jobmanager -->|"saves profile"| node_store
node_store -->|"persists jobs and profiles"| node_postgres
node_store -->|"loads profile history"| node_result
node_api -->|"returns status and results"| node_client

click node_server "https://github.com/adm-l/data_profiler/blob/master/cmd/server/main.go"
click node_config "https://github.com/adm-l/data_profiler/blob/master/internal/config/config.go"
click node_api "https://github.com/adm-l/data_profiler/blob/master/internal/api/router.go"
click node_jobmanager "https://github.com/adm-l/data_profiler/blob/master/internal/worker/manager.go"
click node_domain "https://github.com/adm-l/data_profiler/blob/master/internal/domain/models.go"
click node_adapter "https://github.com/adm-l/data_profiler/blob/master/internal/adapters/adapter.go"
click node_profiler "https://github.com/adm-l/data_profiler/blob/master/internal/adapters/profile.go"
click node_quality "https://github.com/adm-l/data_profiler/blob/master/internal/quality/quality.go"
click node_pii "https://github.com/adm-l/data_profiler/blob/master/internal/pii/detect.go"
click node_drift "https://github.com/adm-l/data_profiler/blob/master/internal/drift/drift.go"
click node_store "https://github.com/adm-l/data_profiler/blob/master/internal/store/store.go"
click node_result "https://github.com/adm-l/data_profiler/blob/master/internal/domain/models.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_config,node_api,node_jobmanager,node_domain,node_client toneBlue
class node_adapter,node_profiler,node_source,node_postgres toneAmber
class node_quality,node_pii,node_drift toneMint
class node_store,node_result toneRose
```

## Components

### Service API

- [Server Startup](https://github.com/adm-l/data_profiler/blob/master/cmd/server/main.go): `cmd/server/main.go`
- [Configuration](https://github.com/adm-l/data_profiler/blob/master/internal/config/config.go): `internal/config/config.go`
- [HTTP API](https://github.com/adm-l/data_profiler/blob/master/internal/api/router.go): `internal/api/router.go`
- [Job Manager](https://github.com/adm-l/data_profiler/blob/master/internal/worker/manager.go): `internal/worker/manager.go`
- [Profile Models](https://github.com/adm-l/data_profiler/blob/master/internal/domain/models.go): `internal/domain/models.go`

### Profiling

- [Source Adapters](https://github.com/adm-l/data_profiler/blob/master/internal/adapters/adapter.go): `internal/adapters/adapter.go`
- [Table Profiler](https://github.com/adm-l/data_profiler/blob/master/internal/adapters/profile.go): `internal/adapters/profile.go`

### Quality And History

- [Quality Rules](https://github.com/adm-l/data_profiler/blob/master/internal/quality/quality.go): `internal/quality/quality.go`
- [PII Detection](https://github.com/adm-l/data_profiler/blob/master/internal/pii/detect.go): `internal/pii/detect.go`
- [Drift Comparison](https://github.com/adm-l/data_profiler/blob/master/internal/drift/drift.go): `internal/drift/drift.go`

### Persistence

- [Metadata Store](https://github.com/adm-l/data_profiler/blob/master/internal/store/store.go): `internal/store/store.go`
- [Profile Results](https://github.com/adm-l/data_profiler/blob/master/internal/domain/models.go): `internal/domain/models.go`

### Other components

- **Client**
- **Source Databases**
- **Metadata PostgreSQL**

## Connections

- Server Startup → Configuration: loads settings (evidence: `cmd/server/main.go`)
- Server Startup → Metadata Store: creates store (evidence: `cmd/server/main.go`)
- Server Startup → Job Manager: starts workers (evidence: `cmd/server/main.go`)
- Server Startup → HTTP API: serves routes (evidence: `cmd/server/main.go`)
- Client → HTTP API: submits job (evidence: `README.md`)
- HTTP API → Job Manager: creates job (evidence: `internal/api/router.go`)
- HTTP API → Metadata Store: reads job and profiles (evidence: `internal/api/router.go`)
- Job Manager → Metadata Store: manages job state (evidence: `internal/worker/manager.go`)
- Job Manager → Source Adapters: runs profiling (evidence: `internal/worker/manager.go`)
- Source Adapters → Source Databases: connects and discovers (evidence: `README.md`)
- Table Profiler → Source Databases: queries table metrics (evidence: `README.md`)
- Source Adapters → Table Profiler: profiles table (evidence: `internal/adapters/adapter.go`)
- Job Manager → Quality Rules: evaluates profile (evidence: `internal/worker/manager.go`)
- Job Manager → PII Detection: detects sensitive data (evidence: `internal/worker/manager.go`)
- Job Manager → Drift Comparison: compares snapshots (evidence: `internal/worker/manager.go`)
- Job Manager → Metadata Store: saves profile (evidence: `internal/worker/manager.go`)
- Metadata Store → Metadata PostgreSQL: persists jobs and profiles (evidence: `README.md`)
- Metadata Store → Profile Results: loads profile history (evidence: `internal/store/store.go`)
- HTTP API → Client: returns status and results (evidence: `README.md`)
