# capitalone/dataprofiler architecture

> Architecture diagram of the GitHub repository capitalone/dataprofiler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/capitalone/dataprofiler
- Repository: https://github.com/capitalone/dataprofiler

## Overview

##### Project Type & Purpose
- Type: Open-source Python library / tool for data profiling and sensitive-data detection.
- Purpose: 1) Auto-detect and load data (CSV, JSON, Avro, Parquet, Text, URL, DataFrame).
2) Profile data (structured, unstructured, graph) via statistical and ML pipelines.
3) Generate reports and allow profile updates, merges, and diffs.

##### High‐Level Components
A. Data Ingestion Layer
– **Data class** (`dataprofiler/data_readers/Data`, factory).
– **DataReader modules** (`csv_data.py`, `json_data.py`, `avro_data.py`, etc.)
– Output: Pandas DataFrame or equivalent.

B. Profiling Core
– **Profiler orchestrator** (`dataprofiler/profiler.py` entrypoint).
– **Profile Builder** (`profile_builder.py`): constructs the `global_stats` and `data_stats`.
– Three strategy sub-modules:
1. **Structured Profilers** (`float_column_profile.py`, `categorical_column_profile.py`, …)
2. **Unstructured Profiler** (`unstructured_text_profile.py`, `data_labeler_profile.py`)
3. **Graph Profiler** (`graph_profiler.py`)

C. Labeling Pipeline
– **Labeler abstraction** (`base_data_labeler.py`, `base_model.py`).
– **Concrete labelers** (regex, character-level CNN, column-name, structured and unstructured DL models).
– **Embeddings & pretrained models** (in `dataprofiler/labelers/embeddings` and `resources/labelers/*`).

D. Reporting & Export
– **Report generators** (`reports/graphs.py`, `reports/utils.py`).
– JSON- or dict-based reports (`report()` method) with optional pretty‐print.

E. Configuration & Utilities
– **Settings** (`settings.py`).
– **Logging** (`dp_logging.py`).
– **Random seed utilities** (`rng_utils.py`).
– **Plugin support** (`plugins/decorators.py`).

F. External Dependencies
– **Pandas** (data handling).
– **TensorFlow / Keras** (deep learning labelers).
– **NetworkX** or similar (graph profiling).
– **Numpy, Scipy** (statistics).
– **File format libs** (fastavro, pyarrow).

##### Relationships & Data Flow
1. **User code** calls `Data(…)` →
2. **DataReader** auto-selects appropriate subclass → loads file → returns DataFrame.
3. **User code** calls `Profiler(data)` →
4. **Profiler** invokes **Profile Builder** →
a. Routes to Structured / Unstructured / Graph profiling strategies
b. Each strategy consults Labelers where applicable
c. Computes stats via numerical & categorical profilers
5. **Profile object** assembled (global_stats + data_stats).
6. **Report API** serializes the profile → returns JSON/dict.
7. (Optional) **profile.update_profile()**, **profile + other_profile**, **profile.diff()** follow similar flows.

##### Architectural Patterns & Principles
- **Factory / Strategy** for DataReaders and Profiling modes
- **Modular design**: separate concerns (I/O, core logic, ML, reporting)
- **Plugin extension points** via decorators and resource directories
- **Resource abstraction**: pretrained models and parameter JSON files live under `/resources`

## Diagram (Mermaid)

```mermaid
graph TB

    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        Data["Data class"]:::io
        CSV["CSV Reader"]:::io
        JSON["JSON Reader"]:::io
        Avro["Avro Reader"]:::io
        Parquet["Parquet Reader"]:::io
        TextReader["Text Reader"]:::io
        GraphReader["Graph Reader"]:::io
        BufUtil["Buffer Utils"]:::io
        DataUtils["Data Utils"]:::io
    end

    %% Profiling Core
    subgraph "Profiling Core"
        Profiler["Profiler Orchestrator"]:::core
        Builder["Profile Builder"]:::core

        subgraph "Structured Profilers"
            Float["Float Profiler"]:::core
            Int["Integer Profiler"]:::core
            Categorical["Categorical Profiler"]:::core
            Datetime["Datetime Profiler"]:::core
            Order["Order Profiler"]:::core
            TextStruct["Text Profiler"]:::core
        end

        subgraph "Unstructured Profilers"
            UText["Unstructured Text Profiler"]:::core
            ULabel["Unstructured Labeler Profiler"]:::core
        end

        GraphProf["Graph Profiler"]:::core
    end

    %% Labeling Pipeline
    subgraph "Labeling Pipeline"
        BaseLabeler["Base Data Labeler"]:::ml
        BaseModel["Base Model"]:::ml
        Regex["Regex Labeler"]:::ml
        CNN["Char-level CNN"]:::ml
        ColName["Column-name Labeler"]:::ml
        TFHelper["TensorFlow Helper"]:::ml
        DataLabelers["DataLabelers Registry"]:::ml
        Embeddings["Embeddings"]:::ml
        Resources["Pretrained Models"]:::ml
    end

    %% Reporting & Export
    subgraph "Reporting & Export"
        GraphReports["Graph Reports"]:::report
        ReportUtils["Report Utils"]:::report
    end

    %% Config & Utilities
    subgraph "Config & Utilities"
        Settings["Settings"]:::util
        Logging["Logging"]:::util
        RNG["Random Seed Utils"]:::util
        Plugins["Plugin Decorators"]:::util
    end

    %% External Dependencies
    subgraph "External Dependencies"
        Pandas["Pandas"]:::external
        TF["TensorFlow/Keras"]:::external
        NX["NetworkX"]:::external
        Numpy["NumPy/SciPy"]:::external
        FileLibs["File-format libs"]:::external
    end

    %% Data Flow
    UserCode((User Code)) --> Data
    Data -->|"auto-select reader"| CSV
    Data --> JSON
    Data --> Avro
    Data --> Parquet
    Data --> TextReader
    Data --> GraphReader
    Data --> BufUtil
    Data --> DataUtils
    CSV & JSON & Avro & Parquet & TextReader & GraphReader -->|returns DataFrame| Profiler
    Profiler -->|invokes| Builder

    Builder --> Float
    Builder --> Int
    Builder --> Categorical
    Builder --> Datetime
    Builder --> Order
    Builder --> TextStruct
    Builder --> UText
    Builder --> ULabel
    Builder --> GraphProf

    ULabel --> BaseLabeler
    ULabel --> BaseModel
    Float --> BaseLabeler
    TextStruct --> BaseLabeler

    Regex --> BaseModel
    CNN --> BaseModel
    ColName --> BaseModel
    TFHelper --> BaseModel
    DataLabelers --> BaseLabeler
    BaseLabeler --> Embeddings
    BaseModel --> Resources

    Builder --> ProfileObj((Profile Object))
    ProfileObj --> GraphReports
    ProfileObj --> ReportUtils
    GraphReports -->|JSON/dict| UserCode
    ReportUtils -->|pretty-print| UserCode

    %% Config & Util usage
    Profiler --> Settings
    Profiler --> Logging
    Profiler --> RNG
    Profiler --> Plugins

    %% External integrations
    CSV --> Pandas
    JSON --> Pandas
    Avro --> FileLibs
    Parquet --> FileLibs
    TextReader --> Pandas
    GraphReader --> NX
    Builder --> Numpy
    Builder --> TF

    %% Click Events
    click Data "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/data_readers/data.py"
    click CSV "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/data_readers/csv_data.py"
    click JSON "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/data_readers/json_data.py"
    click Avro "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/data_readers/avro_data.py"
    click Parquet "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/data_readers/parquet_data.py"
    click TextReader "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/data_readers/text_data.py"
    click GraphReader "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/data_readers/graph_data.py"
    click BufUtil "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/data_readers/filepath_or_buffer.py"
    click DataUtils "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/data_readers/data_utils.py"
    click Profiler "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profiler.py"
    click Builder "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profilers/profile_builder.py"
    click Float "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profilers/float_column_profile.py"
    click Int "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profilers/int_column_profile.py"
    click Categorical "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profilers/categorical_column_profile.py"
    click Datetime "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profilers/datetime_column_profile.py"
    click Order "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profilers/order_column_profile.py"
    click TextStruct "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profilers/text_column_profile.py"
    click UText "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profilers/unstructured_text_profile.py"
    click ULabel "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profilers/unstructured_labeler_profile.py"
    click GraphProf "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/profilers/graph_profiler.py"
    click BaseLabeler "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/labelers/base_data_labeler.py"
    click BaseModel "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/labelers/base_model.py"
    click Regex "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/labelers/regex_model.py"
    click CNN "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/labelers/character_level_cnn_model.py"
    click ColName "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/labelers/column_name_model.py"
    click TFHelper "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/labelers/char_load_tf_model.py"
    click DataLabelers "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/labelers/data_labelers.py"
    click Embeddings "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/labelers/embeddings/glove.840B.300d-char.txt"
    click Resources "https://github.com/capitalone/dataprofiler/tree/main/resources/labelers/"
    click GraphReports "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/reports/graphs.py"
    click ReportUtils "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/reports/utils.py"
    click Settings "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/settings.py"
    click Logging "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/dp_logging.py"
    click RNG "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/rng_utils.py"
    click Plugins "https://github.com/capitalone/dataprofiler/blob/main/dataprofiler/plugins/decorators.py"

    %% Styles
    classDef io fill:#a8e6cf,stroke:#2e7d32;
    classDef core fill:#aed581,stroke:#558b2f;
    classDef ml fill:#ffd54f,stroke:#ff8f00;
    classDef report fill:#ce93d8,stroke:#6a1b9a;
    classDef util fill:#eeeeee,stroke:#757575;
    classDef external fill:#ffffff,stroke:#000000,stroke-dasharray: 2 2;
```
