How capitalone/dataprofiler works
Project Type & Purpose
• Type: Open-source Python library / tool for data profiling and sensitive-data detection.
• Purpose: 1) Auto-detect and load data (CSV, JSON, Avro, Parquet, Text, URL, DataFrame).
2) Profile data (structured, unstructured, graph) via statistical and ML pipelines.
3) Generate reports and allow profile updates, merges, and diffs.
High‐Level Components
A. Data Ingestion Layer
– Data class (dataprofiler/data_readers/Data, factory).
– DataReader modules (csv_data.py, json_data.py, avro_data.py, etc.)
– Output: Pandas DataFrame or equivalent.
B. Profiling Core
– Profiler orchestrator (dataprofiler/profiler.py entrypoint).
– Profile Builder (profile_builder.py): constructs the global_stats and data_stats.
– Three strategy sub-modules:
1. Structured Profilers (float_column_profile.py, categorical_column_profile.py, …)
2. Unstructured Profiler (unstructured_text_profile.py, data_labeler_profile.py)
3. Graph Profiler (graph_profiler.py)
C. Labeling Pipeline
– Labeler abstraction (base_data_labeler.py, base_model.py).
– Concrete labelers (regex, character-level CNN, column-name, structured and unstructured DL models).
– Embeddings & pretrained models (in dataprofiler/labelers/embeddings and resources/labelers/*).
D. Reporting & Export
– Report generators (reports/graphs.py, reports/utils.py).
– JSON- or dict-based reports (report() method) with optional pretty‐print.
E. Configuration & Utilities
– Settings (settings.py).
– Logging (dp_logging.py).
– Random seed utilities (rng_utils.py).
– Plugin support (plugins/decorators.py).
F. External Dependencies
– Pandas (data handling).
– TensorFlow / Keras (deep learning labelers).
– NetworkX or similar (graph profiling).
– Numpy, Scipy (statistics).
– File format libs (fastavro, pyarrow).
Relationships & Data Flow
1. User code calls Data(…) →
2. DataReader auto-selects appropriate subclass → loads file → returns DataFrame.
3. User code calls Profiler(data) →
4. Profiler invokes Profile Builder →
a. Routes to Structured / Unstructured / Graph profiling strategies
b. Each strategy consults Labelers where applicable
c. Computes stats via numerical & categorical profilers
5. Profile object assembled (global_stats + data_stats).
6. Report API serializes the profile → returns JSON/dict.
7. (Optional) profile.update_profile(), profile + other_profile, profile.diff() follow similar flows.
Architectural Patterns & Principles
• Factory / Strategy for DataReaders and Profiling modes
• Modular design: separate concerns (I/O, core logic, ML, reporting)
• Plugin extension points via decorators and resource directories
• Resource abstraction: pretrained models and parameter JSON files live under /resources