# praneetha826/data_quality architecture

> Architecture diagram of the GitHub repository praneetha826/data_quality, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/praneetha826/data_quality
- Repository: https://github.com/praneetha826/data_quality

## Overview

The product is a Streamlit enterprise data-quality assessment application with semantic retrieval and RAG-based recommendations. Users upload or choose a dataset, receive profiling and quality findings, and can query an AI assistant for contextual explanations; the dashboard also exposes assessment history and saved datasets. The README documents the full assessment-to-RAG stages, while the source index confirms the app’s module dependencies and selected implementation calls. Some stages are documented but their implementation was not sampled, so those relationships are cited to the README rather than inferred from code.

Main entry points: [`app.py`](https://github.com/praneetha826/data_quality/blob/main/app.py) (Streamlit dashboard), [`src/ingestion/loader_factory.py`](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/loader_factory.py) (Format loader dispatch), [`src/ingestion/base_loader.py`](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/base_loader.py) (Common data asset), [`src/rag_pipeline.py`](https://github.com/praneetha826/data_quality/blob/main/src/rag_pipeline.py) (RAG orchestration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interface"]
  node_app["Streamlit dashboard<br/>[app.py]"]
end

subgraph group_ingestion["Data ingestion"]
  node_ingest_factory["Format loader dispatch<br/>[loader_factory.py]"]
  node_asset["Common data asset<br/>[base_loader.py]"]
  node_csv_loader["CSV loader<br/>[csv_loader.py]"]
  node_excel_loader["Excel loader<br/>[excel_loader.py]"]
  node_json_loader["JSON loader<br/>[json_loader.py]"]
  node_xml_loader["XML loader<br/>[xml_loader.py]"]
  node_txt_loader["Text loader<br/>[txt_loader.py]"]
  node_pdf_loader["PDF loader<br/>[pdf_loader.py]"]
  node_normalizer["Common normalization<br/>[normalizer.py]"]
end

subgraph group_assessment["Quality assessment"]
  node_profiling["Data profiling<br/>[profiling.py]"]
  node_quality["Quality metrics<br/>[quality_metrics.py]"]
  node_metadata["Metadata generation"]
end

subgraph group_rag["Retrieval and AI"]
  node_embeddings["Text embeddings<br/>[embeddings.py]"]
  node_vector_store[("FAISS vector store<br/>[vector_store.py]")]
  node_rag_pipeline["RAG orchestration<br/>[rag_pipeline.py]"]
  node_llm["LLM client<br/>[llm_client.py]"]
end

subgraph group_persistence["Persistence"]
  node_database[("Assessment database<br/>[database.py]")]
end

node_user(("User"))

node_user -->|"uploads or queries"| node_app
node_app -->|"selects format loader"| node_ingest_factory
node_ingest_factory -->|"dispatches to"| node_csv_loader
node_ingest_factory -->|"dispatches to"| node_excel_loader
node_ingest_factory -->|"dispatches to"| node_json_loader
node_ingest_factory -->|"dispatches to"| node_xml_loader
node_ingest_factory -->|"dispatches to"| node_txt_loader
node_ingest_factory -->|"dispatches to"| node_pdf_loader
node_csv_loader -->|"creates asset"| node_asset
node_excel_loader -->|"creates asset"| node_asset
node_json_loader -->|"creates asset"| node_asset
node_xml_loader -->|"creates asset"| node_asset
node_txt_loader -->|"creates asset"| node_asset
node_pdf_loader -->|"creates asset"| node_asset
node_ingest_factory -->|"normalizes formats"| node_normalizer
node_app -->|"profiles dataset"| node_profiling
node_app -->|"assesses quality"| node_quality
node_app -->|"generates metadata"| node_metadata
node_app -->|"creates embeddings"| node_embeddings
node_app -->|"stores vectors"| node_vector_store
node_app -->|"submits assistant query"| node_rag_pipeline
node_rag_pipeline -->|"uses dataset metadata"| node_metadata
node_rag_pipeline -->|"requests generation"| node_llm
node_app -->|"persists assessments"| node_database
node_app -->|"shows findings and recommendations"| node_user
node_quality -->|"feeds quality details"| node_metadata
node_metadata -->|"provides text for vectors"| node_embeddings
node_embeddings -->|"supplies vectors"| node_vector_store
node_vector_store -->|"retrieves context"| node_rag_pipeline
node_app -->|"reads saved history"| node_database

click node_app "https://github.com/praneetha826/data_quality/blob/main/app.py"
click node_ingest_factory "https://github.com/praneetha826/data_quality/blob/main/src/ingestion/loader_factory.py"
click node_asset "https://github.com/praneetha826/data_quality/blob/main/src/ingestion/base_loader.py"
click node_csv_loader "https://github.com/praneetha826/data_quality/blob/main/src/ingestion/csv_loader.py"
click node_excel_loader "https://github.com/praneetha826/data_quality/blob/main/src/ingestion/excel_loader.py"
click node_json_loader "https://github.com/praneetha826/data_quality/blob/main/src/ingestion/json_loader.py"
click node_xml_loader "https://github.com/praneetha826/data_quality/blob/main/src/ingestion/xml_loader.py"
click node_txt_loader "https://github.com/praneetha826/data_quality/blob/main/src/ingestion/txt_loader.py"
click node_pdf_loader "https://github.com/praneetha826/data_quality/blob/main/src/ingestion/pdf_loader.py"
click node_normalizer "https://github.com/praneetha826/data_quality/blob/main/src/ingestion/normalizer.py"
click node_profiling "https://github.com/praneetha826/data_quality/blob/main/src/profiling.py"
click node_quality "https://github.com/praneetha826/data_quality/blob/main/src/quality_metrics.py"
click node_metadata "https://github.com/praneetha826/data_quality/blob/main/src/metadata_generator.py"
click node_embeddings "https://github.com/praneetha826/data_quality/blob/main/src/embeddings.py"
click node_vector_store "https://github.com/praneetha826/data_quality/blob/main/src/vector_store.py"
click node_rag_pipeline "https://github.com/praneetha826/data_quality/blob/main/src/rag_pipeline.py"
click node_llm "https://github.com/praneetha826/data_quality/blob/main/src/llm_client.py"
click node_database "https://github.com/praneetha826/data_quality/blob/main/src/database.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_user toneBlue
class node_ingest_factory,node_asset,node_csv_loader,node_excel_loader,node_json_loader,node_xml_loader,node_txt_loader,node_pdf_loader,node_normalizer toneAmber
class node_profiling,node_quality,node_metadata toneMint
class node_embeddings,node_vector_store,node_rag_pipeline,node_llm toneRose
class node_database toneIndigo
```

## Components

### User interface

- [Streamlit dashboard](https://github.com/praneetha826/data_quality/blob/main/app.py): `app.py`

### Data ingestion

- [Format loader dispatch](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/loader_factory.py): `src/ingestion/loader_factory.py`
- [Common data asset](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/base_loader.py): `src/ingestion/base_loader.py`
- [CSV loader](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/csv_loader.py): `src/ingestion/csv_loader.py`
- [Excel loader](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/excel_loader.py): `src/ingestion/excel_loader.py`
- [JSON loader](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/json_loader.py): `src/ingestion/json_loader.py`
- [XML loader](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/xml_loader.py): `src/ingestion/xml_loader.py`
- [Text loader](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/txt_loader.py): `src/ingestion/txt_loader.py`
- [PDF loader](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/pdf_loader.py): `src/ingestion/pdf_loader.py`
- [Common normalization](https://github.com/praneetha826/data_quality/blob/main/src/ingestion/normalizer.py): `src/ingestion/normalizer.py`

### Quality assessment

- [Data profiling](https://github.com/praneetha826/data_quality/blob/main/src/profiling.py): `src/profiling.py`
- [Quality metrics](https://github.com/praneetha826/data_quality/blob/main/src/quality_metrics.py): `src/quality_metrics.py`
- [Metadata generation](https://github.com/praneetha826/data_quality/blob/main/src/metadata_generator.py): `src/metadata_generator.py`

### Retrieval and AI

- [Text embeddings](https://github.com/praneetha826/data_quality/blob/main/src/embeddings.py): `src/embeddings.py`
- [FAISS vector store](https://github.com/praneetha826/data_quality/blob/main/src/vector_store.py): `src/vector_store.py`
- [RAG orchestration](https://github.com/praneetha826/data_quality/blob/main/src/rag_pipeline.py): `src/rag_pipeline.py`
- [LLM client](https://github.com/praneetha826/data_quality/blob/main/src/llm_client.py): `src/llm_client.py`

### Persistence

- [Assessment database](https://github.com/praneetha826/data_quality/blob/main/src/database.py): `src/database.py`

### Other components

- **User**

## Connections

- User → Streamlit dashboard: uploads or queries (evidence: `README.md`)
- Streamlit dashboard → Format loader dispatch: selects format loader (evidence: `app.py`)
- Format loader dispatch → CSV loader: dispatches to (evidence: `src/ingestion/loader_factory.py`)
- Format loader dispatch → Excel loader: dispatches to (evidence: `src/ingestion/loader_factory.py`)
- Format loader dispatch → JSON loader: dispatches to (evidence: `src/ingestion/loader_factory.py`)
- Format loader dispatch → XML loader: dispatches to (evidence: `src/ingestion/loader_factory.py`)
- Format loader dispatch → Text loader: dispatches to (evidence: `src/ingestion/loader_factory.py`)
- Format loader dispatch → PDF loader: dispatches to (evidence: `src/ingestion/loader_factory.py`)
- CSV loader → Common data asset: creates asset (evidence: `src/ingestion/csv_loader.py`)
- Excel loader → Common data asset: creates asset (evidence: `src/ingestion/excel_loader.py`)
- JSON loader → Common data asset: creates asset (evidence: `src/ingestion/json_loader.py`)
- XML loader → Common data asset: creates asset (evidence: `src/ingestion/xml_loader.py`)
- Text loader → Common data asset: creates asset (evidence: `src/ingestion/txt_loader.py`)
- PDF loader → Common data asset: creates asset (evidence: `src/ingestion/pdf_loader.py`)
- Format loader dispatch → Common normalization: normalizes formats (evidence: `README.md`)
- Streamlit dashboard → Data profiling: profiles dataset (evidence: `app.py`)
- Streamlit dashboard → Quality metrics: assesses quality (evidence: `app.py`)
- Streamlit dashboard → Metadata generation: generates metadata (evidence: `app.py`)
- Streamlit dashboard → Text embeddings: creates embeddings (evidence: `app.py`)
- Streamlit dashboard → FAISS vector store: stores vectors (evidence: `app.py`)
- Streamlit dashboard → RAG orchestration: submits assistant query (evidence: `app.py`)
- RAG orchestration → Metadata generation: uses dataset metadata (evidence: `src/rag_pipeline.py`)
- RAG orchestration → LLM client: requests generation (evidence: `src/rag_pipeline.py`)
- Streamlit dashboard → Assessment database: persists assessments (evidence: `app.py`)
- Streamlit dashboard → User: shows findings and recommendations (evidence: `README.md`)
- Quality metrics → Metadata generation: feeds quality details (evidence: `README.md`)
- Metadata generation → Text embeddings: provides text for vectors (evidence: `README.md`)
- Text embeddings → FAISS vector store: supplies vectors (evidence: `README.md`)
- FAISS vector store → RAG orchestration: retrieves context (evidence: `README.md`)
- Streamlit dashboard → Assessment database: reads saved history (evidence: `README.md`)
