How praneetha826/data_quality works
The product is a Streamlit enterprise data-quality assessment application with semantic retrieval and RAG-based recommendations. Users upload or choose a dataset, receive profiling and quality findings, and can query an AI assistant for contextual explanations; the dashboard also exposes assessment history and saved datasets. The README documents the full assessment-to-RAG stages, while the source index confirms the app’s module dependencies and selected implementation calls. Some stages are documented but their implementation was not sampled, so those relationships are cited to the README rather than inferred from code.
Main entry points: app.py Streamlit dashboard · loader_factory.py Format loader dispatch · base_loader.py Common data asset · rag_pipeline.py RAG orchestration
Components 19
User interface
- Streamlit dashboard
app.py
Data ingestion
- Format loader dispatch
src/ingestion/loader_factory.py - Common data asset
src/ingestion/base_loader.py - CSV loader
src/ingestion/csv_loader.py - Excel loader
src/ingestion/excel_loader.py - JSON loader
src/ingestion/json_loader.py - XML loader
src/ingestion/xml_loader.py - Text loader
src/ingestion/txt_loader.py - PDF loader
src/ingestion/pdf_loader.py - Common normalization
src/ingestion/normalizer.py
Quality assessment
- Data profiling
src/profiling.py - Quality metrics
src/quality_metrics.py - Metadata generation
src/metadata_generator.py
Retrieval and AI
- Text embeddings
src/embeddings.py - FAISS vector store
src/vector_store.py - RAG orchestration
src/rag_pipeline.py - LLM client
src/llm_client.py
Persistence
- Assessment database
src/database.py
Other components
- User
Connections 30
- User → Streamlit dashboard: uploads or queries
README.md - Streamlit dashboard → Format loader dispatch: selects format loader
app.py - Format loader dispatch → CSV loader: dispatches to
src/ingestion/loader_factory.py - Format loader dispatch → Excel loader: dispatches to
src/ingestion/loader_factory.py - Format loader dispatch → JSON loader: dispatches to
src/ingestion/loader_factory.py - Format loader dispatch → XML loader: dispatches to
src/ingestion/loader_factory.py - Format loader dispatch → Text loader: dispatches to
src/ingestion/loader_factory.py - Format loader dispatch → PDF loader: dispatches to
src/ingestion/loader_factory.py - CSV loader → Common data asset: creates asset
src/ingestion/csv_loader.py - Excel loader → Common data asset: creates asset
src/ingestion/excel_loader.py - JSON loader → Common data asset: creates asset
src/ingestion/json_loader.py - XML loader → Common data asset: creates asset
src/ingestion/xml_loader.py - Text loader → Common data asset: creates asset
src/ingestion/txt_loader.py - PDF loader → Common data asset: creates asset
src/ingestion/pdf_loader.py - Format loader dispatch → Common normalization: normalizes formats
README.md - Streamlit dashboard → Data profiling: profiles dataset
app.py - Streamlit dashboard → Quality metrics: assesses quality
app.py - Streamlit dashboard → Metadata generation: generates metadata
app.py - Streamlit dashboard → Text embeddings: creates embeddings
app.py - Streamlit dashboard → FAISS vector store: stores vectors
app.py - Streamlit dashboard → RAG orchestration: submits assistant query
app.py - RAG orchestration → Metadata generation: uses dataset metadata
src/rag_pipeline.py - RAG orchestration → LLM client: requests generation
src/rag_pipeline.py - Streamlit dashboard → Assessment database: persists assessments
app.py - Streamlit dashboard → User: shows findings and recommendations
README.md - Quality metrics → Metadata generation: feeds quality details
README.md - Metadata generation → Text embeddings: provides text for vectors
README.md - Text embeddings → FAISS vector store: supplies vectors
README.md - FAISS vector store → RAG orchestration: retrieves context
README.md - Streamlit dashboard → Assessment database: reads saved history
README.md