# criscotero/nlp-chatdash architecture

> Architecture diagram of the GitHub repository criscotero/nlp-chatdash, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/criscotero/nlp-chatdash
- Repository: https://github.com/criscotero/nlp-chatdash

## Overview

##### Project type and purpose
- NLP ChatDash is a full-stack, Python-based RAG (Retrieval-Augmented Generation) search and FAQ system.
- It ingests PDF documents, extracts embeddings and topics, builds a vector index, and exposes three user interfaces:
- A REST API (FastAPI)
- A Streamlit dashboard
- A Gradio chat UI

##### File structure
- Top-level:
- app/ – core application code
- data/ – document inputs and generated artifacts
- Makefile, main.py – orchestration entry points
- requirements.txt, pyproject.toml – dependencies
- app/ subdivides into:
- api/ – FastAPI service
- dashboard/ – Streamlit app
- ui/ – Gradio app
- cli.py – Typer CLI commands (build-index, ingest, topics, eval, etc.)
- features/ – code for embeddings, clustering, topic modeling
- pipeline/ – ingestion and indexing workflows, registry, logging
- rag/ – RAG training, inference, evaluation, synthesis
- utils/ & config.py – shared utilities and configuration

##### README for additional insights
- Workflows orchestrated via Make targets: setup, train, init-corpus, api, dash, gradio, eval, topics, pipeline
- Data flow:
- PDFs in data/docs/
- make init-corpus → ingest PDFs → deduplicate → generate topics (c-TF-IDF) → compute embeddings → build vector index
- make train → build or retrain FAQ index for RAG
- Interfaces load models/indices at runtime

##### Elements
- Clear labels (component names and file or module examples)
- Yellow for model training/inference
- Orange for UI layers

##### Detail advice
- Makefile/CLI kicking off each pipeline step.
- That all three frontends share the same config and artifacts directory via app/config.py.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interfaces Layer
    Streamlit["Streamlit Dashboard"]:::ui
    Gradio["Gradio Chat UI"]:::ui
    FastAPI["FastAPI Service"]:::service

    %% CLI / Orchestration
    CLI["CLI & Makefile"]:::cli

    %% Backend Pipeline & Services
    subgraph "Ingestion & Indexing Pipeline"
        direction TB
        Ingest["ingest.py"]:::pipeline
        Initialize["initialize.py"]:::pipeline
        TopicsPipeline["topics.py"]:::pipeline
        Registry["registry.py"]:::pipeline
    end

    subgraph "Feature Extraction"
        direction TB
        FE_Embeddings["embeddings.py"]:::pipeline
        FE_Topics["topics.py"]:::pipeline
        FE_Clustering["clustering.py"]:::pipeline
        FE_DocsIndex["docs_index.py"]:::pipeline
    end

    subgraph "RAG Model & FAQ Index"
        direction TB
        RAG_Train["train.py"]:::model
        RAG_Eval["eval.py"]:::model
        RAG_Infer["infer.py"]:::model
        RAG_Synth["synthesis.py"]:::model
    end

    %% Shared Configuration & Utilities
    Config["config.py"]:::util
    Utils_Paths["paths.py"]:::util
    Utils_Logging["logging.py"]:::util
    Utils_Hash["hash.py"]:::util
    Utils_Text["text.py"]:::util

    %% Data & Artifacts Storage Layer
    subgraph "Storage"
        direction TB
        DocsStore["Document Store"]:::storage
        Artifacts["Artifact Storage"]:::storage
    end

    %% Data Flows
    CLI -->|triggers| Ingest
    CLI -->|triggers| Initialize
    CLI -->|triggers| TopicsPipeline
    CLI -->|triggers| Registry

    DocsStore -->|reads raw PDFs| Ingest
    Ingest -->|writes artifacts| Artifacts
    Initialize -->|initializes corpus| Artifacts
    TopicsPipeline -->|generates topics| Artifacts
    Registry -->|manages pipeline| Artifacts

    Artifacts -->|source for features| FE_Embeddings
    Artifacts -->|source for features| FE_Topics
    Artifacts -->|source for features| FE_Clustering
    Artifacts -->|source for features| FE_DocsIndex

    FE_Embeddings -->|stores embeddings| Artifacts
    FE_Topics -->|stores topic data| Artifacts
    FE_Clustering -->|stores clusters| Artifacts
    FE_DocsIndex -->|builds index| Artifacts

    Artifacts -->|training data| RAG_Train
    Artifacts -->|evaluation data| RAG_Eval
    RAG_Train -->|FAQ index| Artifacts
    RAG_Eval -->|metrics| Artifacts

    Artifacts -->|runtime artifacts| RAG_Infer
    Artifacts -->|runtime artifacts| RAG_Synth

    %% Serving & Inference
    FastAPI -->|calls inference| RAG_Infer
    FastAPI -->|calls synthesis| RAG_Synth
    Streamlit -->|direct inference| RAG_Infer
    Gradio -->|direct inference| RAG_Infer

    %% Config & Utils usage
    Config --> Ingest
    Config --> Initialize
    Config --> TopicsPipeline
    Config --> Registry
    Config --> FE_Embeddings
    Config --> FE_Topics
    Config --> FE_Clustering
    Config --> FE_DocsIndex
    Config --> RAG_Train
    Config --> RAG_Eval
    Config --> RAG_Infer
    Config --> RAG_Synth
    Config --> FastAPI
    Config --> Streamlit
    Config --> Gradio

    Utils_Paths --> Ingest
    Utils_Logging --> Ingest
    Utils_Hash --> Ingest
    Utils_Text --> Ingest

    %% Click Events
    click DocsStore "https://github.com/criscotero/nlp-chatdash/tree/main/data/docs/"
    click Artifacts "https://github.com/criscotero/nlp-chatdash/tree/main/data/artifacts_notebook/"
    click Ingest "https://github.com/criscotero/nlp-chatdash/blob/main/app/pipeline/ingest.py"
    click Initialize "https://github.com/criscotero/nlp-chatdash/blob/main/app/pipeline/initialize.py"
    click TopicsPipeline "https://github.com/criscotero/nlp-chatdash/blob/main/app/pipeline/topics.py"
    click Registry "https://github.com/criscotero/nlp-chatdash/blob/main/app/pipeline/registry.py"
    click FE_Embeddings "https://github.com/criscotero/nlp-chatdash/blob/main/app/features/embeddings.py"
    click FE_Topics "https://github.com/criscotero/nlp-chatdash/blob/main/app/features/topics.py"
    click FE_Clustering "https://github.com/criscotero/nlp-chatdash/blob/main/app/features/clustering.py"
    click FE_DocsIndex "https://github.com/criscotero/nlp-chatdash/blob/main/app/features/docs_index.py"
    click RAG_Train "https://github.com/criscotero/nlp-chatdash/blob/main/app/rag/train.py"
    click RAG_Eval "https://github.com/criscotero/nlp-chatdash/blob/main/app/rag/eval.py"
    click RAG_Infer "https://github.com/criscotero/nlp-chatdash/blob/main/app/rag/infer.py"
    click RAG_Synth "https://github.com/criscotero/nlp-chatdash/blob/main/app/rag/synthesis.py"
    click FastAPI "https://github.com/criscotero/nlp-chatdash/blob/main/app/api/main.py"
    click Streamlit "https://github.com/criscotero/nlp-chatdash/blob/main/app/dashboard/dashboard_app.py"
    click Gradio "https://github.com/criscotero/nlp-chatdash/blob/main/app/ui/gradio_app.py"
    click CLI "https://github.com/criscotero/nlp-chatdash/tree/main/Makefile"
    click CLI "https://github.com/criscotero/nlp-chatdash/blob/main/main.py"
    click CLI "https://github.com/criscotero/nlp-chatdash/blob/main/app/cli.py"
    click Config "https://github.com/criscotero/nlp-chatdash/blob/main/app/config.py"
    click Utils_Paths "https://github.com/criscotero/nlp-chatdash/blob/main/app/utils/paths.py"
    click Utils_Logging "https://github.com/criscotero/nlp-chatdash/blob/main/app/utils/logging.py"
    click Utils_Hash "https://github.com/criscotero/nlp-chatdash/blob/main/app/utils/hash.py"
    click Utils_Text "https://github.com/criscotero/nlp-chatdash/blob/main/app/utils/text.py"

    %% Styling
    classDef storage fill:#D0E6FF,stroke:#0366D6,color:#0366D6;
    classDef pipeline fill:#E6F4EA,stroke:#1B7F3B,color:#1B7F3B;
    classDef model fill:#FFF4D6,stroke:#D29922,color:#D29922;
    classDef service fill:#FFE6E6,stroke:#D14F4F,color:#D14F4F;
    classDef ui fill:#FFEBD6,stroke:#D87A2A,color:#D87A2A;
    classDef cli fill:#E8EAF6,stroke:#3F51B5,color:#3F51B5;
    classDef util fill:#F5F5F5,stroke:#9E9E9E,color:#616161;
```
