# shivp102/emp_policy_assistant architecture

> Architecture diagram of the GitHub repository shivp102/emp_policy_assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shivp102/emp_policy_assistant
- Repository: https://github.com/shivp102/emp_policy_assistant

## Overview

Employee Policy Assistant is a Streamlit chat application that grounds policy answers in uploaded or stored PDFs. Its principal workflow checks or rebuilds the index, retrieves and reranks relevant chunks for a question, then generates a cited answer and displays its sources. Separate product branches support policy uploads, Langfuse tracing, and offline retrieval evaluation. The README documents these capabilities; sampled source confirms the main ingestion, retrieval, chat, tracing, and evaluation orchestration, while some component details are documented but not sampled.

Main entry points: [`src/observability/rag_trace.py`](https://github.com/shivp102/emp_policy_assistant/blob/main/src/observability/rag_trace.py) (RAG turn trace), [`src/ingestion/pipeline.py`](https://github.com/shivp102/emp_policy_assistant/blob/main/src/ingestion/pipeline.py) (Ingestion pipeline), [`app/streamlit_app.py`](https://github.com/shivp102/emp_policy_assistant/blob/main/app/streamlit_app.py) (Streamlit app), [`src/evaluation/runner.py`](https://github.com/shivp102/emp_policy_assistant/blob/main/src/evaluation/runner.py) (Evaluation runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Policy ingestion"]
  node_pdfs["Policy PDFs"]
  node_ingest["Ingestion pipeline<br/>[pipeline.py]"]
  node_loader["PDF loader<br/>[document_loader.py]"]
  node_chunker["Policy chunker<br/>[chunker.py]"]
  node_embedder["HuggingFace embedder<br/>[embedder.py]"]
  node_state["Ingest state<br/>[ingest_state.py]"]
  node_manifest["Ingest manifest"]
  node_store["Chroma store<br/>[chroma_store.py]"]
  node_chroma[("ChromaDB index")]
end

subgraph group_retrieval["Search and answers"]
  node_retrieval["Retrieval pipeline<br/>[pipeline.py]"]
  node_retriever["Vector retriever<br/>[retriever.py]"]
  node_reranker["Cross-encoder reranker<br/>[reranker.py]"]
  node_generator["Answer generator"]
end

subgraph group_interface["Chat interface"]
  node_app["Streamlit app<br/>[streamlit_app.py]"]
  node_upload["PDF uploads<br/>[pdf_upload.py]"]
end

subgraph group_observability["Tracing"]
  node_trace["RAG turn trace<br/>[rag_trace.py]"]
  node_lfclient["Langfuse client<br/>[langfuse_client.py]"]
  node_langfuse[("Langfuse Cloud")]
end

subgraph group_evaluation["Evaluation"]
  node_dataset["Policy QA dataset<br/>[policy_qa_v1.jsonl]"]
  node_evalrunner["Evaluation runner<br/>[runner.py]"]
  node_metrics["Retrieval metrics<br/>[metrics.py]"]
end

node_user(("Employee"))
node_openai(("OpenAI model"))

node_user -->|"asks policy"| node_app
node_app -->|"saves uploads"| node_upload
node_upload -->|"writes PDFs"| node_pdfs
node_app -->|"checks and rebuilds"| node_ingest
node_ingest -->|"loads PDFs"| node_loader
node_loader -->|"reads"| node_pdfs
node_ingest -->|"chunks pages"| node_chunker
node_ingest -->|"rebuilds index"| node_store
node_ingest -->|"checks and saves fingerprints"| node_state
node_state -->|"reads and writes"| node_manifest
node_store -->|"uses embeddings"| node_embedder
node_store -->|"persists chunks"| node_chroma
node_app -->|"runs chat turn"| node_trace
node_trace -->|"retrieves context"| node_retrieval
node_trace -->|"generates answer"| node_generator
node_retrieval -->|"gets candidates"| node_retriever
node_retrieval -->|"reranks candidates"| node_reranker
node_retriever -->|"embeds query"| node_embedder
node_retriever -->|"searches vectors"| node_store
node_generator -->|"invokes model"| node_openai
node_trace -.->|"configures and flushes"| node_lfclient
node_trace -.->|"records trace and spans"| node_langfuse
node_lfclient -.->|"gets client"| node_langfuse
node_app -->|"shows answer and sources"| node_user
node_dataset -->|"provides questions"| node_evalrunner
node_evalrunner -->|"evaluates retrieval"| node_retrieval
node_evalrunner -.->|"runs optional full RAG"| node_trace
node_evalrunner -->|"scores results"| node_metrics
node_evalrunner -.->|"runs experiment"| node_langfuse

click node_app "https://github.com/shivp102/emp_policy_assistant/blob/main/app/streamlit_app.py"
click node_upload "https://github.com/shivp102/emp_policy_assistant/blob/main/src/utils/pdf_upload.py"
click node_pdfs "https://github.com/shivp102/emp_policy_assistant/tree/main/data/company_policies"
click node_ingest "https://github.com/shivp102/emp_policy_assistant/blob/main/src/ingestion/pipeline.py"
click node_loader "https://github.com/shivp102/emp_policy_assistant/blob/main/src/ingestion/document_loader.py"
click node_chunker "https://github.com/shivp102/emp_policy_assistant/blob/main/src/ingestion/chunker.py"
click node_embedder "https://github.com/shivp102/emp_policy_assistant/blob/main/src/ingestion/embedder.py"
click node_state "https://github.com/shivp102/emp_policy_assistant/blob/main/src/utils/ingest_state.py"
click node_store "https://github.com/shivp102/emp_policy_assistant/blob/main/src/vector_store/chroma_store.py"
click node_trace "https://github.com/shivp102/emp_policy_assistant/blob/main/src/observability/rag_trace.py"
click node_lfclient "https://github.com/shivp102/emp_policy_assistant/blob/main/src/observability/langfuse_client.py"
click node_retrieval "https://github.com/shivp102/emp_policy_assistant/blob/main/src/retrieval/pipeline.py"
click node_retriever "https://github.com/shivp102/emp_policy_assistant/blob/main/src/retrieval/retriever.py"
click node_reranker "https://github.com/shivp102/emp_policy_assistant/blob/main/src/retrieval/reranker.py"
click node_generator "https://github.com/shivp102/emp_policy_assistant/blob/main/src/generation/answer_generator.py"
click node_dataset "https://github.com/shivp102/emp_policy_assistant/blob/main/eval/datasets/policy_qa_v1.jsonl"
click node_evalrunner "https://github.com/shivp102/emp_policy_assistant/blob/main/src/evaluation/runner.py"
click node_metrics "https://github.com/shivp102/emp_policy_assistant/blob/main/src/evaluation/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pdfs,node_ingest,node_loader,node_chunker,node_embedder,node_state,node_manifest,node_store,node_chroma toneBlue
class node_retrieval,node_retriever,node_reranker,node_generator toneAmber
class node_app,node_upload toneMint
class node_trace,node_lfclient,node_langfuse toneRose
class node_dataset,node_evalrunner,node_metrics,node_user,node_openai toneIndigo
```

## Components

### Policy ingestion

- [Policy PDFs](https://github.com/shivp102/emp_policy_assistant/tree/main/data/company_policies): `data/company_policies`
- [Ingestion pipeline](https://github.com/shivp102/emp_policy_assistant/blob/main/src/ingestion/pipeline.py): `src/ingestion/pipeline.py`
- [PDF loader](https://github.com/shivp102/emp_policy_assistant/blob/main/src/ingestion/document_loader.py): `src/ingestion/document_loader.py`
- [Policy chunker](https://github.com/shivp102/emp_policy_assistant/blob/main/src/ingestion/chunker.py): `src/ingestion/chunker.py`
- [HuggingFace embedder](https://github.com/shivp102/emp_policy_assistant/blob/main/src/ingestion/embedder.py): `src/ingestion/embedder.py`
- [Ingest state](https://github.com/shivp102/emp_policy_assistant/blob/main/src/utils/ingest_state.py): `src/utils/ingest_state.py`
- **Ingest manifest**
- [Chroma store](https://github.com/shivp102/emp_policy_assistant/blob/main/src/vector_store/chroma_store.py): `src/vector_store/chroma_store.py`
- **ChromaDB index**

### Search and answers

- [Retrieval pipeline](https://github.com/shivp102/emp_policy_assistant/blob/main/src/retrieval/pipeline.py): `src/retrieval/pipeline.py`
- [Vector retriever](https://github.com/shivp102/emp_policy_assistant/blob/main/src/retrieval/retriever.py): `src/retrieval/retriever.py`
- [Cross-encoder reranker](https://github.com/shivp102/emp_policy_assistant/blob/main/src/retrieval/reranker.py): `src/retrieval/reranker.py`
- [Answer generator](https://github.com/shivp102/emp_policy_assistant/blob/main/src/generation/answer_generator.py): `src/generation/answer_generator.py`

### Chat interface

- [Streamlit app](https://github.com/shivp102/emp_policy_assistant/blob/main/app/streamlit_app.py): `app/streamlit_app.py`
- [PDF uploads](https://github.com/shivp102/emp_policy_assistant/blob/main/src/utils/pdf_upload.py): `src/utils/pdf_upload.py`

### Tracing

- [RAG turn trace](https://github.com/shivp102/emp_policy_assistant/blob/main/src/observability/rag_trace.py): `src/observability/rag_trace.py`
- [Langfuse client](https://github.com/shivp102/emp_policy_assistant/blob/main/src/observability/langfuse_client.py): `src/observability/langfuse_client.py`
- **Langfuse Cloud**

### Evaluation

- [Policy QA dataset](https://github.com/shivp102/emp_policy_assistant/blob/main/eval/datasets/policy_qa_v1.jsonl): `eval/datasets/policy_qa_v1.jsonl`
- [Evaluation runner](https://github.com/shivp102/emp_policy_assistant/blob/main/src/evaluation/runner.py): `src/evaluation/runner.py`
- [Retrieval metrics](https://github.com/shivp102/emp_policy_assistant/blob/main/src/evaluation/metrics.py): `src/evaluation/metrics.py`

### Other components

- **Employee**
- **OpenAI model**

## Connections

- Employee → Streamlit app: asks policy
- Streamlit app → PDF uploads: saves uploads
- PDF uploads → Policy PDFs: writes PDFs
- Streamlit app → Ingestion pipeline: checks and rebuilds
- Ingestion pipeline → PDF loader: loads PDFs
- PDF loader → Policy PDFs: reads
- Ingestion pipeline → Policy chunker: chunks pages
- Ingestion pipeline → Chroma store: rebuilds index
- Ingestion pipeline → Ingest state: checks and saves fingerprints
- Ingest state → Ingest manifest: reads and writes
- Chroma store → HuggingFace embedder: uses embeddings
- Chroma store → ChromaDB index: persists chunks
- Streamlit app → RAG turn trace: runs chat turn
- RAG turn trace → Retrieval pipeline: retrieves context
- RAG turn trace → Answer generator: generates answer
- Retrieval pipeline → Vector retriever: gets candidates
- Retrieval pipeline → Cross-encoder reranker: reranks candidates
- Vector retriever → HuggingFace embedder: embeds query
- Vector retriever → Chroma store: searches vectors
- Answer generator → OpenAI model: invokes model
- RAG turn trace → Langfuse client: configures and flushes
- RAG turn trace → Langfuse Cloud: records trace and spans
- Langfuse client → Langfuse Cloud: gets client
- Streamlit app → Employee: shows answer and sources
- Policy QA dataset → Evaluation runner: provides questions
- Evaluation runner → Retrieval pipeline: evaluates retrieval
- Evaluation runner → RAG turn trace: runs optional full RAG
- Evaluation runner → Retrieval metrics: scores results
- Evaluation runner → Langfuse Cloud: runs experiment