# baejoseph/cis-retriever architecture

> Architecture diagram of the GitHub repository baejoseph/cis-retriever, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/baejoseph/cis-retriever
- Repository: https://github.com/baejoseph/cis-retriever

## Overview

##### Project type and purpose
- RAG-Lite is a full-stack document Q&A application built in Python with a Streamlit front-end. It implements Retrieval-Augmented Generation (RAG) over user-uploaded DOCX files, performing hierarchical chunking, semantic retrieval, and evidence-based response generation via OpenAI (or local LLMs).

##### File structure
- Top-level files:
- config.yaml, env.template – configuration and environment variables
- pyproject.toml, uv.lock – dependency & lock files
- README.md, rag_prompt.md – documentation
- src/ directory:
- app.py – Streamlit UI → orchestrates uploads, slider controls, chat interface
- parser.py & ingester.py – document ingestion, hierarchical chunking, duplicate detection
- rag_pipeline.py – core RAG workflow: retrieve chunks → call LLM → assemble evidence citations
- openai_services.py – abstraction over external LLM & embedding APIs (OpenAI, Ollama)
- logger.py & log_time.py – centralized logging and timing decorators
- helpers.py – utility functions
- tests/ directory: YAML-driven retrieval tests and sample DOCX files
- images/ – UI assets

##### Key architecture insights from README
- Three logical layers: UI (Streamlit), Business Logic (RAG pipeline & parser), External Services (LLM & embeddings)
- Caching layer for processed documents and embeddings
- Dependency injection for testability
- Configurable retrieval parameters (top-K, similarity threshold)

##### Main components
a. Frontend (Streamlit UI)
b. Backend Application Layer
- Document Ingestion & Chunking Service
- Retrieval & Ranking Service (vector store + cosine similarity)
- RAG Orchestrator (rag_pipeline)
- OpenAI Services (LLM & embeddings)
c. Caching / Storage
- Processed documents cache
- Embedding / vector index store
d. External Dependencies
- OpenAI API or Local LLM (Ollama)
- AWS or other persistence if used (S3, Redis, etc.)
e. Configuration & Logging
- config.yaml / .env
- logger service

##### Relationships and data flow
1. User uploads DOCX via Streamlit → Streamlit UI
2. UI invokes Document Ingestion service → parser/ingester
3. Ingestion produces hierarchical chunks → stored in cache & vector store
4. User asks a question → UI passes query to RAG Orchestrator
5. RAG Orchestrator retrieves top-K chunks from vector store (cosine sim)
6. Orchestrator calls OpenAI Services for answer generation + citations
7. Response returned to UI

##### Architectural patterns & design principles
- Layered architecture: clear separation UI / Application / External Services
- Modular, testable services with dependency injection
- Caching for idempotent ingestion & vector lookups
- Config-driven parameters (chunk size, thresholds)

##### Final reminders
- Don’t omit the caching layer—it’s central to performance.
- Both ingestion and query time data flows.
- Keep labels concise but unambiguous; avoid generic terms like “Module.”

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "UI Layer"
        UI["Streamlit UI"]:::frontend
        IMAGES["UI Assets"]:::assets
    end

    subgraph "Backend Application"
        PARSER["Parser Service"]:::application
        INGESTER["Ingestion Service"]:::application
        RAG["RAG Orchestrator & Retrieval Service"]:::application
    end

    subgraph "Caching / Storage"
        PROCESSED_CACHE["Processed Documents Cache"]:::storage
        VECTOR_STORE["Embedding / Vector Store"]:::storage
    end

    subgraph "External Services"
        OAI["OpenAI & Local LLM API"]:::external
    end

    subgraph "Shared Services"
        HELPERS["Helpers Utility"]:::shared
        LOGGER["Logger Service"]:::shared
        LOGTIME["Timing Decorators"]:::shared
    end

    subgraph "Configuration"
        CONFIG_FILE["config.yaml"]:::config
        ENV_FILE["env.template"]:::config
    end

    subgraph "Tests & Samples"
        TEST_YAML["Retrieval Tests"]:::tests
        TEST_DOCS["Sample DOCX Files"]:::tests
    end

    %% Data Flows
    UI -->|"upload DOCX\nHTTP/JSON"| PARSER
    PARSER -->|"extract chunks"| INGESTER
    INGESTER -->|"store chunks"| PROCESSED_CACHE
    INGESTER -->|"generate embeddings"| VECTOR_STORE

    UI -->|"user query\nHTTP/JSON"| RAG
    RAG -->|"retrieve top-K\ncosine similarity"| VECTOR_STORE
    RAG -->|"call LLM API\nJSON"| OAI
    OAI -->|"response\nJSON"| RAG
    RAG -->|"answer + citations"| UI

    %% Config & Utils
    CONFIG_FILE -->|"params"| PARSER
    ENV_FILE -->|"env vars"| RAG
    HELPERS --> PARSER
    HELPERS --> INGESTER
    HELPERS --> RAG
    LOGGER -.-> PARSER
    LOGGER -.-> INGESTER
    LOGGER -.-> RAG
    LOGTIME -.-> PARSER
    LOGTIME -.-> INGESTER
    LOGTIME -.-> RAG

    TEST_YAML --> PARSER
    TEST_YAML --> INGESTER

    TEST_DOCS --> PARSER

    IMAGES --> UI

    %% Click Events
    click UI "https://github.com/baejoseph/cis-retriever/blob/main/src/app.py"
    click PARSER "https://github.com/baejoseph/cis-retriever/blob/main/src/parser.py"
    click INGESTER "https://github.com/baejoseph/cis-retriever/blob/main/src/ingester.py"
    click RAG "https://github.com/baejoseph/cis-retriever/blob/main/src/rag_pipeline.py"
    click OAI "https://github.com/baejoseph/cis-retriever/blob/main/src/openai_services.py"
    click HELPERS "https://github.com/baejoseph/cis-retriever/blob/main/src/helpers.py"
    click LOGGER "https://github.com/baejoseph/cis-retriever/blob/main/src/logger.py"
    click LOGTIME "https://github.com/baejoseph/cis-retriever/blob/main/src/log_time.py"
    click CONFIG_FILE "https://github.com/baejoseph/cis-retriever/blob/main/config.yaml"
    click ENV_FILE "https://github.com/baejoseph/cis-retriever/blob/main/env.template"
    click TEST_YAML "https://github.com/baejoseph/cis-retriever/blob/main/tests/retrieval_tests.yaml"
    click TEST_DOCS "https://github.com/baejoseph/cis-retriever/tree/main/tests/documents/"
    click IMAGES "https://github.com/baejoseph/cis-retriever/tree/main/images/"

    %% Styles
    classDef frontend fill:#dae8fc,stroke:#6c8ebf,color:#1a237e;
    classDef application fill:#d5e8d4,stroke:#82b366,color:#274e13;
    classDef storage fill:#f8cecc,stroke:#b85450,color:#791e1e,shape:cylinder;
    classDef external fill:#e1d5e7,stroke:#9673a6,color:#4a235a,shape:cloud;
    classDef shared fill:#f5f5f5,stroke:#999999,color:#333333;
    classDef config fill:#fff2cc,stroke:#d6b656,color:#664d03;
    classDef tests fill:#fdebd0,stroke:#d89b5f,color:#5b2c06;
    classDef assets fill:#d0e0e3,stroke:#6a7f83,color:#1b2830;
```
