How fufankeji/fufan-chat-api works
1. System boundary and purpose
- This repo is a Python backend for an LLM-based Q&A platform, with a separately built Vue frontend served from static/dist.
- The production backend lives primarily under server/; the rest of the tree contains reusable data-processing code, RAG evaluation code, deployment assets, and playground demos.
2. Request entry and API surface
- Runtime starts from startup.py, which boots the backend app in server/main.py.
- HTTP routes are assembled in server/api_router.py and fan out into specialized chat endpoints for general chat, knowledge-base chat, search chat, agent chat, and recommendation chat.
- The backend is designed as a service layer around FastAPI-style request handling, with streaming callbacks for model responses.
3. Conversation and chat orchestration
- The core orchestration is in server/chat/.
- chat.py acts as the generic conversation entry point, while agent_chat.py, knowledge_base_chat.py, search_engine_chat.py, and recommendation_chat.py implement the different application scenarios.
- Conversation state and history are part of the request flow, so each chat mode can combine current user input with stored session context.
4. LLM integration layer
- server/model_workers/ abstracts model backends and keeps the app decoupled from a specific provider.
- The repo targets both local/open-source models and hosted APIs, with examples and fixes for GLM/ChatGLM-style models called out in the README.
- server/minx_chat_openai.py and server/embeddings_api.py indicate a provider-agnostic interface for chat generation and embeddings.
- Streaming generation is important in the architecture: model output is not treated as a single blocking response.
5. Persistence and application metadata
- server/db/ is the relational persistence layer, with SQLAlchemy-style models and repositories.
- It stores users, conversations, messages, knowledge bases, and knowledge files, which are the main business entities in the system.
- This database layer supports authentication/authorization, session history, and knowledge-base ownership boundaries.
6. Knowledge base subsystem
- server/knowledge_base/ manages local KB lifecycle: initialization, caching, document indexing, and retrieval service selection.
- There are two vector-store backends: Faiss and Milvus, surfaced through kb_service/faiss_kb_service.py and kb_service/milvus_kb_service.py.
- kb_cache/ indicates a cache layer so KB metadata and vector-store handles can be reused efficiently.
- This subsystem is the main boundary between uploaded/content files and RAG query-time retrieval.
7. Document ingestion and text processing
- Ingestion starts outside the main server in document_loaders/, text_splitter/, and data/parse/.
- PDF and other source documents are normalized, parsed, and split into chunks before indexing.
- The data/parse area also includes rule-based and unstructured parsing logic, showing that ingestion is not a simple file upload path but a preprocessing pipeline.
8. Retrieval-Augmented Generation flow
- For KB Q&A, the typical flow is: user query → retrieve relevant chunks from Faiss/Milvus → assemble prompt with history/context → generate answer through the selected model backend.
- Reranking is a first-class part of retrieval: server/reranker/ and the fufanrag/retriever/ code show that initial retrieval results are refined before generation.
- This RAG path is the main architecture for private knowledge-base question answering.
9. Web search / internet retrieval path
- The search scenario adds an external search provider and then re-indexes retrieved web content for downstream answering.
- server/agent/tools/duckgosearch.py and the README’s Serper/Google-search flow suggest that external search is treated as a tool-backed retrieval source rather than a separate app.
- This path differs from private KB Q&A by introducing an internet-fetch, extraction, rerank, and temporary indexing stage.
10. Agent subsystem and tool use
- server/agent/ contains the agent framework, tool selection, callback handling, and custom agent definitions.
- The agent path is boundary-driven: the model can call tools, receive tool outputs, and continue reasoning before final response generation.
- This subsystem is distinct from plain chat because it adds execution of external tools and tool-routing logic.
11. Recommendation scenario
- server/chat/recommendation_chat.py implements the recommendation-oriented conversational flow.
- Supporting data lives under scripts/course_data/, which suggests the recommendation system is built around educational content and user behavior/history.
- This is not a generic recommender microservice; it is integrated into the same conversational backend and uses chat history plus domain data.
12. RAG evaluation and offline experiments
- The fufanrag/ package is a self-contained evaluation/research pipeline, separate from the runtime API.
- It includes dataset handling, retrieval, generation, metrics, and evaluator logic, plus sample test/index/result artifacts under fufanrag/data/.
- This package is important architecturally because it provides the measurement loop for improving retrieval and generation quality, but it is not the live serving path.
13. Deployment and infrastructure
- docker/ provides containerization, and docs/ includes operational guidance for MySQL and Milvus.
- The system depends on MySQL for relational state and on a vector database layer for RAG; Faiss is used for local vector stores, while Milvus is used where a service-backed vector DB is needed.
- The architecture is therefore a multi-store design: relational DB for app state, vector store for semantic retrieval, and model endpoints for inference.
14. Non-production and example code
- playground/ contains many isolated demos for FastAPI, FastChat, LangChain, MySQL, and model deployment experiments.
- These are useful for understanding integration patterns, but they are not the core runtime graph of the product.
- For an architecture diagram, they should be shown as supporting examples, not as part of the serving path.
15. Key high-level data flow
- User request enters the API gateway/router, is routed to a chat mode, fetches session/user metadata from MySQL, optionally retrieves from a vector store or external search, optionally invokes tools through the agent layer, and finally calls the selected LLM backend to stream an answer back.
- Offline flows run in parallel: document parsing and preprocessing populate knowledge bases and indexes, while evaluation pipelines measure retrieval/generation quality.
Main entry points: api_router.py API router · minx_chat_openai.py Model API · knowledge_base_chat.py KB chat · chat.py Chat core
Components 23
Serving runtime
Live API, chat orchestration, persistence, model access, and retrieval.
- Startup · Bootstraps the backend process.
startup.py - Server app · Creates and runs the backend application.
server/main.py - API router · Assembles request routes and dispatches chat scenarios.
server/api_router.py - Chat core · Generic conversation entry point and shared chat utilities.
server/chat/chat.py - KB chat · Knowledge-base question answering orchestration.
server/chat/knowledge_base_chat.py - Search chat · Internet-search-backed answering flow.
server/chat/search_engine_chat.py - Agent chat · Tool-augmented reasoning and action loop.
server/chat/agent_chat.py - Reco chat · Conversational recommendation flow.
server/chat/recommendation_chat.py - Database layer · Models and repositories for users, conversations, messages, knowledge bases, and files.
server/db/base.py - Conversation memory · Buffers and loads conversation history for chat flows.
server/memory/conversation_db_buffer_memory.py - Model API · Provider-agnostic chat and embedding interfaces.
server/minx_chat_openai.py - Model workers · Pluggable model backends for local and hosted inference.
server/model_workers/base.py - KB services · Knowledge-base lifecycle, caching, and vector-store service selection.
server/knowledge_base/kb_service/base.py - KB cache · Caches KB metadata and vector-store handles.
server/knowledge_base/kb_cache/base.py - Reranker · Refines retrieval results before generation.
server/reranker/reranker.py - Agent tools · Search tool integration and tool selection for agents.
server/agent/tools_select.py
Offline pipelines
Ingestion, parsing, preprocessing, and evaluation tooling that support the runtime.
- Loaders · Parses source files into normalized documents.
document_loaders/interface.py - Parsing · Rule-based and unstructured document parsing pipeline.
data/parse/unstructured_processor.py - Text split · Splits parsed text into retrieval-friendly chunks.
text_splitter/chinese_recursive_text_splitter.py - RAG eval · Offline retrieval, generation, and metric evaluation workflow.
fufanrag/pipeline/pipeline.py
Supporting assets
Deployment, built frontend, docs, and playground examples.
- Docker · Containerization and compose deployment assets.
docker/Dockerfile - Frontend dist · Prebuilt frontend bundle served from static/dist.
static/dist/index.html - Playground · Integration demos and experimental entry points.
playground/test_startup/startupV1.py
Connections 29
- Startup → Server app: boots · Process startup enters the server application.
- Server app → API router: registers routes · The app exposes HTTP endpoints through the router.
- API router → Chat core: generic chat · Routes fan into the shared conversation entry point.
- API router → KB chat: kb qa · Routes KB question answering requests.
- API router → Search chat: web search · Routes internet-retrieval chat requests.
- API router → Agent chat: agent mode · Routes tool-using agent requests.
- API router → Reco chat: recommendation · Routes recommendation-oriented conversations.
- Chat core → Conversation memory: loads history · Shared chat flow combines current input with session context.
- Chat core → Database layer: stores state · Conversation state and business entities are persisted in MySQL.
- Chat core → Model API: streams response · Chat flow invokes model generation through provider-agnostic APIs.
- Model API → Model workers: uses backends · Chat and embedding APIs delegate to concrete model workers.
- KB chat → KB services: retrieves chunks · KB chat queries the vector-store service layer.
- KB chat → KB cache: reuses handles · KB metadata and store handles are cached for efficiency.
- KB chat → Reranker: refines results · Retrieved passages are reranked before prompt assembly.
- KB chat → Model API: generates answer · KB RAG answers are streamed from the model layer.
- Search chat → Agent tools: search tool · Web search is exposed as a tool-backed retrieval path.
- Search chat → Reranker: reranks web docs · Fetched content is refined before answering.
- Agent chat → Agent tools: selects tools · Agent flow chooses and executes tools.
- Agent chat → Model API: reasoning loop · The agent alternates tool outputs and model reasoning.
- Reco chat → Database layer: reads course data · Recommendation flow uses stored application data.
- Reco chat → Model API: responds · Recommendations are delivered through the same LLM layer.
- Loaders → Parsing: normalizes docs · Document loaders feed the parsing pipeline.
- Parsing → Text split: chunks text · Parsed documents are split for indexing.
- Text split → KB services: indexes KB · Chunked content becomes knowledge-base index material.
- RAG eval → KB services: tests retrieval · Offline evaluation measures retrieval quality.
- RAG eval → Model API: tests generation · Evaluation also exercises generation behavior.
- Docker → Startup: runs · Container deployment starts the backend entrypoint.
- Frontend dist → API router: calls api · The built frontend consumes the backend routes.
- Playground → Model API: experiments · Playground code demonstrates integration patterns.