How phamxuankhang/vietnamese_law_assistant_chatbot works
Main Components
• Data Source & Raw Data Repository:
• The “BoPhapDienDienTu” directory contains the original legal texts (JSON and HTML files).
• Data Crawling and Insertion Module:
• Located in chatbot_v1/src/crawl_and_insert_to_db, this module is responsible for crawling the legal data, parsing content, and storing it in a MySQL database. It includes database connection handling (db.py), schema models (models.py), and utility functions.
• Embedding Generation Module:
• Found in chatbot_v1/src/embedding, this code preprocesses text and creates vector embeddings (using language models or Sentence Transformers). The embeddings are stored (possibly both in files under data/ and within Weaviate).
• LLM Response Module:
• In chatbot_v1/src/LLM_response, this subsystem integrates with external LLMs (like Google Palm) to generate responses. It includes modules for generating responses (LLM_generator.py/llm.py), routing requests (semantic_router.py), and retrieving context-specific data (retriever.py).
• Database Systems:
• A traditional MySQL database stores structured legal information as indicated by the detailed schema in the README.
• A vector database (Weaviate) is deployed via Docker (weaviate-docker/) to store and query text embeddings.
• REST API & Web Backend:
• The backend is built with Python & FastAPI. It exposes endpoints such as search (/api/search), detailed law article view (/api/articles/{article_id}), topics (/api/topics), and sections (/api/sections).
• Frontend (User Interface):
• The “user_interface” directory holds a React-based web UI with pages (e.g., Vietnamese_Law_Assistant_Chatbot_v1.py and v2.py) that interact with the backend API via HTTP requests.
• Additional Modules (chatbot_v2):
• There is a secondary version (chatbot_v2) with NER and relation creation modules. These address enhanced text processing, such as Named Entity Recognition for legal texts and creating relationships between legal entities.
Relationships and Data Flow
• The Raw Data (BoPhapDienDienTu) is first processed by the Crawl Module, which extracts and inserts structured data into the MySQL Database.
• The Embedding Module takes cleaned/processed text from the database (or from intermediate processed files) and produces vector representations that are inserted into the Weaviate vector database.
• The LLM Response Module then leverages the vector database to retrieve contextually similar legal documents and uses the LLM (Google Palm) to generate natural language responses.
• The FastAPI backend acts as an orchestrator, exposing these capabilities through REST API endpoints.
• Finally, the Frontend (built in React) sends requests to the FastAPI backend and displays response data (search results, law article details, etc.) to users.
Architectural Patterns and Design Principles
• Modularization: Each functionality (crawling, embedding, LLM response) is built as an independent module to ensure separation of concerns.
• Service Orientation: Although packaged together, the components (crawling, embedding, LLM response) can be seen as microservices that interact via defined interfaces.
• API-Driven Design: The FastAPI endpoints define clear contracts between the frontend and backend, and between services.
• Containerization: The Weaviate vector database is run in a Docker container, and other services can similarly be containerized for deployment.
Technologies, Frameworks, and Libraries
• Backend: Python (FastAPI, SQLAlchemy, PyMySQL)
• Data Processing: BeautifulSoup, embedding libraries (e.g., Sentence Transformers), LangChain
• Database: MySQL for structured legal data and Weaviate for vector operations.
• Frontend: React, Tailwind CSS, Axios, React Query
• DevOps: Docker, Docker Compose
Specific Elements
• Clear labels for each component (e.g., “Data Crawler”, “Embedding Service”, “LLM Handler”, “MySQL Database”, “Weaviate Vector DB”, “API Gateway (FastAPI)”, “React Frontend”).
• From the Data Crawler to MySQL.
• From the Embedding Service to Weaviate.
• From the FastAPI backend to both MySQL and Weaviate.
• Notations for key technologies (e.g., “Python 3.8”, “FastAPI”, “SQLAlchemy”, “Docker”, “Weaviate”, “React”) to provide context.
• For components like chatbot_v2, you might include sub-modules for “NER” and “Relation Creation” and show how they feed into the overall data workflow.