How salam59/django-slack-rag-bot works
Project type and purpose
• This is a monolithic Django-based backend service exposing a Slack Bot interface. It integrates a Retrieval-Augmented Generation (RAG) pipeline to answer questions about system design by extracting images and text from a PDF, embedding that content, indexing it in MongoDB, retrieving relevant chunks, and composing answers with an LLM.
File structure
• Top-level:
• README.md, requirements.txt, docker-compose.yaml, manage.py, db.sqlite3 (local fallback)
• src/app: the Django project (“settings.py”, “urls.py”, ASGI/WSGI entrypoints, Celery setup)
• src/bot: a Django app housing models, views (incoming Slack event endpoint), tasks (PDF parsing, indexing, RAG retrieval), utils, admin, tests, migrations
• src/slacky: Slack messaging helper library
• src/helpers: environment loader
• src/data: static assets & PDF sources, including the system-design PDF and images used in answers
• src/notebooks: exploratory Jupyter workflows (pdf mining, setup)
README for additional insights
• Tech stack: Django, Cloudflare Tunnels, Celery (broker + result backend via env vars), pdfminer.six for PDF extraction, jina.ai for embeddings, Mistral AI for generation, MongoDB Atlas with vector search, Slack API
Diagramming the architecture
a. Main components
• Slack platform (user encounters)
• Cloudflare Tunnel (tunnels external traffic to local/hosted Django)
• Django Web Server (ASGI/WSGI)
• Celery Worker(s)
• Celery Broker & Result Backend (e.g., Redis/RabbitMQ as per CELERY_* vars)
• PDF Processor (pdfminer.six)
• Embedding Service (jina.ai embedding API)
• Vector Store (MongoDB Atlas with vector search index)
• LLM Generation Service (Mistral AI API)
• Data Storage (src/data folder assets, SQLite fallback)
• Slacky Helper (utility in code to abstract Slack message formatting)
b. Relationships and interactions
1. User sends a message in Slack → Slack Events API → Cloudflare Tunnel → Django “/slack/events/” endpoint (views.py)
2. Django view enqueues a Celery task (bot/tasks.py) for the RAG pipeline
3. Celery Worker pulls task:
i. If first-time or updated, parse PDF via pdfminer.six → produce text & images → send embedding requests to jina.ai → index vectors into MongoDB
ii. For a query: embed query text via jina.ai → perform vector search in MongoDB → retrieve top-k chunks (text + images)
iii. Send retrieved context + user query to Mistral AI → receive generated answer
4. Celery Worker uses Slacky.messages to post response (text + images) back to Slack via Slack Web API
5. Django responds to Slack Events API call with HTTP 200 to acknowledge
c. Architectural patterns & design principles
• Separation of concerns: Web/API layer vs. background processing (Celery)
• RAG pipeline broken into modular tasks: PDF parsing, embedding, indexing, retrieval, generation
• Externalized services: LLM, embedding service, vector DB, Slack API — keep core logic thin
• Use of Django apps to isolate bot logic from project settings
d. Key technologies & frameworks
• Django (REST endpoint, settings management, URL routing)
• Celery (task queue, broker & result backend via environment)
• pdfminer.six (PDF text + image extraction)
• jina.ai (text embedding service)
• MongoDB Atlas (vector search index)
• Mistral AI (LLM completion API)
• Cloudflare Tunnel (expose local dev server)
• Slack API / Slacky.messages (message and file upload flows)