How hss3107/ai_app works
Main Components
• Web Frontend – The Django templates (in the “app/templates/app” directory) provide user-facing pages (e.g., home, login, register, transcript_query).
• Django Application (Backend) – The “ai_project/ai_project” directory contains settings, URLs, WSGI, and ASGI configuration; “ai_project/app” contains models, views, middleware, and processors that implement business logic.
• AI / Document Processing Pipeline – This is represented by several directories:
– “processed_documents”, “transcripts”, “summaries”, and “content_maps” are used to store raw and processed document data.
– “ai_formatted_transcripts” and “Chain” (with JSON execution logs and a chain_logger module) capture AI chain execution details.
– “processors” (like smart_document_processor.py and nlp_resources.py) implement the AI logic and document handling.
• Vector Stores – The “vector_stores” folder holds many subdirectories that contain FAISS index files (index.faiss and index.pkl) for semantic search and similarity matching.
• Data Storage – The project uses SQLite (found as “db.sqlite3”) locally; however, other production databases might be used (as indicated by the presence of a “postgresql.crt”).
• Deployment and Infrastructure – The “docker” directory (with Prod Dockerfile, entrypoint.sh, and uwsgi.ini) and GitHub workflows outline the CI/CD/deployment pipeline.
Relationships and Data Flow
• The Web Frontend interacts with the Django backend using standard HTTP flows. User requests hit URL endpoints (defined in “urls.py”) and are routed to views.
• The Django backend (models, views, serializers, middleware) handles business logic, including user sessions, document uploads, and data processing.
• Once documents are uploaded (in the “media/temp_documents” or similar), they are processed into “processed_documents”, with detailed processing information logged.
• Transcripts and summaries are generated, and content maps are produced from the document contents.
• The AI processors (in “app/processors”) read formatted transcripts and automatically create embeddings. These embeddings are then stored and indexed using FAISS in the “vector_stores” directories. Data flows from document processing components to the vector store for retrieval during search queries.
• External services (e.g., video analysis indicated by “Video Analysis API.docx”) or additional AI modules might be invoked via chain execution (as logged in “Chain”).
• Use the Model-View-Controller (or Model-Template-View) pattern to represent Django’s separation of concerns.
• A clear separation between presentation (templates/front-end), business logic (views/controllers & middleware), and data persistence (database, file storage).
• A separate processing pipeline for AI tasks that includes document ingestion, processing (text extraction, summarization, embedding generation), and indexing (vector stores).
• Docker containerization and deployment components as part of an infrastructure layer.
Technologies, Frameworks, and Libraries
• Django (web framework)
• FAISS (vector similarity indexing)
• Python libraries for NLP/document processing (as evidenced by modules such as smart_document_processor.py and nlp_resources.py)
• Docker for deployment
• Likely additional libraries for logging, JSON processing, and possibly machine learning frameworks in the processors
– At the top, show the Client/Browser layer (with UI pages).
• URL Routing and Controllers (views)
• Template Rendering (static and dynamic content)
– Side-by-side or below, include two clusters:
• Data Layer: database (SQLite, potential PostgreSQL), media storage for PDFs and images.
• AI Processing Pipeline:
– Document Ingestion Module (from media/temp_documents)
– Processing & Formatting Module (processed_documents, transcripts, summaries, content_maps)
– NLP & Embedding Generation (processors, chain execution logs)
– Finally, a cluster for the Vector Stores (all FAISS indexes) that connect back to the AI pipeline.
– At the bottom or side, add an Infrastructure layer:
• Docker (Prod build, entrypoint) and CI/CD (GitHub workflows).