skandharaj1307/hybridrag

Purpose and Entry Points

This repository implements a retrieval-augmented analysis pipeline over software-quality artifacts: structured metadata plus logs, screenshots/OCR, SRS PDFs, defects, reports, and test cases. Primary operational entry points are main.py, index_qdrant.py, query_pipeline.py, evaluate_retrieval.py, and sync_views.py.

Source Data and Parsing Boundary

Raw corpus material resides under dataset/structured and dataset/unstructured. Format-specific parsers normalize unstructured inputs before downstream chunking:

• Logs: parsers/log_parser.py, dataset/unstructured/failure_logs

• SRS PDF: parsers/pdf_parser.py, dataset/unstructured/srs/srs_document.pdf

• Screenshot/OCR data: parsers/screenshot_parser.py, dataset/unstructured/screenshots

• Structured records loaded from Supabase: parsers/supabase_loader.py

This separates source-specific extraction from retrieval-oriented document construction.

Content Chunking and Metadata

Artifact-specific chunkers convert parsed content into retrievable units, preserving the semantic structure appropriate to each source type:

• chunkers/log_chunker.py

• chunkers/srs_chunker.py

• chunkers/screenshot_chunker.py

• chunkers/defect_chunker.py

• chunkers/report_chunker.py

• chunkers/testcase_chunker.py

metadata/metadata_builder.py constructs metadata attached to chunks; utils/id_generator.py supports stable identifiers. The checked-in dataset/structured/metadata.json appears to be a structured metadata artifact.

Indexing and Vector Infrastructure

index_qdrant.py is the indexing stage that likely embeds chunks and writes them into Qdrant. Vector-store connection management is isolated in retrieval/qdrant_client_provider.py; Qdrant querying is implemented in retrieval/qdrant_retriever.py.

Embedding is an explicit abstraction in retrieval/embedder.py, keeping vector-model selection separate from parsing, chunking, and query orchestration. Qdrant is the material retrieval infrastructure dependency.

Hybrid Retrieval and Query Control

Retrieval combines lexical and vector search:

• BM25 path: retrieval/bm25_retriever.py

• Dense/Qdrant path: retrieval/qdrant_retriever.py

• Combination logic: retrieval/hybrid_retriever.py

• Query dispatch/routing: retrieval/query_router.py

query_pipeline.py is the principal query-time coordinator: it accepts a query, routes or retrieves relevant context, and passes that context to answer-generation or execution stages. evaluate_retrieval.py assesses retrieval quality, with question inputs available in vanilla_rag_questions.txt.

LLM and Structured-Query Boundary

llm/mistral_service.py provides the generative-model integration, making Mistral an external inference service/runtime dependency. llm/sql_executor.py establishes a distinct path for executing SQL, likely when a routed request requires structured data rather than—or alongside—document retrieval.

This boundary is important: retrieval provides grounded text context, while SQL execution accesses structured information through a controlled executor rather than directly from LLM output.

Supabase Data Layer and View Synchronization

Supabase is the structured-data backend. supabase/migrations/20260801221712_create_context_views.sql defines database context views, while sync_views.py synchronizes or materializes them for pipeline use. parsers/supabase_loader.py is the bridge from those database records into the corpus/indexing flow.

Runtime configuration is centralized in config/config.py. The project is Python-based, as indicated by pyproject.toml, requirements.txt, and .python-version.