How palani1munusamy/javadocrag works
JavaDoc RAG ingests Oracle JDK 25 API documentation (and optional PDFs), then serves cited developer answers through a hybrid retrieval pipeline. Ingestion discovers and fetches pages, parses them into structured sections, chunks content, and builds text and image indexes. At query time, the API parses symbols and intent, performs direct symbol lookup or hybrid retrieval, and asks Claude to formulate a cited response. The graph also includes suggestions, chat, and image-assisted queries. Some source excerpts are partial; edges for unsampled internals follow explicit README documentation.
Main entry points: ingest.py Ingest CLI · serve.py FastAPI Service · hybrid_retriever.py Hybrid Retriever · db_store.py SQLite Metadata
Components 18
Ingestion
- Ingest CLI
src/cli/ingest.py - Javadoc Crawler
src/crawler/crawler.py - HTML Parser
src/crawler/parser.py - PDF Parser
src/crawler/pdf_parser.py - Section Chunker
src/chunker/chunker.py
Knowledge Stores
- SQLite Metadata
src/indexer/db_store.py - Text Embedder
src/indexer/embedder.py - BM25 Index
src/indexer/bm25_index.py - FAISS Index
src/indexer/faiss_store.py - Image Embedder
src/indexer/image_embedder.py
Query and Retrieval
- Query Parser
src/query/query_parser.py - Hybrid Retriever
src/indexer/hybrid_retriever.py
Answer Service
- FastAPI Service
src/cli/serve.py - Answer Generator
src/generator/generator.py - Web Interface
static/index.html
Other components
- Developer
- Oracle JDK Docs
- Anthropic Claude
Connections 32
- Developer → FastAPI Service: asks questions
- Developer → Web Interface: uses interface
- Web Interface → FastAPI Service: submits queries
- Ingest CLI → Javadoc Crawler: runs crawl
- Javadoc Crawler → Oracle JDK Docs: fetches pages
- Javadoc Crawler → SQLite Metadata: stores HTML
- Ingest CLI → SQLite Metadata: reads pages
- Ingest CLI → HTML Parser: parses HTML
- Ingest CLI → PDF Parser: parses PDFs
- Ingest CLI → Section Chunker: chunks sections
- HTML Parser → Section Chunker: returns sections
- PDF Parser → Section Chunker: returns sections
- Ingest CLI → SQLite Metadata: writes chunks
- Ingest CLI → Text Embedder: embeds chunks
- Ingest CLI → BM25 Index: builds index
- Ingest CLI → FAISS Index: builds vectors
- Ingest CLI → Image Embedder: embeds images
- Ingest CLI → FAISS Index: stores image vectors
- FastAPI Service → Query Parser: parses question
- Query Parser → FastAPI Service: returns symbols
- FastAPI Service → SQLite Metadata: looks up symbols
- FastAPI Service → Hybrid Retriever: requests matches
- Hybrid Retriever → BM25 Index: searches terms
- Hybrid Retriever → Text Embedder: embeds query
- Hybrid Retriever → FAISS Index: searches vectors
- Hybrid Retriever → SQLite Metadata: loads chunks
- FastAPI Service → Hybrid Retriever: searches images
- Hybrid Retriever → Image Embedder: embeds query image
- Hybrid Retriever → FAISS Index: searches image vectors
- FastAPI Service → Answer Generator: supplies context
- Answer Generator → Anthropic Claude: generates answer
- Answer Generator → FastAPI Service: returns citations