How trailblazer09/referencerover works
This is a command-line PDF RAG evaluation harness. A user supplies a question or requests an evaluation run; the CLI loads PDFs, indexes their text, retrieves relevant chunks, and passes the question and context to an LLM. Results include references, grounding status, and per-call token and latency metrics. The graph separates ingestion/indexing, question answering, and external services. Some implementation is unsampled, so ingestion and CLI wiring also rely on the README’s description; uncertain internal calls are omitted.
Main entry points: rag_evaluation.py CLI workflow · main.py CLI entry · pdfs Source PDFs · evaluation_questions.json Evaluation questions
Components 16
Ingestion and indexing
- Source PDFs
pdfs - PDF extraction
rag_evaluation.py - Text chunking
rag_evaluation.py - Embeddings
rag_evaluation.py - Document indexing
rag_evaluation.py
Answering and evaluation
- CLI workflow
rag_evaluation.py - Evaluation questions
evaluation_questions.json - Question evaluation
rag_evaluation.py - Semantic search
rag_evaluation.py - Grounding check
rag_evaluation.py - LLM call
rag_evaluation.py - Answers and metrics
External services
- Qdrant
- Gemini or Groq
Other components
- User
- CLI entry
main.py
Connections 18
- User → CLI entry: runs
- CLI entry → CLI workflow: invokes
- CLI workflow → PDF extraction: loads PDFs
- Source PDFs → PDF extraction: provides files
- PDF extraction → Text chunking: extracts text
- CLI workflow → Document indexing: indexes documents
- Document indexing → Embeddings: embeds chunks
- Document indexing → Qdrant: writes vectors
- CLI workflow → Question evaluation: evaluates questions
- Evaluation questions → CLI workflow: supplies questions
- Question evaluation → Semantic search: retrieves context
- Semantic search → Embeddings: embeds query
- Semantic search → Qdrant: searches vectors
- Question evaluation → LLM call: requests answer
- LLM call → Gemini or Groq: sends prompt
- Question evaluation → Grounding check: checks context
- Question evaluation → Answers and metrics: returns answer and metrics
- CLI workflow → Answers and metrics: prints references