How abhiramk6/research-paper-analyze works
1. Overall architecture
This is a LangGraph-orchestrated document-evaluation pipeline for research papers. The system takes either an arXiv URL or a local PDF, turns the paper into structured text, extracts checkable claims, retrieves external evidence, runs several grounded evaluators, and writes a Markdown report.
2. Entry points
app.py is the interactive Streamlit front end for submitting a paper and downloading results. pipeline.py is the core orchestration layer and defines the active evaluation flow: input → ingest → extract_claims → retrieve_evidence → evaluate_consistency → evaluate_grammar → evaluate_novelty → evaluate_fabrication → build_report.
3. Ingestion and parsing boundary
ingestion/downloader.py handles paper acquisition from arXiv or local file paths. parser/pdf_parser.py converts PDFs into a lightweight textual structure suitable for downstream processing. This boundary isolates raw document handling from the LLM-driven analysis stages.
4. Context management and chunking
chunker/token_chunker.py creates rolling token windows with overlap to stay under the model’s context limit. This is important because the system is designed around bounded, traceable LLM calls rather than passing the whole paper at once.
5. Claim extraction subsystem
claims/claim_extractor.py converts paper text into a reduced taxonomy of operational claims: contribution, result, comparison, factual, and method. This is the main semantic pivot in the pipeline: downstream evaluation operates on claims, not on the whole paper indiscriminately.
6. Evidence retrieval subsystem
retrieval/retriever.py coordinates external lookup using retrieval/query_builder.py and retrieval/evidence_ranker.py, with settings in config/retrieval.yaml. The README indicates a scholarly-first strategy with fallback to broader web search, so external evidence is treated as supporting context for checkable claims rather than as a generic search result dump.
7. Verification layer
verification/verifier_agent.py compares extracted claims against both paper-local context and retrieved evidence. This is the grounding boundary: it determines whether a claim is supported, contradicted, or remains uncertain before the later scoring agents synthesize higher-level judgments.
8. Evaluation agents
The agents/ directory contains specialized LLM-driven evaluators:
- consistency_agent.py for overall internal consistency
- grammar_agent.py for writing quality
- novelty_agent.py for literature-backed novelty synthesis
- credibility_agent.py for fabrication-risk synthesis
- assessment_synthesis_agent.py for combining assessment signals
agents/llm_client.py centralizes model access, likely to Gemini per the README and .env.example.
9. Scoring and model layer
scoring/credibility_model.py and models/schema.py define the structured outputs and scoring artifacts used across the pipeline. This layer provides the data contract between extraction, retrieval, verification, and report generation.
10. Caching and runtime resilience
utils/cache_manager.py suggests memoization or artifact reuse across runs, which matters because retrieval and LLM calls are expensive. The README also notes conservative fallback behavior when Gemini output or retrieval is unavailable, so the architecture is designed to degrade rather than fail noisily.
11. Reporting and outputs
reporter/report_writer.py writes the final Markdown report to reports/<paper_id>/report.md. This is the system’s durable output boundary: all upstream analysis is funneled into a human-readable audit artifact that includes scores like consistency, grammar, novelty, and fabrication risk.
12. Operational shape
The repo is a single-process application with two user-facing surfaces: Streamlit for interaction and file-based Markdown reports for persistence. External dependencies that materially shape the architecture are the LLM provider, PDF inputs, and scholarly/web retrieval sources; everything else is internal orchestration and transformation.
Main entry points: pipeline.py Pipeline · retriever.py Retriever · llm_client.py LLM client · credibility_agent.py Credibility
Components 21
User surface
Interactive entry point and persisted report output.
- App · Interactive front end for submitting papers and downloading results.
app.py
Core flow
Orchestration and major analysis stages for a paper run.
- Pipeline · Core LangGraph-style orchestration from input through report generation.
pipeline.py - Claims · Reduces paper text into checkable claim types such as contribution, result, comparison, factual, and method.
claims/claim_extractor.py - Retriever · Coordinates external evidence lookup for claims.
retrieval/retriever.py - Verifier · Compares claims against paper context and retrieved evidence to determine support status.
verification/verifier_agent.py
LLM evaluators
Specialized model-backed evaluators and shared client.
- Consistency · Evaluates overall internal consistency of the paper and extracted findings.
agents/consistency_agent.py - Grammar · Assesses writing quality and language clarity.
agents/grammar_agent.py - Novelty · Synthesizes literature-backed novelty judgments.
agents/novelty_agent.py - Credibility · Synthesizes fabrication-risk and credibility signals.
agents/credibility_agent.py - Synthesis · Combines assessment signals into a final structured judgment.
agents/assessment_synthesis_agent.py - LLM client · Centralized access layer for model calls used by agents and pipeline stages.
agents/llm_client.py
Support layer
Parsing, chunking, retrieval, caching, and schema contracts.
- Downloader · Acquires paper content from arXiv URLs or local file paths.
ingestion/downloader.py - PDF parser · Converts PDFs into lightweight structured text for downstream analysis.
parser/pdf_parser.py - Chunker · Creates overlapping token windows to keep model calls within context limits.
chunker/token_chunker.py - Query builder · Forms scholarly-first and fallback search queries for evidence lookup.
retrieval/query_builder.py - Ranker · Orders retrieved sources by relevance and usefulness for verification.
retrieval/evidence_ranker.py - Credibility model · Defines credibility and fabrication-risk scoring logic or structure.
scoring/credibility_model.py - Schema · Shared data contracts for extracted claims, evidence, and assessment artifacts.
models/schema.py - Cache · Memoizes or reuses artifacts across runs to reduce repeated retrieval and LLM cost.
utils/cache_manager.py - Retrieval config · Retrieval configuration for evidence lookup behavior.
config/retrieval.yaml
Artifacts
Run outputs and generated reports.
- Report · Example persisted evaluation report for a paper run.
reports/test-case/report.md
Connections 27
- App → Pipeline: submits paper · The UI hands user input to the orchestration layer.
- Pipeline → Downloader: ingest · Pipeline acquires paper source from URL or local file.
- Downloader → PDF parser: raw pdf · Downloaded paper is parsed into structured text.
- PDF parser → Chunker: paper text · Parsed text is windowed for bounded processing.
- Chunker → Claims: chunks · Chunked text feeds claim extraction.
- Claims → Schema: claim objects · Extracted claims are produced in shared structured form.
- Pipeline → Retriever: retrieve evidence · Pipeline requests external evidence for claims.
- Retriever → Query builder: build queries · Retrieval constructs search queries from claims and context.
- Retriever → Ranker: rank evidence · Candidate sources are scored and ordered.
- Retriever → Retrieval config: uses config · Retrieval behavior is driven by configuration settings.
- Retriever → Cache: reuse artifacts · Retrieval may reuse cached results across runs.
- Pipeline → Verifier: verify claims · Claims and retrieved evidence are checked for support or contradiction.
- Verifier → Schema: verification output · Verification produces structured grounding results.
- Pipeline → Consistency: evaluate consistency · Pipeline invokes the consistency evaluator.
- Pipeline → Grammar: evaluate grammar · Pipeline invokes the writing-quality evaluator.
- Pipeline → Novelty: evaluate novelty · Pipeline invokes the novelty synthesizer.
- Pipeline → Credibility: evaluate fabrication · Pipeline invokes the credibility and fabrication-risk evaluator.
- Consistency → LLM client: model call · Consistency assessment is generated through shared model access.
- Grammar → LLM client: model call · Grammar assessment is generated through shared model access.
- Novelty → LLM client: model call · Novelty synthesis is generated through shared model access.
- Credibility → Credibility model: scores risk · Fabrication-risk logic is grounded in the scoring model.
- Credibility → LLM client: model call · Credibility synthesis may use shared model access.
- Pipeline → Synthesis: combine signals · Pipeline merges evaluator outputs into a final assessment.
- Synthesis → Schema: final summary · Combined assessment is emitted in shared structured form.
- Pipeline → Report: write report · Final structured findings are rendered to a Markdown report.
- Pipeline → Cache: store run state · Pipeline may persist intermediate artifacts for reuse.
- App → Report: download output · The UI exposes the generated report to the user.