# jamwithai/arxiv-paper-curator architecture

> Architecture diagram of the GitHub repository jamwithai/arxiv-paper-curator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jamwithai/arxiv-paper-curator
- Repository: https://github.com/jamwithai/arxiv-paper-curator

## Overview

##### Project Type and Purpose
- This is a full-stack, production-grade RAG (Retrieval-Augmented Generation) system called “arXiv Paper Curator.”
- Purpose: fetch papers from arXiv, parse PDFs, store metadata/content, index for search, and expose a FastAPI-based research assistant API.

##### File Structure
- Top level: Dockerfile, compose.yml, Makefile, src/, airflow/, notebooks/, static/, tests/.
- airflow/: Dockerfile, entrypoint, requirements, dags/ defining ETL pipelines (arxiv_ingestion, arxiv_paper_ingestion).
- notebooks/: Week-1 infra setup, Week-2 arXiv integration guides.
- compose.yml orchestrates FastAPI, PostgreSQL, OpenSearch, Airflow, Ollama.

##### README Insights
- Phases: Week 1 sets up infra (FastAPI, Postgres, OpenSearch, Airflow, Ollama).
- Week 2 builds data ingestion pipeline (ArxivClient, PDFParserService, MetadataFetcher, Airflow DAGs).
- Future: full RAG pipeline with vector search, LLM orchestration, observability.
- Emphasizes Docker Compose, Python 3.12+, UV package manager, testing.

##### Elements
- Airflow → MetadataFetcher → arXiv API → PDFParser → Postgres → OpenSearch → FastAPI → Ollama

##### Detail
- Major modules (routers, services, repositories, db interfaces)
- Both control flow (Airflow triggers) and data flow (paper content, metadata, search queries)
- Design patterns (e.g., “Asynchronous I/O”, “Dependency Injection”)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Docker Compose
    DockerCompose["Docker Compose\n(compose.yml)"]:::container
    DockerCompose --> FastAPIContainer
    DockerCompose --> PostgreSQLContainer
    DockerCompose --> OpenSearchContainer
    DockerCompose --> AirflowContainer
    DockerCompose --> OllamaContainer

    %% FastAPI Container
    subgraph "FastAPI App" 
        direction TB
        FastAPIContainer["FastAPI\n(port 8000)"]:::container
        Main["main.py"]:::service
        PingRouter["ping Router"]:::service
        PapersRouter["papers Router"]:::service
        MetadataService["MetadataFetcher Service"]:::service
        ArxivClientFS["ArxivClient"]:::service
        PDFParserFS["PDFParser Service"]:::service
        DoclingFS["Docling Parser"]:::service
        OllamaClientFS["Ollama Client"]:::service
        PaperRepoFS["PaperRepository"]:::service
        Config["Configuration"]:::service
        Dependencies["Dependency Injection"]:::service
        Middlewares["Middlewares"]:::service
        Main --> PingRouter
        Main --> PapersRouter
        PapersRouter --> MetadataService
        PapersRouter --> OllamaClientFS
        MetadataService --> PaperRepoFS
        MetadataService --> PDFParserFS
        PDFParserFS --> DoclingFS
        MetadataService --> ArxivClientFS
        PaperRepoFS --> PostgreSQL[(PostgreSQL\n:5432)]:::db
        PapersRouter --> OpenSearch[(OpenSearch\n:9200,5601)]:::db
        PapersRouter --> OllamaClientFS
        OllamaClientFS --> OllamaServer
    end

    %% PostgreSQL Container
    subgraph "PostgreSQL Database"
        PostgreSQLContainer["PostgreSQL\n(port 5432)"]:::db
    end

    %% OpenSearch Container
    subgraph "OpenSearch Cluster"
        OpenSearchContainer["OpenSearch\n(ports 9200,5601)"]:::db
    end

    %% Airflow Container
    subgraph "Airflow"
        AirflowContainer["Airflow\n(port 8080)"]:::container
        AirflowScheduler["Scheduler & Webserver"]:::service
        ArxivIngestionDAG["arxiv_ingestion DAG"]:::service
        PaperIngestionDAG["arxiv_paper_ingestion DAG"]:::service
        AirflowScheduler --> ArxivIngestionDAG
        AirflowScheduler --> PaperIngestionDAG
        ArxivIngestionDAG --> MetadataService
    end

    %% Ollama Container
    subgraph "Ollama LLM Server"
        OllamaContainer["Ollama\n(port 11434)"]:::external
        OllamaServer["LLM Service"]:::external
    end

    %% External arXiv API
    arXivAPI(("arXiv API")):::external
    ArxivClientFS --> arXivAPI
    ArxivClientFS --> PDFParserFS

    %% Styles
    classDef container stroke-dasharray: 5 5, fill:#f0f0f0, stroke:#333;
    classDef service fill:#bbdefb, stroke:#333, stroke-width:1px;
    classDef db fill:#c8e6c9, stroke:#333, stroke-width:1px;
    classDef external fill:#ffe082, stroke:#333, stroke-width:1px;

    %% Click Events
    click Main "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/main.py"
    click PingRouter "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/routers/ping.py"
    click PapersRouter "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/routers/papers.py"
    click MetadataService "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/services/metadata_fetcher.py"
    click ArxivClientFS "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/services/arxiv/client.py"
    click PDFParserFS "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/services/pdf_parser/parser.py"
    click DoclingFS "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/services/pdf_parser/docling.py"
    click OllamaClientFS "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/services/ollama/client.py"
    click PaperRepoFS "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/repositories/paper.py"
    click PostgreSQLContainer "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/db/factory.py"
    click PostgreSQLContainer "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/db/interfaces/postgresql.py"
    click Config "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/config.py"
    click Dependencies "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/dependencies.py"
    click Middlewares "https://github.com/jamwithai/arxiv-paper-curator/blob/main/src/middlewares.py"
    click ArxivIngestionDAG "https://github.com/jamwithai/arxiv-paper-curator/blob/main/airflow/dags/arxiv_ingestion/tasks.py"
    click PaperIngestionDAG "https://github.com/jamwithai/arxiv-paper-curator/blob/main/airflow/dags/arxiv_paper_ingestion.py"
    click AirflowContainer "https://github.com/jamwithai/arxiv-paper-curator/tree/main/airflow/Dockerfile"
    click AirflowContainer "https://github.com/jamwithai/arxiv-paper-curator/blob/main/airflow/entrypoint.sh"
    click DockerCompose "https://github.com/jamwithai/arxiv-paper-curator/blob/main/compose.yml"
```
