# alleninstitute/openai_tools architecture

> Architecture diagram of the GitHub repository alleninstitute/openai_tools, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alleninstitute/openai_tools
- Repository: https://github.com/alleninstitute/openai_tools

## Overview

##### Project type and purpose
- This is a Python‐based toolset and library (“papers_extractor”) for:
– Parsing and extracting text from scientific publication PDFs (and XML)
– Summarizing documents via the OpenAI ChatGPT API
– Embedding document text (PDFs or PubMed abstracts) with OpenAI embeddings
– Storing and caching results in a local on‐disk database to minimize repeated API calls
– Generating t-SNE visualizations (PNG or interactive HTML) of the embedding space
- The top‐level “scripts” folder provides three CLI entry‐points: pdf_summary.py, list_pdfs_embedding.py, pubmed_embedding.py

##### File structure
- src/papers_extractor – core library modules
– parsers: pdf_parser.py, xml_parser.py, database_parser.py, pubmed_papers_parser.py, arxiv_parser.py
– openai_parsers.py handles chunking, prompts, and calls to the OpenAI API
– multi_paper.py, unique_paper.py define data models and deduplication logic
– long_text.py splits text for multi‐chunk summarization
- scripts – user‐facing commands that wire together parsing, embedding, summarization, database, and plotting
- example – sample inputs (PDFs) and outputs (text, compressed text, HTML, PNG)
- tests – unit tests for each parser and model component
- docs – Sphinx documentation configuration
- setup.py, requirements.txt, dev_requirements.txt – packaging and CI

##### README for additional insights
- Database layer uses DiskCache to store raw text, summaries, embeddings keyed by PDF content hash
- External dependencies: OpenAI API, adjustText (for label placement), matplotlib/plotly for plots

##### Key takeaways
- Every major component and its role in the pipeline—parsing, caching, API, plotting
- Both the happy path (cache hit) and miss path (API call)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        PDF_SUMMARY["pdf_summary.py"]:::cli
        LIST_PDFS["list_pdfs_embedding.py"]:::cli
        PUBMED_EMB["pubmed_embedding.py"]:::cli
    end

    %% Core Library
    subgraph "Core Library"
        direction TB
        subgraph "Parsers"
            direction TB
            PDF_PARSER["PDF Parser"]:::core
            XML_PARSER["XML Parser"]:::core
            PUBMED_PARSER["PubMed Parser"]:::core
            ARXIV_PARSER["ArXiv Parser"]:::core
        end
        DB_PARSER["Database Parser"]:::core
        TEXT_CHUNKER["Text Chunker"]:::core
        OPENAI_CLIENT["OpenAI Client"]:::core
        MULTI_PAPER["Multi-paper Model"]:::core
        UNIQUE_PAPER["Unique-paper Model"]:::core
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        direction TB
        DISKCACHE["DiskCache DB"]:::persistence
        FILE_SYSTEM["Local File System"]:::persistence
    end

    %% Visualization Layer
    subgraph "Visualization Layer"
        direction TB
        TSNE["t-SNE Reducer"]:::viz
        PLOT_GEN["Plot Generator"]:::viz
    end

    %% External Services
    subgraph "External Services"
        direction TB
        OPENAI_API["OpenAI API"]:::external
        PUBMED_API["PubMed API"]:::external
        ARXIV_API["ArXiv API"]:::external
    end

    %% Connections
    PDF_SUMMARY -->|invokes| PDF_PARSER
    PDF_SUMMARY -->|invokes| OPENAI_CLIENT
    PDF_SUMMARY -->|invokes| TSNE
    PDF_SUMMARY -->|invokes| PLOT_GEN

    LIST_PDFS -->|invokes| PDF_PARSER
    LIST_PDFS -->|invokes| OPENAI_CLIENT
    LIST_PDFS -->|invokes| TSNE
    LIST_PDFS -->|invokes| PLOT_GEN

    PUBMED_EMB -->|invokes| PUBMED_PARSER
    PUBMED_EMB -->|invokes| OPENAI_CLIENT
    PUBMED_EMB -->|invokes| TSNE
    PUBMED_EMB -->|invokes| PLOT_GEN

    PDF_PARSER -->|raw text| DB_PARSER
    XML_PARSER -->|raw text| DB_PARSER
    PUBMED_PARSER -->|raw text| DB_PARSER
    ARXIV_PARSER -->|raw text| DB_PARSER

    DB_PARSER -->|check cache| DISKCACHE
    DISKCACHE -->|hit/miss info| DB_PARSER
    DB_PARSER -->|store/retrieve| FILE_SYSTEM

    DB_PARSER -->|on miss| OPENAI_CLIENT
    OPENAI_CLIENT -->|text chunk| OPENAI_API
    OPENAI_API -->|response| OPENAI_CLIENT

    OPENAI_CLIENT -->|summaries/embeddings| DISKCACHE

    OPENAI_CLIENT -->|embeddings| TSNE
    TSNE -->|reduced data| PLOT_GEN
    PLOT_GEN -->|PNG/HTML| FILE_SYSTEM

    PUBMED_PARSER -->|metadata| PUBMED_API
    ARXIV_PARSER -->|metadata| ARXIV_API

    %% Click Events
    click PDF_SUMMARY "https://github.com/alleninstitute/openai_tools/blob/master/scripts/pdf_summary.py"
    click LIST_PDFS "https://github.com/alleninstitute/openai_tools/blob/master/scripts/list_pdfs_embedding.py"
    click PUBMED_EMB "https://github.com/alleninstitute/openai_tools/blob/master/scripts/pubmed_embedding.py"
    click PDF_PARSER "https://github.com/alleninstitute/openai_tools/blob/master/src/papers_extractor/pdf_parser.py"
    click XML_PARSER "https://github.com/alleninstitute/openai_tools/blob/master/src/papers_extractor/xml_parser.py"
    click PUBMED_PARSER "https://github.com/alleninstitute/openai_tools/blob/master/src/papers_extractor/pubmed_papers_parser.py"
    click ARXIV_PARSER "https://github.com/alleninstitute/openai_tools/blob/master/src/papers_extractor/arxiv_parser.py"
    click DB_PARSER "https://github.com/alleninstitute/openai_tools/blob/master/src/papers_extractor/database_parser.py"
    click TEXT_CHUNKER "https://github.com/alleninstitute/openai_tools/blob/master/src/papers_extractor/long_text.py"
    click OPENAI_CLIENT "https://github.com/alleninstitute/openai_tools/blob/master/src/papers_extractor/openai_parsers.py"
    click MULTI_PAPER "https://github.com/alleninstitute/openai_tools/blob/master/src/papers_extractor/multi_paper.py"
    click UNIQUE_PAPER "https://github.com/alleninstitute/openai_tools/blob/master/src/papers_extractor/unique_paper.py"
    click DISKCACHE "https://github.com/alleninstitute/openai_tools/blob/master/src/papers_extractor/database_parser.py"
    click FILE_SYSTEM "https://github.com/alleninstitute/openai_tools/tree/master/example"
    
    %% Styles
    classDef cli fill:#cce5ff,stroke:#005cbf,color:#005cbf
    classDef core fill:#d4edda,stroke:#155724,color:#155724
    classDef persistence fill:#fff3cd,stroke:#856404,color:#856404
    classDef viz fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef external fill:#f8d7da,stroke:#721c24,color:#721c24
```
