How alleninstitute/openai_tools works
Project type and purpose
• This is a Python‐based toolset and library (“papers_extractor”) for:
– Parsing and extracting text from scientific publication PDFs (and XML)
– Summarizing documents via the OpenAI ChatGPT API
– Embedding document text (PDFs or PubMed abstracts) with OpenAI embeddings
– Storing and caching results in a local on‐disk database to minimize repeated API calls
– Generating t-SNE visualizations (PNG or interactive HTML) of the embedding space
• The top‐level “scripts” folder provides three CLI entry‐points: pdf_summary.py, list_pdfs_embedding.py, pubmed_embedding.py
File structure
• src/papers_extractor – core library modules
– parsers: pdf_parser.py, xml_parser.py, database_parser.py, pubmed_papers_parser.py, arxiv_parser.py
– openai_parsers.py handles chunking, prompts, and calls to the OpenAI API
– multi_paper.py, unique_paper.py define data models and deduplication logic
– long_text.py splits text for multi‐chunk summarization
• scripts – user‐facing commands that wire together parsing, embedding, summarization, database, and plotting
• example – sample inputs (PDFs) and outputs (text, compressed text, HTML, PNG)
• tests – unit tests for each parser and model component
• docs – Sphinx documentation configuration
• setup.py, requirements.txt, dev_requirements.txt – packaging and CI
README for additional insights
• Database layer uses DiskCache to store raw text, summaries, embeddings keyed by PDF content hash
• External dependencies: OpenAI API, adjustText (for label placement), matplotlib/plotly for plots
Key takeaways
• Every major component and its role in the pipeline—parsing, caching, API, plotting
• Both the happy path (cache hit) and miss path (API call)