How romilly/s2ag-corpus works
Project Type and Purpose
• This is a data-ingestion and transformation tool (an ETL pipeline) that:
– Downloads/full loads seven Semantic Scholar datasets.
– Imports them into a PostgreSQL database.
– Applies incremental diffs to update the database.
– Provides an API for querying paper, citation, and reference data.
– Generates citation graphs in DOT/SVG format.
• No GUI; controlled by a CLI entrypoint and supported by interactive Jupyter notebooks.
Main Components
a. External Services
– Semantic Scholar API (data source)
– PostgreSQL database (data sink)
– File system storage (raw datasets and diffs)
b. CLI Entrypoint
– synchronize.py
• Reads .env config (BASE_DIR, API key, DB URLs)
• Invokes the Synchronizer to orchestrate download, load, and apply-diff steps.
c. Core Modules (src/s2ag_corpus)
1. synchronisation/
• config.py: Loads and exposes environment‐based settings.
• synchronizer.py: Coordinates the ETL workflow (download → import → diff).
2. requester/
• requester.py: Wraps HTTP calls to the Semantic Scholar API (with rate/ retry logic).
3. datasets/
• download_datasets.py: Fetches full dataset releases and stores them under BASE_DIR.
• import_dataset.py: Reads downloaded dataset files (JSON or .gz), streams them into DB.
• dataset_definitions.py: Defines the seven supported dataset schemas and file patterns.
• dataset_loader.py: High-level orchestration for loading each dataset in correct order.
4. database/
• sql.py: SQL statement templates for table creation and data manipulation.
• parser.py: Transforms raw JSON lines into CSV or parameterized DB inserts.
• database_catalogue.py: Tracks which releases/diffs have been applied in a metadata table.
5. diffs/
• do_diffs.py: Downloads incremental diff bundles for each dataset since last release.
• apply_diffs.py: Parses and applies inserts/updates/deletes to the existing tables.
6. api.py
• Exposes functions to query the loaded data (e.g., get paper by SHA, list references).
7. citation_graphs/
• find_links.py: Traverses references/citations to build a graph structure.
• dot_file.py & new_dot_writer.py & dot_writer.py: Generate DOT files from graph objects.
• citation.py: Defines node/edge models for the citation graph.
8. helpers/
• file_manager.py: Ensures correct directory structure and file existence under BASE_DIR.
• monitor.py & logging_monitor.py & mock_monitor.py: Progress reporting and logging abstractions.
• paper.py: Data models for paper metadata used across modules.
d. SQL Schema Definitions
– sql/create_*.sql: DDL for each table variation (with/without keys, unlogged for speed).
e. Testing Suite
– test/:
• unit/: Tests for downloader, parser, graph logic.
• integration/: End-to-end tests covering API, diffs, synchronizer.
• e2e/: Verifies full dataset load into a test database.
f. Notebooks & Documentation
– notebooks/: Exploratory notebooks for dataset exploration and graph generation.
– docs/: Static documentation and dataset metadata JSON under docs/datasets.
Relationships and Data Flow
1. CLI (synchronize.py) → Synchronizer
2. Synchronizer → Requester → Semantic Scholar API → Raw files on disk
3. Synchronizer → Dataset Loader → Parser → Database (via psycopg2/ SQL templates)
4. Synchronizer → Difff modules → Updates on Database
5. api.py → Exposes DB queries for downstream modules
6. citation_graphs → Uses api.py → Builds in-memory graph → Writes DOT → Renders SVG
7. Jupyter notebooks → Imported modules (api, citation_graphs) → Interactive analysis
Architectural Patterns & Principles
• Modular separation of concerns: downloader, importer, diff applier, API, graph generator.
• Config-driven execution using .env and config.py.
• Incremental/append-only diff application with metadata tracking.
• Testability via mock implementations in helpers and a dedicated test DB.
• Extensibility: adding new datasets or graph types by updating dataset_definitions.py and citation modules.