# romilly/s2ag-corpus architecture

> Architecture diagram of the GitHub repository romilly/s2ag-corpus, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/romilly/s2ag-corpus
- Repository: https://github.com/romilly/s2ag-corpus

## Overview

##### Project Type and Purpose
- This is a data-ingestion and transformation tool (an ETL pipeline) that:
– Downloads/full loads seven Semantic Scholar datasets.
– Imports them into a PostgreSQL database.
– Applies incremental diffs to update the database.
– Provides an API for querying paper, citation, and reference data.
– Generates citation graphs in DOT/SVG format.
- No GUI; controlled by a CLI entrypoint and supported by interactive Jupyter notebooks.

##### Main Components
a. External Services
– Semantic Scholar API (data source)
– PostgreSQL database (data sink)
– File system storage (raw datasets and diffs)
b. CLI Entrypoint
– synchronize.py
- Reads .env config (BASE_DIR, API key, DB URLs)
- Invokes the Synchronizer to orchestrate download, load, and apply-diff steps.
c. Core Modules (src/s2ag_corpus)
1. synchronisation/
- config.py: Loads and exposes environment‐based settings.
- synchronizer.py: Coordinates the ETL workflow (download → import → diff).
2. requester/
- requester.py: Wraps HTTP calls to the Semantic Scholar API (with rate/ retry logic).
3. datasets/
- download_datasets.py: Fetches full dataset releases and stores them under BASE_DIR.
- import_dataset.py: Reads downloaded dataset files (JSON or .gz), streams them into DB.
- dataset_definitions.py: Defines the seven supported dataset schemas and file patterns.
- dataset_loader.py: High-level orchestration for loading each dataset in correct order.
4. database/
- sql.py: SQL statement templates for table creation and data manipulation.
- parser.py: Transforms raw JSON lines into CSV or parameterized DB inserts.
- database_catalogue.py: Tracks which releases/diffs have been applied in a metadata table.
5. diffs/
- do_diffs.py: Downloads incremental diff bundles for each dataset since last release.
- apply_diffs.py: Parses and applies inserts/updates/deletes to the existing tables.
6. api.py
- Exposes functions to query the loaded data (e.g., get paper by SHA, list references).
7. citation_graphs/
- find_links.py: Traverses references/citations to build a graph structure.
- dot_file.py & new_dot_writer.py & dot_writer.py: Generate DOT files from graph objects.
- citation.py: Defines node/edge models for the citation graph.
8. helpers/
- file_manager.py: Ensures correct directory structure and file existence under BASE_DIR.
- monitor.py & logging_monitor.py & mock_monitor.py: Progress reporting and logging abstractions.
- paper.py: Data models for paper metadata used across modules.
d. SQL Schema Definitions
– sql/create_*.sql: DDL for each table variation (with/without keys, unlogged for speed).
e. Testing Suite
– test/:
- unit/: Tests for downloader, parser, graph logic.
- integration/: End-to-end tests covering API, diffs, synchronizer.
- e2e/: Verifies full dataset load into a test database.
f. Notebooks & Documentation
– notebooks/: Exploratory notebooks for dataset exploration and graph generation.
– docs/: Static documentation and dataset metadata JSON under docs/datasets.

##### Relationships and Data Flow
1. CLI (synchronize.py) → Synchronizer
2. Synchronizer → Requester → Semantic Scholar API → Raw files on disk
3. Synchronizer → Dataset Loader → Parser → Database (via psycopg2/ SQL templates)
4. Synchronizer → Difﬀ modules → Updates on Database
5. api.py → Exposes DB queries for downstream modules
6. citation_graphs → Uses api.py → Builds in-memory graph → Writes DOT → Renders SVG
7. Jupyter notebooks → Imported modules (api, citation_graphs) → Interactive analysis

##### Architectural Patterns & Principles
- Modular separation of concerns: downloader, importer, diff applier, API, graph generator.
- Config-driven execution using .env and config.py.
- Incremental/append-only diff application with metadata tracking.
- Testability via mock implementations in helpers and a dedicated test DB.
- Extensibility: adding new datasets or graph types by updating dataset_definitions.py and citation modules.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Services
    subgraph "External Services"
        API["Semantic Scholar API"]:::external
    end

    %% Data Stores
    FILESYS["File System\n(raw datasets & diffs)"]:::store
    DB["PostgreSQL Database"]:::store

    %% CLI Entrypoint
    CLI["synchronize.py\n(CLI Entrypoint)"]:::internal

    %% Configuration & Orchestration
    subgraph "Synchronisation Layer"
        CFG["config.py\n(Load .env settings)"]:::internal
        SYNC["synchronizer.py\n(Coordinates ETL)"]:::internal
    end

    %% Request Layer
    REQ["requester.py\n(HTTP wrapper, rate/retry)"]:::internal

    %% Dataset Modules
    subgraph "Datasets"
        DEF["dataset_definitions.py\n(Define schemas & patterns)"]:::internal
        DOWN["download_datasets.py\n(Fetch full releases)"]:::internal
        IMP["import_dataset.py\n(Stream into DB)"]:::internal
        LOADER["dataset_loader.py\n(Orchestrate load order)"]:::internal
    end

    %% Database Modules
    subgraph "Database"
        SQL["sql.py\n(SQL templates & DDL)"]:::internal
        PARSE["parser.py\n(JSON→CSV/DB inserts)"]:::internal
        CAT["database_catalogue.py\n(Track applied releases/diffs)"]:::internal
    end

    %% Diff Modules
    subgraph "Diffs"
        DIFFD["do_diffs.py\n(Download incremental diffs)"]:::internal
        DIFFA["apply_diffs.py\n(Apply inserts/updates/deletes)"]:::internal
    end

    %% API & Graph Generation
    subgraph "API & Citation Graphs"
        APIEX["api.py\n(Expose DB queries)"]:::internal
        FIND["find_links.py\n(Traverse refs/citations)"]:::internal
        CIT["citation.py\n(Node/edge models)"]:::internal
        DOT1["dot_file.py\n(Generate DOT files)"]:::internal
        DOT2["dot_writer.py\n(Legacy DOT writer)"]:::internal
        DOT3["new_dot_writer.py\n(New DOT writer)"]:::internal
    end

    %% Helpers
    subgraph "Helpers"
        FM["file_manager.py\n(Manage BASE_DIR structure)"]:::helper
        MON["monitor.py\n(Progress abstraction)"]:::helper
        LMON["logging_monitor.py\n(Logging monitor)"]:::helper
        MMON["mock_monitor.py\n(Mock monitor for tests)"]:::helper
        PAPER["paper.py\n(Paper metadata model)"]:::helper
    end

    %% Notebooks & Documentation
    subgraph "Notebooks & Docs"
        NB["notebooks/\n(Jupyter notebooks)"]:::docs
        DOCS["docs/datasets/\n(JSON metadata)"]:::docs
    end

    %% Relationships
    CLI --> CFG
    CLI --> SYNC
    SYNC --> REQ
    REQ -->|download JSON.gz| API
    API -->|store files| FILESYS

    SYNC --> LOADER
    LOADER --> DOWN
    LOADER --> IMP
    DOWN --> FILESYS
    IMP --> PARSE
    PARSE --> DB
    LOADER --> DEF
    SYNC --> DIFFD
    SYNC --> DIFFA
    DIFFD --> FILESYS
    DIFFA --> DB

    APIEX --> DB
    FIND --> APIEX
    FIND --> CIT
    CIT --> DOT1
    CIT --> DOT2
    CIT --> DOT3
    DOT1 --> FILESYS
    DOT2 --> FILESYS
    DOT3 --> FILESYS

    NB --> APIEX
    NB --> FIND

    %% Click Events
    click CLI "https://github.com/romilly/s2ag-corpus/blob/master/synchronize.py"
    click CFG "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/synchronisation/config.py"
    click SYNC "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/synchronisation/synchronizer.py"
    click REQ "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/requester/requester.py"
    click DEF "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/datasets/dataset_definitions.py"
    click DOWN "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/datasets/download_datasets.py"
    click IMP "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/datasets/import_dataset.py"
    click LOADER "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/datasets/dataset_loader.py"
    click SQL "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/database/sql.py"
    click PARSE "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/database/parser.py"
    click CAT "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/database/database_catalogue.py"
    click DIFFD "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/diffs/do_diffs.py"
    click DIFFA "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/diffs/apply_diffs.py"
    click APIEX "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/api.py"
    click FIND "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/citation_graphs/find_links.py"
    click DOT1 "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/citation_graphs/dot_file.py"
    click DOT2 "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/citation_graphs/dot_writer.py"
    click DOT3 "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/citation_graphs/new_dot_writer.py"
    click CIT "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/citation_graphs/citation.py"
    click FM "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/helpers/file_manager.py"
    click MON "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/helpers/monitor.py"
    click LMON "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/helpers/logging_monitor.py"
    click MMON "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/helpers/mock_monitor.py"
    click PAPER "https://github.com/romilly/s2ag-corpus/blob/master/src/s2ag_corpus/helpers/paper.py"
    click DOCS "https://github.com/romilly/s2ag-corpus/tree/master/docs/datasets/"
    click NB "https://github.com/romilly/s2ag-corpus/tree/master/notebooks/"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px
    classDef store fill:#90EE90,stroke:#000,stroke-width:1px
    classDef helper fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef docs fill:#E0E0E0,stroke:#000,stroke-width:1px

    %% Legend
    subgraph "Legend"
        A["Blue: Internal Modules"]:::internal
        B["Orange: External Services"]:::external
        C["Green: Data Stores"]:::store
        D["Gray: Helper Utilities"]:::helper
        E["Light Gray: Notebooks & Docs"]:::docs
    end
```
