# cverluise/patcit architecture

> Architecture diagram of the GitHub repository cverluise/patcit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cverluise/patcit
- Repository: https://github.com/cverluise/patcit

## Overview

##### Project type and purpose
- patCit is a data-processing/ETL pipeline and library (not a client/server web app). It ingests patent and front-page citation corpora, applies NLP models (classification, extraction, enrichment), versions data/models via DVC, and exposes the results on BigQuery.

##### File structure
- Top-level directories:
- data, models: versioned via DVC for raw/enriched datasets and trained spaCy models
- patcit: the core Python package subdivided into subpackages:
– bq: BigQuery ingestion/export
– brew: brewing data recipes
– data: raw data download/processing
– grobid: calls to GROBID for parsing citations
– model: spaCy pipeline configuration (classification, fine-tuning, evaluation)
– serialize: convert pipeline outputs into JSON/CSV, validate via JSON Schemas
– utils: shared utilities and CLI entrypoints
- schema: JSON Schema definitions for serialized outputs
- notebook: example Colab notebooks for exploration
- docs: MkDocs-based user/developer documentation
- Configuration files: pyproject.toml, setup.py, Makefile, .pre-commit-config.yaml, mkdocs.yml, scripts for TLS preparation
- Deployment: no web server; uses Google Cloud (BigQuery) and external services (GROBID, biblio-glutton, Google APIs)

##### Extract architecture hints from README
- Data sources: DOCDB (front-page), USPTO full-text patents via Google Patents OCR
- NLP stack: spaCy, GROBID, biblio-glutton, Google Match API
- Storage: BigQuery tables for front-page & in-text citations
- Versioning: DVC handles large data and models
- Outputs: BigQuery, Zenodo, downloadable artifacts

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    DocDB["DOCDB Bulk (front-page)"]:::external
    GPAT["Google Patents OCR (in-text)"]:::external

    %% Ingestion & Cleaning
    subgraph "Ingestion & Cleaning"
        data_pkg["patcit/data"]:::internal
        data_main["main.py"]:::internal
    end
    click data_pkg "https://github.com/cverluise/patcit/tree/master/patcit/data/"
    click data_main "https://github.com/cverluise/patcit/blob/master/patcit/data/main.py"

    DocDB -->|raw XML| data_pkg
    GPAT -->|raw OCR JSON| data_pkg
    data_pkg -->|"cleaned raw JSON"| DVC_data

    %% Parsing (GROBID)
    subgraph "Parsing (GROBID)"
        grobid_pkg["patcit/grobid"]:::internal
        grobid_io["io.py"]:::internal
        grobid_main["main.py"]:::internal
    end
    click grobid_pkg "https://github.com/cverluise/patcit/tree/master/patcit/grobid/"
    click grobid_io "https://github.com/cverluise/patcit/blob/master/patcit/grobid/io.py"
    click grobid_main "https://github.com/cverluise/patcit/blob/master/patcit/grobid/main.py"

    DVC_data -->|"raw files"| grobid_pkg

    %% Classification & Extraction (spaCy)
    subgraph "Classification & Extraction (spaCy)"
        model_pkg["patcit/model"]:::internal
        model_main["main.py"]:::internal
        finetune["finetune.py"]:::internal
        evaluate["evaluate.py"]:::internal
    end
    click model_pkg "https://github.com/cverluise/patcit/tree/master/patcit/model/"
    click model_main "https://github.com/cverluise/patcit/blob/master/patcit/model/main.py"
    click finetune "https://github.com/cverluise/patcit/blob/master/patcit/model/finetune.py"
    click evaluate "https://github.com/cverluise/patcit/blob/master/patcit/model/evaluate.py"

    grobid_pkg -->|"parsed citations"| model_pkg

    %% Enrichment
    subgraph "Enrichment"
        GL["biblio-glutton"]:::external
        GMatch["Google Match API"]:::external
    end

    model_pkg -->|"spaCy Doc"| GL
    model_pkg -->|"spaCy Doc"| GMatch

    %% Serialization & Validation
    subgraph "Serialization & Validation"
        serialize_pkg["patcit/serialize"]:::internal
        ser_main["main.py"]:::internal
        bibref["bibref.py"]:::internal
        intext["intext.py"]:::internal
        lib["lib.py"]:::internal
        subgraph "Validation"
            val_schema["schema/ JSON Schemas"]:::schema
            val_py["validation/*.py"]:::internal
        end
    end
    click serialize_pkg "https://github.com/cverluise/patcit/tree/master/patcit/serialize/"
    click ser_main "https://github.com/cverluise/patcit/blob/master/patcit/serialize/main.py"
    click bibref "https://github.com/cverluise/patcit/blob/master/patcit/serialize/bibref.py"
    click intext "https://github.com/cverluise/patcit/blob/master/patcit/serialize/intext.py"
    click lib "https://github.com/cverluise/patcit/blob/master/patcit/serialize/lib.py"
    click val_schema "https://github.com/cverluise/patcit/tree/master/schema/"
    click val_py "https://github.com/cverluise/patcit/tree/master/patcit/serialize/validation/"

    GL -->|"matched DOIs"| serialize_pkg
    GMatch -->|"matched patents"| serialize_pkg

    serialize_pkg -->|"validated JSON/CSV"| DVC_data
    serialize_pkg -->|"validated JSON/CSV"| DVC_models

    %% Export to BigQuery
    subgraph "Export"
        bq_pkg["patcit/bq"]:::internal
        bq_main["main.py"]:::internal
    end
    click bq_pkg "https://github.com/cverluise/patcit/tree/master/patcit/bq/"
    click bq_main "https://github.com/cverluise/patcit/blob/master/patcit/bq/main.py"

    serialize_pkg --> bq_pkg
    bq_pkg -->|"table load"| BQ

    %% Storage
    DVC_data["DVC: data/"]:::storage
    DVC_models["DVC: models/"]:::storage
    BQ["BigQuery Public Datasets"]:::bigquery
    click DVC_data "https://github.com/cverluise/patcit/tree/master/data/"
    click DVC_models "https://github.com/cverluise/patcit/tree/master/models/"

    %% Consumption
    subgraph "Consumption"
        nb["notebook/"]:::notebook
        docs["docs/"]:::docs
    end
    click nb "https://github.com/cverluise/patcit/tree/master/notebook/"
    click docs "https://github.com/cverluise/patcit/tree/master/docs/"

    BQ --> nb
    BQ --> docs

    %% Styles
    classDef internal fill:#D0E8FF,stroke:#024e82;
    classDef external fill:#D5F5E3,stroke:#028f5b;
    classDef storage fill:#FFE5A1,stroke:#b07f00;
    classDef bigquery fill:#FFD8A8,stroke:#b07f00;
    classDef schema fill:#F6E8FF,stroke:#8e44ad;
    classDef notebook fill:#E8F5E9,stroke:#2e7d32;
    classDef docs fill:#E3F2FD,stroke:#1565C0;
```
