# gleanwork/indexing-api-connectors architecture

> Architecture diagram of the GitHub repository gleanwork/indexing-api-connectors, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gleanwork/indexing-api-connectors
- Repository: https://github.com/gleanwork/indexing-api-connectors

## Overview

##### Project type and purpose
- This repository is a set of sample, self-contained connectors (small batch processes) demonstrating how to push documents into Glean’s Indexing API.
- It is not a monolithic web app but rather a collection of independent Python scripts, each packaged with its own dependencies and setup.

##### File structure
- Top level:
– README.md (project overview)
– Two connector subdirectories:
- upload_shortcuts/
- wikipedia/
- In each connector directory:
– README.md (usage instructions)
– requirements.txt (Python dependencies)
– setup.sh (environment setup script)
– <connector_script>.py (the driver that pulls data and calls Glean’s API)

##### README
- Confirms the purpose: sample connectors for Glean’s Indexing API.
- Directs you to per-directory READMEs for run instructions.

##### Main components
a. Connector Components (two instances):
– upload_shortcuts Connector
– wikipedia Connector
Each consists of:
- Environment (virtualenv created by setup.sh)
- Python runtime + dependencies from requirements.txt
- Connector script (.py)
b. Data Sources:
– Local shortcuts data (whatever upload_shortcuts reads)
– Wikipedia bulk data (wikipedia_bulk_index.py) or Wikipedia API
c. External System:
– Glean Indexing API (HTTP endpoints)
d. Supporting artifacts (for each connector):
– setup.sh (initialization)
– requirements.txt → pip install

##### Relationships and interactions
- Data Source → Connector Script: connector reads or fetches raw objects.
- Connector Script → Glean Indexing API: sends HTTP POST/PUT requests.
- setup.sh → Environment: bootstraps virtualenv and installs requirements.
- README.md files → Operator: instructions for running connectors.

##### Architectural patterns
- Each connector is a standalone batch job or micro-service.
- They follow a simple Extract→Transform→Load (ETL) pattern:
– Extract from source
– Transform to Glean document format (JSON)
– Load via Indexing API
- No shared code between connectors—favors isolation and simplicity.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Documentation
    ProjectOverview["README.md (Project Overview)"]:::docs
    click ProjectOverview "https://github.com/gleanwork/indexing-api-connectors/blob/main/README.md"

    %% Data Sources
    DataShortcuts["Local shortcuts data"]:::datasource
    DataWikipedia["Wikipedia bulk data"]:::datasource

    %% External API
    GleanAPI["Glean Indexing API"]:::external

    %% upload_shortcuts Connector
    subgraph "upload_shortcuts Connector"
        direction TB
        USC_Readme["README.md"]:::docs
        USC_Setup["setup.sh"]:::connector
        USC_Reqs["requirements.txt"]:::connector
        USC_Script["upload_shortcuts.py"]:::connector
    end
    click USC_Readme "https://github.com/gleanwork/indexing-api-connectors/blob/main/upload_shortcuts/README.md"
    click USC_Setup "https://github.com/gleanwork/indexing-api-connectors/blob/main/upload_shortcuts/setup.sh"
    click USC_Reqs "https://github.com/gleanwork/indexing-api-connectors/blob/main/upload_shortcuts/requirements.txt"
    click USC_Script "https://github.com/gleanwork/indexing-api-connectors/blob/main/upload_shortcuts/upload_shortcuts.py"
    click USC_Container "https://github.com/gleanwork/indexing-api-connectors/tree/main/upload_shortcuts/"

    %% wikipedia Connector
    subgraph "wikipedia Connector"
        direction TB
        WPC_Readme["README.md"]:::docs
        WPC_Setup["setup.sh"]:::connector
        WPC_Reqs["requirements.txt"]:::connector
        WPC_Script["wikipedia_bulk_index.py"]:::connector
    end
    click WPC_Readme "https://github.com/gleanwork/indexing-api-connectors/blob/main/wikipedia/README.md"
    click WPC_Setup "https://github.com/gleanwork/indexing-api-connectors/blob/main/wikipedia/setup.sh"
    click WPC_Reqs "https://github.com/gleanwork/indexing-api-connectors/blob/main/wikipedia/requirements.txt"
    click WPC_Script "https://github.com/gleanwork/indexing-api-connectors/blob/main/wikipedia/wikipedia_bulk_index.py"
    click WPC_Container "https://github.com/gleanwork/indexing-api-connectors/tree/main/wikipedia/"

    %% Data Flows
    DataShortcuts -->|"extract"| USC_Script
    USC_Setup -->|"bootstraps env"| USC_Script
    USC_Reqs -->|"installs deps"| USC_Script
    USC_Script -->|"load via HTTP POST /index"| GleanAPI

    DataWikipedia -->|"extract"| WPC_Script
    WPC_Setup -->|"bootstraps env"| WPC_Script
    WPC_Reqs -->|"installs deps"| WPC_Script
    WPC_Script -->|"load via HTTP POST /index"| GleanAPI

    %% Styles
    classDef connector fill:#c2f0c2,stroke:#0f0;
    classDef datasource fill:#ffe699,stroke:#e69138;
    classDef external fill:#c9daf8,stroke:#6c8ebf;
    classDef docs fill:#d9d9d9,stroke:#808080;
```
