How gleanwork/indexing-api-connectors works
Project type and purpose
• This repository is a set of sample, self-contained connectors (small batch processes) demonstrating how to push documents into Glean’s Indexing API.
• It is not a monolithic web app but rather a collection of independent Python scripts, each packaged with its own dependencies and setup.
File structure
• Top level:
– README.md (project overview)
– Two connector subdirectories:
• upload_shortcuts/
• wikipedia/
• In each connector directory:
– README.md (usage instructions)
– requirements.txt (Python dependencies)
– setup.sh (environment setup script)
– <connector_script>.py (the driver that pulls data and calls Glean’s API)
README
• Confirms the purpose: sample connectors for Glean’s Indexing API.
• Directs you to per-directory READMEs for run instructions.
Main components
a. Connector Components (two instances):
– upload_shortcuts Connector
– wikipedia Connector
Each consists of:
• Environment (virtualenv created by setup.sh)
• Python runtime + dependencies from requirements.txt
• Connector script (.py)
b. Data Sources:
– Local shortcuts data (whatever upload_shortcuts reads)
– Wikipedia bulk data (wikipedia_bulk_index.py) or Wikipedia API
c. External System:
– Glean Indexing API (HTTP endpoints)
d. Supporting artifacts (for each connector):
– setup.sh (initialization)
– requirements.txt → pip install
Relationships and interactions
• Data Source → Connector Script: connector reads or fetches raw objects.
• Connector Script → Glean Indexing API: sends HTTP POST/PUT requests.
• setup.sh → Environment: bootstraps virtualenv and installs requirements.
• README.md files → Operator: instructions for running connectors.
Architectural patterns
• Each connector is a standalone batch job or micro-service.
• They follow a simple Extract→Transform→Load (ETL) pattern:
– Extract from source
– Transform to Glean document format (JSON)
– Load via Indexing API
• No shared code between connectors—favors isolation and simplicity.