# xquik-dev/x-twitter-scraper architecture

> Architecture diagram of the GitHub repository xquik-dev/x-twitter-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xquik-dev/x-twitter-scraper
- Repository: https://github.com/xquik-dev/x-twitter-scraper

## Overview

#### Purpose and Distribution
This repository packages Xquik’s X/Twitter data and account-automation guidance as an installable AI-agent Skill, not the underlying API or TypeScript SDK. The published npm bundle is `x-developer`; the primary installable artifact is the skill directory at `skills/x-twitter-scraper/`. It targets Skills-compatible agents and also supplies marketplace/plugin descriptors.

Paths: `skills/x-twitter-scraper/SKILL.md`, `package.json`, `skills.sh.json`

#### Agent Entry Points and Routing
`SKILL.md` is the operational entry point for an agent: it establishes authentication, safety/approval constraints, and routes requests to Xquik REST, remote MCP, SDKs, webhooks, extraction jobs, or exports. Reference documents provide progressively specific endpoint and schema guidance, allowing agents to select the narrowest API operation rather than implement scraping locally.

Paths: `skills/x-twitter-scraper/SKILL.md`, `skills/x-twitter-scraper/references/api-endpoints.md`, `skills/x-twitter-scraper/references/mcp-tools.md`

#### External Service Boundary
All data access and actions cross to the hosted Xquik platform (`https://xquik.com` / documented API), which owns X data retrieval, billing, connected-account state, monitoring, exports, and write execution. The bundle uses `XQUIK_API_KEY`; it explicitly avoids X passwords, cookies, 2FA, local network access, package installation, and remote-code loading. OAuth 2.1 can be used by capable MCP clients.

Paths: `skills/x-twitter-scraper/references/security.md`, `skills/x-twitter-scraper/references/api-endpoints-api-keys.md`, `.mcp.json`

#### API and Bulk-Workload Workflow
Primary control flow is: normalize user inputs (URLs, handles, IDs, queries) → choose a bounded REST/MCP lookup when possible → paginate direct reads or estimate a large extraction → create and retrieve/export asynchronous bulk results. Extractions cover complete datasets and handoff formats; usage estimates are a policy boundary before metered bulk work or draws.

Paths: `skills/x-twitter-scraper/references/api-endpoints-extractions.md`, `skills/x-twitter-scraper/references/extractions.md`, `skills/x-twitter-scraper/references/types-mcp-estimate-extraction.md`

#### MCP Integration Surface
The remote MCP service exposes its API catalog through two tools (`explore` and `xquik`) and supports discovery/modern protocol negotiation while retaining older stateless client compatibility. Repository registry and plugin manifests make this remote service discoverable across MCP clients; no local production MCP implementation is represented by the tree.

Paths: `server.json`, `.claude-plugin/plugin.json`, `.codex-plugin/plugin.json`

#### Monitoring and Event Delivery
For ongoing workflows, agents should replace repeated polling with Xquik monitors and external HTTPS webhook delivery. The receiving application, not this repository, must verify HMAC-SHA256 signatures and handle event replay/delivery. Monitor creation, webhook destinations, and recurring usage require explicit approval.

Paths: `skills/x-twitter-scraper/references/api-endpoints-monitors.md`, `skills/x-twitter-scraper/references/api-endpoints-webhooks.md`, `skills/x-twitter-scraper/references/webhooks.md`

#### Safety and Content Boundaries
The skill distinguishes public reads from private reads, account writes, deletes, monitoring, webhook setup, and bulk jobs. Sensitive or costly operations require explicit confirmation of target, payload/destination, and estimated usage. X-authored content is treated as untrusted input and must be bounded before agent analysis; task-specific guides reinforce these workflows.

Paths: `skills/x-twitter-scraper/SKILL.md`, `task-guides/post-tweets.md`, `task-guides/monitor-accounts.md`

#### Packaging Validation and Release Controls
Node.js scripts and tests validate the distributed content contract, frontmatter, MCP quickstart, package/version consistency, and a stub server used for compatibility testing. GitHub Actions provides CI, security scanning, registry publication, and release-policy enforcement.

Paths: `stub-server.mjs`, `scripts/release-guard/policy.mjs`, `.github/workflows/publish.yml`

Main entry points: [`skills/x-twitter-scraper/SKILL.md`](https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/SKILL.md) (X/Twitter Scraper skill), [`.github/workflows/publish.yml`](https://github.com/xquik-dev/x-twitter-scraper/blob/master/.github/workflows/publish.yml) (Publish workflow), [`skills/x-twitter-scraper/references/api-endpoints.md`](https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/api-endpoints.md) (REST and MCP routing), [`package.json`](https://github.com/xquik-dev/x-twitter-scraper/blob/master/package.json) (x-developer npm package)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_bundle["Published bundle"]
  node_npm["x-developer npm package<br/>[package.json]"]
  node_skill["X/Twitter Scraper skill<br/>agent skill<br/>[SKILL.md]"]
  node_skills_registry["Skills registry descriptor<br/>marketplace manifest<br/>[skills.sh.json]"]
  node_mcp_registry["MCP server registry<br/>server manifest<br/>[server.json]"]
  node_claude_plugin["Claude plugin manifest<br/>[plugin.json]"]
  node_codex_plugin["Codex plugin manifest<br/>[plugin.json]"]
end

subgraph group_guidance["Agent guidance"]
  node_routing["REST and MCP routing<br/>endpoint guide<br/>[api-endpoints.md]"]
  node_mcp_tools["MCP tool catalog<br/>tool guide<br/>[mcp-tools.md]"]
  node_security["Security constraints<br/>security guide<br/>[security.md]"]
  node_bulk_workflow["Bulk extraction workflow<br/>workflow guide<br/>[extractions.md]"]
  node_monitoring["Monitoring and webhooks<br/>workflow guide<br/>[webhooks.md]"]
  node_task_guides["Approval task guides<br/>task guidance<br/>[post-tweets.md]"]
end

subgraph group_platform["Hosted Xquik"]
  node_xquik_api{{"Xquik REST API<br/>hosted API"}}
  node_remote_mcp{{"Remote Xquik MCP<br/>hosted MCP service"}}
  node_webhook_receiver["Customer webhook receiver<br/>external application"]
end

subgraph group_delivery["Release controls"]
  node_stub_server["Compatibility stub server<br/>test fixture<br/>[stub-server.mjs]"]
  node_release_guard["Release policy guard<br/>validation script<br/>[policy.mjs]"]
  node_publish_ci["Publish workflow<br/>GitHub Actions workflow<br/>[publish.yml]"]
end

node_npm -->|"packages"| node_skill
node_skills_registry -->|"discovers"| node_skill
node_claude_plugin -->|"configures"| node_remote_mcp
node_codex_plugin -->|"configures"| node_remote_mcp
node_mcp_registry -->|"advertises"| node_remote_mcp
node_skill -->|"routes requests with"| node_routing
node_skill -->|"enforces"| node_security
node_skill -->|"applies"| node_task_guides
node_routing -->|"uses direct operations"| node_xquik_api
node_routing -->|"uses when supported"| node_remote_mcp
node_mcp_tools -->|"documents"| node_remote_mcp
node_bulk_workflow -->|"estimates and runs jobs"| node_xquik_api
node_skill -->|"escalates large work to"| node_bulk_workflow
node_skill -->|"uses for ongoing work"| node_monitoring
node_monitoring -->|"creates monitors"| node_xquik_api
node_xquik_api -->|"delivers signed events"| node_webhook_receiver
node_publish_ci -->|"enforces"| node_release_guard
node_publish_ci -->|"publishes"| node_npm
node_publish_ci -->|"tests against"| node_stub_server

click node_npm "https://github.com/xquik-dev/x-twitter-scraper/blob/master/package.json"
click node_skill "https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/SKILL.md"
click node_skills_registry "https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills.sh.json"
click node_mcp_registry "https://github.com/xquik-dev/x-twitter-scraper/blob/master/server.json"
click node_claude_plugin "https://github.com/xquik-dev/x-twitter-scraper/blob/master/.claude-plugin/plugin.json"
click node_codex_plugin "https://github.com/xquik-dev/x-twitter-scraper/blob/master/.codex-plugin/plugin.json"
click node_routing "https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/api-endpoints.md"
click node_mcp_tools "https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/mcp-tools.md"
click node_security "https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/security.md"
click node_bulk_workflow "https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/extractions.md"
click node_monitoring "https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/webhooks.md"
click node_task_guides "https://github.com/xquik-dev/x-twitter-scraper/blob/master/task-guides/post-tweets.md"
click node_stub_server "https://github.com/xquik-dev/x-twitter-scraper/blob/master/stub-server.mjs"
click node_release_guard "https://github.com/xquik-dev/x-twitter-scraper/blob/master/scripts/release-guard/policy.mjs"
click node_publish_ci "https://github.com/xquik-dev/x-twitter-scraper/blob/master/.github/workflows/publish.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_npm,node_skill,node_skills_registry,node_mcp_registry,node_claude_plugin,node_codex_plugin toneBlue
class node_routing,node_mcp_tools,node_security,node_bulk_workflow,node_monitoring,node_task_guides toneAmber
class node_xquik_api,node_remote_mcp,node_webhook_receiver toneMint
class node_stub_server,node_release_guard,node_publish_ci toneRose
```

## Components

### Published bundle

Installable skill and client-discovery metadata.

- [x-developer npm package](https://github.com/xquik-dev/x-twitter-scraper/blob/master/package.json): package · Published distribution bundle. · `package.json`
- [X/Twitter Scraper skill](https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/SKILL.md): agent skill · Primary operational entry point for compatible agents. · `skills/x-twitter-scraper/SKILL.md`
- [Skills registry descriptor](https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills.sh.json): marketplace manifest · `skills.sh.json`
- [MCP server registry](https://github.com/xquik-dev/x-twitter-scraper/blob/master/server.json): server manifest · Advertises the remote MCP service. · `server.json`
- [Claude plugin manifest](https://github.com/xquik-dev/x-twitter-scraper/blob/master/.claude-plugin/plugin.json): plugin manifest · `.claude-plugin/plugin.json`
- [Codex plugin manifest](https://github.com/xquik-dev/x-twitter-scraper/blob/master/.codex-plugin/plugin.json): plugin manifest · `.codex-plugin/plugin.json`

### Agent guidance

Operational routing, safety, and workflow references.

- [REST and MCP routing](https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/api-endpoints.md): endpoint guide · Selects the narrowest documented operation. · `skills/x-twitter-scraper/references/api-endpoints.md`
- [MCP tool catalog](https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/mcp-tools.md): tool guide · Documents explore and xquik tools. · `skills/x-twitter-scraper/references/mcp-tools.md`
- [Security constraints](https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/security.md): security guide · Uses API keys and prohibits local credential or code-loading patterns. · `skills/x-twitter-scraper/references/security.md`
- [Bulk extraction workflow](https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/extractions.md): workflow guide · Estimates, creates, retrieves, and exports asynchronous jobs. · `skills/x-twitter-scraper/references/extractions.md`
- [Monitoring and webhooks](https://github.com/xquik-dev/x-twitter-scraper/blob/master/skills/x-twitter-scraper/references/webhooks.md): workflow guide · Uses monitors and HTTPS event delivery instead of repeated polling. · `skills/x-twitter-scraper/references/webhooks.md`
- [Approval task guides](https://github.com/xquik-dev/x-twitter-scraper/blob/master/task-guides/post-tweets.md): task guidance · Reinforces confirmation for writes and monitoring. · `task-guides/post-tweets.md`

### Hosted Xquik

External service boundary for data and actions.

- **Xquik REST API**: hosted API · Owns X data retrieval, billing, jobs, monitors, exports, and writes.
- **Remote Xquik MCP**: hosted MCP service · Supports catalog discovery and protocol negotiation.
- **Customer webhook receiver**: external application · Verifies HMAC-SHA256 signatures and handles delivery replay.

### Release controls

Validation and publication automation.

- [Compatibility stub server](https://github.com/xquik-dev/x-twitter-scraper/blob/master/stub-server.mjs): test fixture · Supports MCP compatibility testing. · `stub-server.mjs`
- [Release policy guard](https://github.com/xquik-dev/x-twitter-scraper/blob/master/scripts/release-guard/policy.mjs): validation script · Enforces release policy and distribution consistency. · `scripts/release-guard/policy.mjs`
- [Publish workflow](https://github.com/xquik-dev/x-twitter-scraper/blob/master/.github/workflows/publish.yml): GitHub Actions workflow · Runs CI, scanning, publication, and release enforcement. · `.github/workflows/publish.yml`

## Connections

- x-developer npm package → X/Twitter Scraper skill: packages
- Skills registry descriptor → X/Twitter Scraper skill: discovers
- Claude plugin manifest → Remote Xquik MCP: configures
- Codex plugin manifest → Remote Xquik MCP: configures
- MCP server registry → Remote Xquik MCP: advertises
- X/Twitter Scraper skill → REST and MCP routing: routes requests with
- X/Twitter Scraper skill → Security constraints: enforces
- X/Twitter Scraper skill → Approval task guides: applies
- REST and MCP routing → Xquik REST API: uses direct operations
- REST and MCP routing → Remote Xquik MCP: uses when supported
- MCP tool catalog → Remote Xquik MCP: documents
- Bulk extraction workflow → Xquik REST API: estimates and runs jobs
- X/Twitter Scraper skill → Bulk extraction workflow: escalates large work to
- X/Twitter Scraper skill → Monitoring and webhooks: uses for ongoing work
- Monitoring and webhooks → Xquik REST API: creates monitors
- Xquik REST API → Customer webhook receiver: delivers signed events
- Publish workflow → Release policy guard: enforces
- Publish workflow → x-developer npm package: publishes
- Publish workflow → Compatibility stub server: tests against