# pingcap/ossinsight architecture

> Architecture diagram of the GitHub repository pingcap/ossinsight, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pingcap/ossinsight
- Repository: https://github.com/pingcap/ossinsight

## Overview

##### Project Type and Purpose
- This is a full-stack, monorepo data-analytics platform (“OSS Insight”) that ingests GitHub events (and related data sources), transforms and aggregates them via ETL and SQL pipelines, and exposes interactive dashboards, an API, and a GPT-powered “Data Explorer” front-end.

##### File-Structure Analysis
- **Top-level directories**
- configs/: YAML definitions for collections, pipelines, materialized-view specs, public-API HTTP endpoint mappings, SQL templates, search recommendations
- etl/: Ruby on Rails ETL service that fetches GitHub events (from BigQuery or GitHub API), HackerNews, SO, populates TiDB tables, maintains import logs, partitions, seeds, runs background jobs
- packages/: multiple Node.js/TS microservices and tools:
– api-server: HTTP API layer, plugin-based architecture (cache, CORS, auth, executors for TiDB/Octokit, rate-limit, metrics)
– pipeline: orchestrates full and incremental pipeline jobs using the configs, runs SQL jobs against TiDB, manages materialized views
– job-server: BullMQ/Redis workers for long-running/explorer tasks (GPT prompts, precomputations)
– prefetch: scheduler to prefill caches, warm up queries
– sync-github-data: Prisma-based service that syncs raw GitHub data in bulk or concurrent batches
– cli: command-line tool to validate or reload configs, run ad hoc checks
– types: shared TypeScript schemas (OpenAPI, query schemas)
- **web/**: React+Docusaurus static site and SPA that calls the API for dashboards, Data Explorer UI, docs, blog
- **docker-compose.yml**, **.github/workflows/**: CI/CD and container orchestration for local development and deployments

##### README Insights
- Describes features: Data Explorer (GPT → SQL → TiDB → charts), Collections, Developer/Repo Analytics, Compare Projects
- Emphasizes TiDB Cloud, Chat2Query integration, shareable widgets, extensible plugin points

##### Main Components & Interactions
a. **External Data Sources**
– GitHub API / GH Archive / BigQuery
– HackerNews API
– StackOverflow API
b. **Ingestion Layer**
– sync-github-data (TypeScript/Prisma) for GitHub bulk/concurrent sync
– etl (Rails) for event logs, HN, SO, import logs, partition management
c. **Primary Data Store**
– TiDB cluster (raw tables, import logs, event tables, partitioned tables)
d. **Transformation / Aggregation Layer**
– pipeline service reads configs/pipelines & materialized view specs → runs SQL against TiDB → updates aggregated/materialized tables
e. **Cache & Queue**
– Redis instance used for:
- query service cache (CachedTableCacheProvider)
- BullMQ job queue for explorer/long-running tasks
f. **Job Workers**
– job-server: processes queues (explorer high/low concurrency), GPT prompt jobs, writes results back or to cache
– prefetch: scheduled pre-computation to warm caches
g. **API Layer**
– api-server (Fastify/Express) loads public_api configs & routes → delegates to executors (TiDBQueryExecutor, OctokitExecutor) → returns JSON/Chart data over REST & WebSocket
– plugin architecture for auth (Auth0), rate limiting, CORS, metrics, etag, swagger docs
h. **Client Layer**
– web (React + Docusaurus): dashboards, Data Explorer UI, dynamic chart components, widgets
– CLI: management of collections and config validation
i. **GPT Integration**
– Chat2Query plugin in api-server routes → sends NL prompt to OpenAI → receives SQL → QueryRunner → TiDB → returns charts

##### Architectural Patterns & Principles
- Microservices per bounded context (ingestion, pipeline, API, jobs, CLI, prefetch)
- ETL → Transform → Serve separation
- Plugin-driven extensibility in API server and pipeline service
- Event-driven job processing (Redis + BullMQ) for async/long-running tasks
- Config-as-code (YAML under configs/) driving pipelines & endpoints

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "External Data Sources"
        GH["GitHub API / GH Archive / BigQuery"]:::external
        HN["HackerNews API"]:::external
        SO["StackOverflow API"]:::external
    end

    subgraph "Ingestion Layer"
        ETL["etl (Rails ETL service)"]:::service
        SYNC["sync-github-data"]:::service
    end

    subgraph "Primary Data Store"
        TIDB["TiDB cluster"]:::datastore
    end

    subgraph "Transformation & Aggregation"
        PIPE["pipeline service"]:::service
    end

    subgraph "Cache & Queue"
        REDIS["Redis"]:::datastore
    end

    subgraph "Job Workers"
        JOB["job-server workers"]:::service
        PREF["prefetch scheduler"]:::service
    end

    subgraph "API Layer"
        API["api-server"]:::service
        RedisPlugin["Redis Plugin"]:::service
        HighQueue["explorer-high-concurrent-queue"]:::service
        LowQueue["explorer-low-concurrent-queue"]:::service
        CoreExecutor["executors (TiDB/Octokit)"]:::service
        V1Routes["public API routes (v1)"]:::service
        BotService["bot-service plugin"]:::service
        Q2SQL["questionToSQL route"]:::service
    end

    subgraph "Clients"
        WEB["web (React + Docusaurus)"]:::client
        CLI["cli tool"]:::client
    end

    subgraph "Configs"
        CFG["configs/"]:::config
    end

    GH -->|ingest| ETL
    HN -->|ingest| ETL
    SO -->|ingest| ETL
    GH -->|sync| SYNC

    ETL -->|writes| TIDB
    SYNC -->|writes| TIDB

    CFG -->|loads| PIPE
    CFG -->|loads| API

    PIPE -->|runs SQL| TIDB

    API <-->|cache| REDIS
    API -->|enqueue jobs| REDIS
    REDIS -->|dispatch| JOB

    PREF -->|warm caches| REDIS
    JOB -->|write results| TIDB
    JOB -->|write cache| REDIS

    WEB -->|REST/WebSocket| API
    CLI -->|manage configs| CFG

    API -->|uses| RedisPlugin
    API -->|uses| HighQueue
    API -->|uses| LowQueue
    API -->|delegates| CoreExecutor
    API -->|routes| V1Routes
    API -->|uses| BotService
    BotService -->|handles| Q2SQL
    Q2SQL -->|runs| CoreExecutor
    CoreExecutor -->|queries| TIDB

    click ETL "https://github.com/pingcap/ossinsight/tree/main/etl/"
    click SYNC "https://github.com/pingcap/ossinsight/tree/main/packages/sync-github-data/src"
    click PIPE "https://github.com/pingcap/ossinsight/tree/main/packages/pipeline"
    click RedisPlugin "https://github.com/pingcap/ossinsight/blob/main/packages/api-server/src/plugins/redis.ts"
    click HighQueue "https://github.com/pingcap/ossinsight/blob/main/packages/api-server/src/plugins/queue/explorer-high-concurrent-queue.ts"
    click LowQueue "https://github.com/pingcap/ossinsight/blob/main/packages/api-server/src/plugins/queue/explorer-low-concurrent-queue.ts"
    click JOB "https://github.com/pingcap/ossinsight/tree/main/packages/job-server"
    click PREF "https://github.com/pingcap/ossinsight/tree/main/packages/prefetch"
    click API "https://github.com/pingcap/ossinsight/tree/main/packages/api-server"
    click CoreExecutor "https://github.com/pingcap/ossinsight/tree/main/packages/api-server/src/core/executor"
    click V1Routes "https://github.com/pingcap/ossinsight/tree/main/packages/api-server/src/routes/v1"
    click BotService "https://github.com/pingcap/ossinsight/tree/main/packages/api-server/src/plugins/services/bot-service"
    click Q2SQL "https://github.com/pingcap/ossinsight/tree/main/packages/api-server/src/routes/bot/questionToSQL"
    click CLI "https://github.com/pingcap/ossinsight/tree/main/packages/cli"
    click WEB "https://github.com/pingcap/ossinsight/tree/main/web/"
    click CFG "https://github.com/pingcap/ossinsight/tree/main/configs/"

    classDef service fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef datastore fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef config fill:#FFA500,stroke:#333,stroke-width:1px
    classDef client fill:#DDA0DD,stroke:#333,stroke-width:1px
```
