How pingcap/ossinsight works
Project Type and Purpose
• This is a full-stack, monorepo data-analytics platform (“OSS Insight”) that ingests GitHub events (and related data sources), transforms and aggregates them via ETL and SQL pipelines, and exposes interactive dashboards, an API, and a GPT-powered “Data Explorer” front-end.
File-Structure Analysis
• Top-level directories
• configs/: YAML definitions for collections, pipelines, materialized-view specs, public-API HTTP endpoint mappings, SQL templates, search recommendations
• etl/: Ruby on Rails ETL service that fetches GitHub events (from BigQuery or GitHub API), HackerNews, SO, populates TiDB tables, maintains import logs, partitions, seeds, runs background jobs
• packages/: multiple Node.js/TS microservices and tools:
– api-server: HTTP API layer, plugin-based architecture (cache, CORS, auth, executors for TiDB/Octokit, rate-limit, metrics)
– pipeline: orchestrates full and incremental pipeline jobs using the configs, runs SQL jobs against TiDB, manages materialized views
– job-server: BullMQ/Redis workers for long-running/explorer tasks (GPT prompts, precomputations)
– prefetch: scheduler to prefill caches, warm up queries
– sync-github-data: Prisma-based service that syncs raw GitHub data in bulk or concurrent batches
– cli: command-line tool to validate or reload configs, run ad hoc checks
– types: shared TypeScript schemas (OpenAPI, query schemas)
• web/: React+Docusaurus static site and SPA that calls the API for dashboards, Data Explorer UI, docs, blog
• docker-compose.yml, .github/workflows/: CI/CD and container orchestration for local development and deployments
README Insights
• Describes features: Data Explorer (GPT → SQL → TiDB → charts), Collections, Developer/Repo Analytics, Compare Projects
• Emphasizes TiDB Cloud, Chat2Query integration, shareable widgets, extensible plugin points
Main Components & Interactions
a. External Data Sources
– GitHub API / GH Archive / BigQuery
– HackerNews API
– StackOverflow API
b. Ingestion Layer
– sync-github-data (TypeScript/Prisma) for GitHub bulk/concurrent sync
– etl (Rails) for event logs, HN, SO, import logs, partition management
c. Primary Data Store
– TiDB cluster (raw tables, import logs, event tables, partitioned tables)
d. Transformation / Aggregation Layer
– pipeline service reads configs/pipelines & materialized view specs → runs SQL against TiDB → updates aggregated/materialized tables
e. Cache & Queue
– Redis instance used for:
• query service cache (CachedTableCacheProvider)
• BullMQ job queue for explorer/long-running tasks
f. Job Workers
– job-server: processes queues (explorer high/low concurrency), GPT prompt jobs, writes results back or to cache
– prefetch: scheduled pre-computation to warm caches
g. API Layer
– api-server (Fastify/Express) loads public_api configs & routes → delegates to executors (TiDBQueryExecutor, OctokitExecutor) → returns JSON/Chart data over REST & WebSocket
– plugin architecture for auth (Auth0), rate limiting, CORS, metrics, etag, swagger docs
h. Client Layer
– web (React + Docusaurus): dashboards, Data Explorer UI, dynamic chart components, widgets
– CLI: management of collections and config validation
i. GPT Integration
– Chat2Query plugin in api-server routes → sends NL prompt to OpenAI → receives SQL → QueryRunner → TiDB → returns charts
Architectural Patterns & Principles
• Microservices per bounded context (ingestion, pipeline, API, jobs, CLI, prefetch)
• ETL → Transform → Serve separation
• Plugin-driven extensibility in API server and pipeline service
• Event-driven job processing (Redis + BullMQ) for async/long-running tasks
• Config-as-code (YAML under configs/) driving pipelines & endpoints