How cortexproject/cortex works
Project Type and Purpose
• Cortex is a cloud-native, distributed, full-backend system written in Go. It provides horizontally scalable, highly available, multi-tenant, long-term storage and querying for Prometheus and OpenTelemetry metrics.
• Key indicators: a “cmd/” directory with multiple microservice binaries (cortex, alertmanager, query-tee, test-exporter, thanosconvert), a rich “pkg/” layer broken into service domains (distributor, ingester, querier, ruler, compactor, storegateway, etc.), and a README describing Prometheus remote_write ingestion, object-store backends (S3/GCS/Swift/Azure), and a ring-based sharding/gossip design.
File-Structure Analysis
• Top-level “cmd/” houses the entrypoints for each service component.
• “pkg/” decomposes the core services and libraries:
• ingestion path: distributor → ingester → chunk or block storage (pkg/storage, pkg/chunk)
• query path: query-frontend → querier → store-gateway → object store
• background / operational services: compactor, ruler, alertmanager, scheduler
• service-discovery & sharding: pkg/ring, pkg/ha (gossip ring), pkg/distributed_execution
• External-facing API in pkg/api for health-checks and query endpoints.
• Configuration / runtime in pkg/configs, pkg/cortex, plus Dockerfiles, Makefiles, GitHub workflows, integration tests, docs under “docs/architecture.md”.
Key Architectural Insights from README
• Multi-tenant isolation, shuffle sharding, consistent hashing ring, gossip or KV (Consul/Etcd/DynamoDB) for membership and ring coordination.
• Object-store backed chunk and new blocks storage.
• Separate write path (remote_write → distributor → ingester → storage) and read path (query_frontend → querier → storegateway → storage).
a. Main Components to include:
• Clients (Prometheus or OTEL exporters) →
• Distributor (HTTP/gRPC ingress) →
• Ingester cluster (in-memory TSDB, replication, gRPC) →
• Object storage (S3/GCS/Swift/Azure) & Index storage
• Store-gateway (serving historical blocks)
• Query-frontend (dedupe, fan-out, rate-limit)
• Querier (executes PromQL, fan-in streams)
• Compactor (block compaction, retention enforcement)
• Ruler (recording rules evaluation)
• Alertmanager (alert routing & dedup)
• Ring/KV store (Consul/Etcd/DynamoDB) & Gossip RPC
• Metrics & Tracing pipelines (OpenTelemetry collector, Grafana Agent)
• Optional tooling: query-tee, test-exporter, thanosconvert
b. Relationships & Data Flows:
1. Prometheus → Distributor (remote_write API)
2. Distributor → Ingester(s) (gRPC, ring-based sharding)
3. Ingester → Object Storage + Index in parallel
4. Queries: User (Grafana) → Query-Frontend → Querier → Store-Gateway → Object Storage
5. Ruler periodically queries / PromQL via Query-Frontend → stores results via Alertmanager API → Alertmanager
6. Compactor watches completed blocks in Object Storage → writes down-sampled or compacted blocks
7. All services join a ring via KV or Gossip for discovery and sharding
8. Service health & metrics scraped by Prometheus
c. Architectural Patterns & Principles:
• Microservices: each binary in “cmd/” fulfills a single responsibility.
• Ring-based consistent hashing for sharding and replication.
• Side-car/gossip/RPC for service discovery and high availability.
• Fan-out/fan-in for query scalability.
• Asynchronous, append-only writes to object store; eventual consistency.
• Multi-tenant isolation via label-prefixing and tenant-ID metadata.
d. Key Technologies & Libraries:
• Go, gRPC (protobuf in pkg/*pb), HTTP APIs (pkg/api), PromQL engine (pkg/querier), TSDB storage libraries, Cortex protobuf codecs.
• Object store SDKs (pkg/storage/bucket), KV clients (Consul, Etcd, DynamoDB).
• Docker + Kubernetes manifests (not shown), Makefile automation, GitHub Actions.