# xai-org/x-algorithm architecture

> Architecture diagram of the GitHub repository xai-org/x-algorithm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xai-org/x-algorithm
- Repository: https://github.com/xai-org/x-algorithm
- Video tour (about a minute): https://gitdiagram.com/xai-org/x-algorithm/video

## Overview

#### Purpose and Entry Points
This monorepo implements X’s For You feed request path plus the offline/streaming systems that produce retrieval data, model scores, safety labels, and enforcement outcomes. The primary feed service starts at `home-mixer/main.rs` and serves For You, Following, and scoring variants through `home-mixer/for_you_server.rs`, `home-mixer/following_feed_server.rs`, and `home-mixer/server.rs`.

#### Pipeline Framework and Feed Orchestration
`candidate-pipeline/` defines the composable execution model: sources, query/candidate hydrators, filters, scorers, selectors, and asynchronous side effects. `home-mixer/` instantiates this framework into feed-specific pipelines:
- `home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs`
- `home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs`
- `home-mixer/candidate_pipeline/following_candidate_pipeline.rs`

A request is hydrated with viewer state, candidates are retrieved and enriched, then filtered, scored, selected, visibility-checked, blended with non-post items, and serialized into timeline output.

#### Retrieval and Candidate Inputs
Candidate sources run in parallel behind `home-mixer/sources/`. In-network retrieval uses Thunder’s recent-post store and Kafka-driven ingestion: `thunder/main.rs`, `thunder/kafka/tweet_events_listener.rs`, `thunder/posts/post_store.rs`. Out-of-network candidates come from Phoenix retrieval and SimClusters, called through sources such as `home-mixer/sources/phoenix_source.rs` and `home-mixer/sources/simclusters_source.rs`.

Phoenix RankAll maintains the retrievable-post index from Kafka events, processing and storing indexed candidates in `phoenix-rankall/src/main.rs`, `phoenix-rankall/src/processor/main_processor.rs`, and `phoenix-rankall/src/store/writer.rs`.

#### Hydration, Filtering, and Visibility Boundary
Query hydrators load viewer follows, blocks, mutes, engagement sequences, served history, and related context under `home-mixer/query_hydrators/`. Candidate hydrators fetch post, author, media, engagement, and safety attributes under `home-mixer/candidate_hydrators/`.

Pre-score filters remove duplicates, unavailable or ineligible content, self-posts, seen/served content, and social-graph exclusions via `home-mixer/filters/`. Visibility filtering is a separate post-selection service boundary: `home-mixer/candidate_hydrators/vf_candidate_hydrator.rs` calls `visibility-filtering/`, and `home-mixer/filters/vf_filter.rs` removes DROP decisions. The visibility service hydrates labels and relationship state, evaluates ordered rules, and returns allow/interstitial/drop:
- `visibility-filtering/main.rs`
- `visibility-filtering/rules/registry.rs`
- `visibility-filtering/safety_label_source/manhattan.rs`

#### Ranking, Selection, and Blending
Phoenix predicts per-viewer/per-post action probabilities; Home Mixer converts these predictions into weighted ranking scores and applies diversity or author adjustments. Core integration points are `home-mixer/scorers/phoenix_scorer.rs`, `home-mixer/scorers/ranking_scorer.rs`, and `home-mixer/params/param.rs`. Scores are optionally reranked for diversity by the standalone `vm-ranker/main.rs` service.

`home-mixer/selectors/top_k_score_selector.rs` selects ranked posts. The outer For You pipeline blends ads, prompts, and Who-to-Follow items using `home-mixer/selectors/blender_selector.rs` and ad-placement logic in `home-mixer/ads/partition_organic_blender.rs`. Post-response side effects publish served history, Kafka events, caches, and metrics under `home-mixer/side_effects/`.

#### Phoenix Model Training and Serving
`phoenix/` contains the recommendation model stack: Python/JAX training and Rust serving. Training models, data preparation, and trainers are under `phoenix/xrex/models/recsys_model.py`, `phoenix/xrex/data/recsys/`, and `phoenix/xrex/train/trainer_recsys.py`. Rust serving exposes retrieval/ranking infrastructure and protobuf APIs in `phoenix/crates/serving/xai-recsys-engine/src/lib.rs`, `phoenix/crates/serving/xai-recsys-server/src/lib.rs`, and `phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto`. The stack includes GPU/CUDA kernels, checkpoint storage, gRPC, and SID/vector retrieval.

#### Label Production, Rules, and Enforcement
Safety/abuse labels are produced off the feed request path. Grox classifies posts/media through plan/task orchestration (`grox/core/engine.py`, `grox/flows/ptos/plan_safety_ptos.py`); Media Model Proxy exposes image/video models over Thrift (`media-model-proxy/src/main/scala/com/twitter/media_understanding/model_proxy/app/Server.scala`, `media-model-proxy/src/main/thrift/com/twitter/cortex_media_annotator/annotate.thrift`).

Scarecrow consumes events and executes BotMaker rules from `botmaker-rules/scarecrow/`; its server/runtime are `scarecrow/Server.scala` and `scarecrow/ScarecrowRuntime.scala`. BotMaker supplies the rule compiler/evaluator in `botmaker/src/java/com/twitter/botmaker/compiler/Compiler.java` and `botmaker/src/scala/com/twitter/botmaker/rule/EventProcessor.scala`. Account-model outputs feed enforcement rules in `abuse-enforcement-service/service-lib/src/service.rs` and `abuse-enforcement-service/service-lib/rules/enforcement_user.yaml`.

#### Infrastructure and External Services
The system is polyglot: Rust request services, Python/JAX/CUDA model training and inference, Scala/Java rule and batch systems, plus Thrift/protobuf contracts. Material dependencies visible in the code include Kafka event transport, Manhattan/Strato storage, Memcached/Twemcache caches, Redis side effects, gRPC/Thrift RPC, BlobStore/O2 checkpoint storage, and Gizmoduck/social-graph/account services. `under-the-hood/` separately aggregates stored labels into transparency reports via Scalding and Strato.

Main entry points: [`home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs`](https://github.com/xai-org/x-algorithm/blob/main/home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs) (For You pipeline), [`home-mixer/scorers/phoenix_scorer.rs`](https://github.com/xai-org/x-algorithm/blob/main/home-mixer/scorers/phoenix_scorer.rs) (Phoenix ranking scorer), [`visibility-filtering/main.rs`](https://github.com/xai-org/x-algorithm/blob/main/visibility-filtering/main.rs) (Visibility Filtering), [`phoenix/crates/serving/xai-recsys-server/src/lib.rs`](https://github.com/xai-org/x-algorithm/blob/main/phoenix/crates/serving/xai-recsys-server/src/lib.rs) (Phoenix serving)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_feed["Feed Serving"]
  node_home_entry{{"Home Mixer<br/>feed request service<br/>[main.rs]"}}
  node_feed_servers["Feed endpoints<br/>request handlers<br/>[for_you_server.rs]"]
  node_pipeline_framework["Candidate pipeline framework<br/>pipeline runtime"]
  node_for_you_pipeline["For You pipeline<br/>feed orchestration"]
  node_hydration_filters["Hydration &amp; prefilters<br/>candidate preparation"]
  node_selection_blending["Selection &amp; blending<br/>timeline assembly"]
  node_side_effects["Response side effects<br/>event publishing"]
end

subgraph group_retrieval["Retrieval &amp; Ranking"]
  node_candidate_sources["Parallel candidate sources<br/>retrieval adapters"]
  node_thunder[("Thunder recent-post store<br/>in-network retrieval<br/>[main.rs]")]
  node_rankall["Phoenix RankAll indexer<br/>indexing pipeline<br/>[main.rs]"]
  node_ranking["Phoenix ranking scorer<br/>score composition<br/>[phoenix_scorer.rs]"]
  node_vm_ranker{{"VM Ranker<br/>diversity reranker<br/>[main.rs]"}}
end

subgraph group_safety["Safety &amp; Enforcement"]
  node_visibility{{"Visibility Filtering<br/>policy decision service<br/>[main.rs]"}}
  node_visibility_rules["Visibility rules<br/>rule registry<br/>[registry.rs]"]
  node_label_store[("Safety label source<br/>Manhattan label storage<br/>[manhattan.rs]")]
  node_grox["Grox safety classification<br/>label production pipeline<br/>[engine.py]"]
  node_media_proxy{{"Media Model Proxy<br/>media inference service<br/>[Server.scala]"}}
  node_scarecrow["Scarecrow<br/>abuse rule runtime<br/>[Server.scala]"]
  node_botmaker["BotMaker<br/>rule compiler and evaluator<br/>[Compiler.java]"]
  node_enforcement{{"Abuse Enforcement<br/>account enforcement service<br/>[service.rs]"}}
end

subgraph group_models["Model Lifecycle"]
  node_phoenix_serving{{"Phoenix serving<br/>retrieval and prediction service<br/>[lib.rs]"}}
  node_phoenix_training["Phoenix training<br/>JAX recommendation training<br/>[trainer_recsys.py]"]
end

node_home_entry -->|"routes requests"| node_feed_servers
node_feed_servers -->|"For You"| node_for_you_pipeline
node_for_you_pipeline -->|"executes with"| node_pipeline_framework
node_for_you_pipeline -->|"hydrates and filters"| node_hydration_filters
node_for_you_pipeline -->|"retrieves candidates"| node_candidate_sources
node_candidate_sources -->|"in-network posts"| node_thunder
node_candidate_sources -->|"retrievable candidates"| node_rankall
node_rankall -->|"builds retrieval data"| node_phoenix_serving
node_for_you_pipeline -->|"scores candidates"| node_ranking
node_ranking -->|"prediction RPC"| node_phoenix_serving
node_ranking -.->|"optional diversity rerank"| node_vm_ranker
node_for_you_pipeline -->|"post-selection check"| node_visibility
node_visibility -->|"evaluates"| node_visibility_rules
node_visibility -->|"hydrates labels"| node_label_store
node_for_you_pipeline -->|"selects and blends"| node_selection_blending
node_selection_blending -->|"published response"| node_side_effects
node_phoenix_training -->|"model checkpoints"| node_phoenix_serving
node_grox -->|"media inference"| node_media_proxy
node_grox -->|"produces safety labels"| node_label_store
node_scarecrow -->|"executes rules"| node_botmaker
node_scarecrow -->|"abuse outcomes"| node_enforcement

click node_home_entry "https://github.com/xai-org/x-algorithm/blob/main/home-mixer/main.rs"
click node_feed_servers "https://github.com/xai-org/x-algorithm/blob/main/home-mixer/for_you_server.rs"
click node_for_you_pipeline "https://github.com/xai-org/x-algorithm/blob/main/home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs"
click node_selection_blending "https://github.com/xai-org/x-algorithm/blob/main/home-mixer/selectors/blender_selector.rs"
click node_thunder "https://github.com/xai-org/x-algorithm/blob/main/thunder/main.rs"
click node_rankall "https://github.com/xai-org/x-algorithm/blob/main/phoenix-rankall/src/main.rs"
click node_phoenix_serving "https://github.com/xai-org/x-algorithm/blob/main/phoenix/crates/serving/xai-recsys-server/src/lib.rs"
click node_ranking "https://github.com/xai-org/x-algorithm/blob/main/home-mixer/scorers/phoenix_scorer.rs"
click node_vm_ranker "https://github.com/xai-org/x-algorithm/blob/main/vm-ranker/main.rs"
click node_visibility "https://github.com/xai-org/x-algorithm/blob/main/visibility-filtering/main.rs"
click node_visibility_rules "https://github.com/xai-org/x-algorithm/blob/main/visibility-filtering/rules/registry.rs"
click node_label_store "https://github.com/xai-org/x-algorithm/blob/main/visibility-filtering/safety_label_source/manhattan.rs"
click node_phoenix_training "https://github.com/xai-org/x-algorithm/blob/main/phoenix/xrex/train/trainer_recsys.py"
click node_grox "https://github.com/xai-org/x-algorithm/blob/main/grox/core/engine.py"
click node_media_proxy "https://github.com/xai-org/x-algorithm/blob/main/media-model-proxy/src/main/scala/com/twitter/media_understanding/model_proxy/app/Server.scala"
click node_scarecrow "https://github.com/xai-org/x-algorithm/blob/main/scarecrow/Server.scala"
click node_botmaker "https://github.com/xai-org/x-algorithm/blob/main/botmaker/src/java/com/twitter/botmaker/compiler/Compiler.java"
click node_enforcement "https://github.com/xai-org/x-algorithm/blob/main/abuse-enforcement-service/service-lib/src/service.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_home_entry,node_feed_servers,node_pipeline_framework,node_for_you_pipeline,node_hydration_filters,node_selection_blending,node_side_effects toneBlue
class node_candidate_sources,node_thunder,node_rankall,node_ranking,node_vm_ranker toneAmber
class node_visibility,node_visibility_rules,node_label_store,node_grox,node_media_proxy,node_scarecrow,node_botmaker,node_enforcement toneMint
class node_phoenix_serving,node_phoenix_training toneRose
```

## Components

### Feed Serving

- [Home Mixer](https://github.com/xai-org/x-algorithm/blob/main/home-mixer/main.rs): feed request service · Serves For You, Following, and scoring requests. · `home-mixer/main.rs`
- [Feed endpoints](https://github.com/xai-org/x-algorithm/blob/main/home-mixer/for_you_server.rs): request handlers · `home-mixer/for_you_server.rs`
- **Candidate pipeline framework**: pipeline runtime · Composable sources, hydrators, filters, scorers, selectors, and side effects.
- [For You pipeline](https://github.com/xai-org/x-algorithm/blob/main/home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs): feed orchestration · Hydrates, retrieves, ranks, selects, checks visibility, and blends output. · `home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs`
- **Hydration & prefilters**: candidate preparation · Loads viewer and candidate context; removes ineligible and seen posts.
- [Selection & blending](https://github.com/xai-org/x-algorithm/blob/main/home-mixer/selectors/blender_selector.rs): timeline assembly · Selects organic posts and blends ads, prompts, and Who-to-Follow. · `home-mixer/selectors/blender_selector.rs`
- **Response side effects**: event publishing · Publishes served history, Kafka events, caches, and metrics.

### Retrieval & Ranking

- **Parallel candidate sources**: retrieval adapters · Combines in-network, Phoenix, and SimClusters candidates.
- [Thunder recent-post store](https://github.com/xai-org/x-algorithm/blob/main/thunder/main.rs): in-network retrieval · Kafka-driven store for recent in-network posts. · `thunder/main.rs`
- [Phoenix RankAll indexer](https://github.com/xai-org/x-algorithm/blob/main/phoenix-rankall/src/main.rs): indexing pipeline · Consumes Kafka events and writes retrievable post candidates. · `phoenix-rankall/src/main.rs`
- [Phoenix ranking scorer](https://github.com/xai-org/x-algorithm/blob/main/home-mixer/scorers/phoenix_scorer.rs): score composition · Turns action predictions into weighted ranking scores and adjustments. · `home-mixer/scorers/phoenix_scorer.rs`
- [VM Ranker](https://github.com/xai-org/x-algorithm/blob/main/vm-ranker/main.rs): diversity reranker · `vm-ranker/main.rs`

### Safety & Enforcement

- [Visibility Filtering](https://github.com/xai-org/x-algorithm/blob/main/visibility-filtering/main.rs): policy decision service · Evaluates labels, relationship state, and ordered rules into allow/interstitial/drop. · `visibility-filtering/main.rs`
- [Visibility rules](https://github.com/xai-org/x-algorithm/blob/main/visibility-filtering/rules/registry.rs): rule registry · `visibility-filtering/rules/registry.rs`
- [Safety label source](https://github.com/xai-org/x-algorithm/blob/main/visibility-filtering/safety_label_source/manhattan.rs): Manhattan label storage · `visibility-filtering/safety_label_source/manhattan.rs`
- [Grox safety classification](https://github.com/xai-org/x-algorithm/blob/main/grox/core/engine.py): label production pipeline · Orchestrates post and media safety classification. · `grox/core/engine.py`
- [Media Model Proxy](https://github.com/xai-org/x-algorithm/blob/main/media-model-proxy/src/main/scala/com/twitter/media_understanding/model_proxy/app/Server.scala): media inference service · Exposes image and video models over Thrift. · `media-model-proxy/src/main/scala/com/twitter/media_understanding/model_proxy/app/Server.scala`
- [Scarecrow](https://github.com/xai-org/x-algorithm/blob/main/scarecrow/Server.scala): abuse rule runtime · Consumes events and executes BotMaker rules. · `scarecrow/Server.scala`
- [BotMaker](https://github.com/xai-org/x-algorithm/blob/main/botmaker/src/java/com/twitter/botmaker/compiler/Compiler.java): rule compiler and evaluator · `botmaker/src/java/com/twitter/botmaker/compiler/Compiler.java`
- [Abuse Enforcement](https://github.com/xai-org/x-algorithm/blob/main/abuse-enforcement-service/service-lib/src/service.rs): account enforcement service · Applies account-model outputs through enforcement rules. · `abuse-enforcement-service/service-lib/src/service.rs`

### Model Lifecycle

- [Phoenix serving](https://github.com/xai-org/x-algorithm/blob/main/phoenix/crates/serving/xai-recsys-server/src/lib.rs): retrieval and prediction service · Provides model-backed vector retrieval and ranking APIs. · `phoenix/crates/serving/xai-recsys-server/src/lib.rs`
- [Phoenix training](https://github.com/xai-org/x-algorithm/blob/main/phoenix/xrex/train/trainer_recsys.py): JAX recommendation training · Prepares recommendation data and trains model checkpoints. · `phoenix/xrex/train/trainer_recsys.py`

## Connections

- Home Mixer → Feed endpoints: routes requests
- Feed endpoints → For You pipeline: For You
- For You pipeline → Candidate pipeline framework: executes with
- For You pipeline → Hydration & prefilters: hydrates and filters
- For You pipeline → Parallel candidate sources: retrieves candidates
- Parallel candidate sources → Thunder recent-post store: in-network posts
- Parallel candidate sources → Phoenix RankAll indexer: retrievable candidates
- Phoenix RankAll indexer → Phoenix serving: builds retrieval data
- For You pipeline → Phoenix ranking scorer: scores candidates
- Phoenix ranking scorer → Phoenix serving: prediction RPC
- Phoenix ranking scorer → VM Ranker: optional diversity rerank
- For You pipeline → Visibility Filtering: post-selection check
- Visibility Filtering → Visibility rules: evaluates
- Visibility Filtering → Safety label source: hydrates labels
- For You pipeline → Selection & blending: selects and blends
- Selection & blending → Response side effects: published response
- Phoenix training → Phoenix serving: model checkpoints
- Grox safety classification → Media Model Proxy: media inference
- Grox safety classification → Safety label source: produces safety labels
- Scarecrow → BotMaker: executes rules
- Scarecrow → Abuse Enforcement: abuse outcomes
