How xai-org/x-algorithm works
Purpose and Entry Points
This monorepo implements X’s For You feed request path plus the offline/streaming systems that produce retrieval data, model scores, safety labels, and enforcement outcomes. The primary feed service starts at home-mixer/main.rs and serves For You, Following, and scoring variants through home-mixer/for_you_server.rs, home-mixer/following_feed_server.rs, and home-mixer/server.rs.
Pipeline Framework and Feed Orchestration
candidate-pipeline/ defines the composable execution model: sources, query/candidate hydrators, filters, scorers, selectors, and asynchronous side effects. home-mixer/ instantiates this framework into feed-specific pipelines:
• home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs
• home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs
• home-mixer/candidate_pipeline/following_candidate_pipeline.rs
A request is hydrated with viewer state, candidates are retrieved and enriched, then filtered, scored, selected, visibility-checked, blended with non-post items, and serialized into timeline output.
Retrieval and Candidate Inputs
Candidate sources run in parallel behind home-mixer/sources/. In-network retrieval uses Thunder’s recent-post store and Kafka-driven ingestion: thunder/main.rs, thunder/kafka/tweet_events_listener.rs, thunder/posts/post_store.rs. Out-of-network candidates come from Phoenix retrieval and SimClusters, called through sources such as home-mixer/sources/phoenix_source.rs and home-mixer/sources/simclusters_source.rs.
Phoenix RankAll maintains the retrievable-post index from Kafka events, processing and storing indexed candidates in phoenix-rankall/src/main.rs, phoenix-rankall/src/processor/main_processor.rs, and phoenix-rankall/src/store/writer.rs.
Hydration, Filtering, and Visibility Boundary
Query hydrators load viewer follows, blocks, mutes, engagement sequences, served history, and related context under home-mixer/query_hydrators/. Candidate hydrators fetch post, author, media, engagement, and safety attributes under home-mixer/candidate_hydrators/.
Pre-score filters remove duplicates, unavailable or ineligible content, self-posts, seen/served content, and social-graph exclusions via home-mixer/filters/. Visibility filtering is a separate post-selection service boundary: home-mixer/candidate_hydrators/vf_candidate_hydrator.rs calls visibility-filtering/, and home-mixer/filters/vf_filter.rs removes DROP decisions. The visibility service hydrates labels and relationship state, evaluates ordered rules, and returns allow/interstitial/drop:
• visibility-filtering/main.rs
• visibility-filtering/rules/registry.rs
• visibility-filtering/safety_label_source/manhattan.rs
Ranking, Selection, and Blending
Phoenix predicts per-viewer/per-post action probabilities; Home Mixer converts these predictions into weighted ranking scores and applies diversity or author adjustments. Core integration points are home-mixer/scorers/phoenix_scorer.rs, home-mixer/scorers/ranking_scorer.rs, and home-mixer/params/param.rs. Scores are optionally reranked for diversity by the standalone vm-ranker/main.rs service.
home-mixer/selectors/top_k_score_selector.rs selects ranked posts. The outer For You pipeline blends ads, prompts, and Who-to-Follow items using home-mixer/selectors/blender_selector.rs and ad-placement logic in home-mixer/ads/partition_organic_blender.rs. Post-response side effects publish served history, Kafka events, caches, and metrics under home-mixer/side_effects/.
Phoenix Model Training and Serving
phoenix/ contains the recommendation model stack: Python/JAX training and Rust serving. Training models, data preparation, and trainers are under phoenix/xrex/models/recsys_model.py, phoenix/xrex/data/recsys/, and phoenix/xrex/train/trainer_recsys.py. Rust serving exposes retrieval/ranking infrastructure and protobuf APIs in phoenix/crates/serving/xai-recsys-engine/src/lib.rs, phoenix/crates/serving/xai-recsys-server/src/lib.rs, and phoenix/crates/serving/xai-recsys-proto/proto/recsys.proto. The stack includes GPU/CUDA kernels, checkpoint storage, gRPC, and SID/vector retrieval.
Label Production, Rules, and Enforcement
Safety/abuse labels are produced off the feed request path. Grox classifies posts/media through plan/task orchestration (grox/core/engine.py, grox/flows/ptos/plan_safety_ptos.py); Media Model Proxy exposes image/video models over Thrift (media-model-proxy/src/main/scala/com/twitter/media_understanding/model_proxy/app/Server.scala, media-model-proxy/src/main/thrift/com/twitter/cortex_media_annotator/annotate.thrift).
Scarecrow consumes events and executes BotMaker rules from botmaker-rules/scarecrow/; its server/runtime are scarecrow/Server.scala and scarecrow/ScarecrowRuntime.scala. BotMaker supplies the rule compiler/evaluator in botmaker/src/java/com/twitter/botmaker/compiler/Compiler.java and botmaker/src/scala/com/twitter/botmaker/rule/EventProcessor.scala. Account-model outputs feed enforcement rules in abuse-enforcement-service/service-lib/src/service.rs and abuse-enforcement-service/service-lib/rules/enforcement_user.yaml.
Infrastructure and External Services
The system is polyglot: Rust request services, Python/JAX/CUDA model training and inference, Scala/Java rule and batch systems, plus Thrift/protobuf contracts. Material dependencies visible in the code include Kafka event transport, Manhattan/Strato storage, Memcached/Twemcache caches, Redis side effects, gRPC/Thrift RPC, BlobStore/O2 checkpoint storage, and Gizmoduck/social-graph/account services. under-the-hood/ separately aggregates stored labels into transparency reports via Scalding and Strato.
Main entry points: for_you_candidate_pipeline.rs For You pipeline · phoenix_scorer.rs Phoenix ranking scorer · main.rs Visibility Filtering · lib.rs Phoenix serving
Components 22
Feed Serving
- Home Mixer · Serves For You, Following, and scoring requests.
home-mixer/main.rs - Feed endpoints
home-mixer/for_you_server.rs - Candidate pipeline framework · Composable sources, hydrators, filters, scorers, selectors, and side effects.
- For You pipeline · Hydrates, retrieves, ranks, selects, checks visibility, and blends output.
home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs - Hydration & prefilters · Loads viewer and candidate context; removes ineligible and seen posts.
- Selection & blending · Selects organic posts and blends ads, prompts, and Who-to-Follow.
home-mixer/selectors/blender_selector.rs - Response side effects · Publishes served history, Kafka events, caches, and metrics.
Retrieval & Ranking
- Parallel candidate sources · Combines in-network, Phoenix, and SimClusters candidates.
- Thunder recent-post store · Kafka-driven store for recent in-network posts.
thunder/main.rs - Phoenix RankAll indexer · Consumes Kafka events and writes retrievable post candidates.
phoenix-rankall/src/main.rs - Phoenix ranking scorer · Turns action predictions into weighted ranking scores and adjustments.
home-mixer/scorers/phoenix_scorer.rs - VM Ranker
vm-ranker/main.rs
Safety & Enforcement
- Visibility Filtering · Evaluates labels, relationship state, and ordered rules into allow/interstitial/drop.
visibility-filtering/main.rs - Visibility rules
visibility-filtering/rules/registry.rs - Safety label source
visibility-filtering/safety_label_source/manhattan.rs - Grox safety classification · Orchestrates post and media safety classification.
grox/core/engine.py - Media Model Proxy · Exposes image and video models over Thrift.
media-model-proxy/src/main/scala/com/twitter/media_understanding/model_proxy/app/Server.scala - Scarecrow · Consumes events and executes BotMaker rules.
scarecrow/Server.scala - BotMaker
botmaker/src/java/com/twitter/botmaker/compiler/Compiler.java - Abuse Enforcement · Applies account-model outputs through enforcement rules.
abuse-enforcement-service/service-lib/src/service.rs
Model Lifecycle
- Phoenix serving · Provides model-backed vector retrieval and ranking APIs.
phoenix/crates/serving/xai-recsys-server/src/lib.rs - Phoenix training · Prepares recommendation data and trains model checkpoints.
phoenix/xrex/train/trainer_recsys.py
Connections 21
- Home Mixer → Feed endpoints: routes requests
- Feed endpoints → For You pipeline: For You
- For You pipeline → Candidate pipeline framework: executes with
- For You pipeline → Hydration & prefilters: hydrates and filters
- For You pipeline → Parallel candidate sources: retrieves candidates
- Parallel candidate sources → Thunder recent-post store: in-network posts
- Parallel candidate sources → Phoenix RankAll indexer: retrievable candidates
- Phoenix RankAll indexer → Phoenix serving: builds retrieval data
- For You pipeline → Phoenix ranking scorer: scores candidates
- Phoenix ranking scorer → Phoenix serving: prediction RPC
- Phoenix ranking scorer → VM Ranker: optional diversity rerank
- For You pipeline → Visibility Filtering: post-selection check
- Visibility Filtering → Visibility rules: evaluates
- Visibility Filtering → Safety label source: hydrates labels
- For You pipeline → Selection & blending: selects and blends
- Selection & blending → Response side effects: published response
- Phoenix training → Phoenix serving: model checkpoints
- Grox safety classification → Media Model Proxy: media inference
- Grox safety classification → Safety label source: produces safety labels
- Scarecrow → BotMaker: executes rules
- Scarecrow → Abuse Enforcement: abuse outcomes