How francobossi/tp-final-distribuidos works
Wikipedia Article Analysis streams English and French Wikipedia articles through a RabbitMQ-backed distributed pipeline to compute five query results. The main flow is client input → Gateway → byte-limited batching → normalization and query fan-out → query-specific filters, projections, and aggregations → ResultSink → client output. Shared protocol and middleware support messaging across stages. The README documents the pipeline and its capabilities; some query-stage connections below rely on that documentation because their source was not excerpted. Fault-tolerance features are described as planned, so they are not represented as implemented components.
Main entry points: result_sink/controller.py Result Sink · handler.py Gateway · client.py Client · query_router/controller.py Query Router
Components 25
Input and Gateway
- Client
src/client/client.py - Parquet Reader
src/dataset/parquet_reader.py - Gateway
src/gateway/handler.py - Input Batcher
src/input_batcher/batcher.py
Shared Pipeline
- External Protocol
src/common/message_protocol/external.py - Internal Messages
src/common/message_protocol/message.py - RabbitMQ Middleware
src/common/middleware/middleware.py - Preprocessor
src/preprocessor/controller.py - Query Router
src/query_router/controller.py
Queries 1 and 2
- Until-2020 Filter
src/until_2020_filter/controller.py - Q1 Results
src/q1/controller.py - Biography Filter
src/q2/biography_filter/controller.py - Image Fetcher
src/q2/image_fetcher/controller.py
Queries 3 and 4
- Word Extractor
src/word_extractor/controller.py - Q3 Word Projector
src/q3/word_projector/controller.py - Q3 Aggregator
src/q3/word_aggregator/controller.py - Q4/Q5 Projector
src/q4_q5_word_projector/controller.py - Q4 Word Counter
src/q4/word_counter/controller.py - Q4 Top-20 Reducer
src/q4/top20_reducer/controller.py
Query 5 and Results
- Q5 Date Router
src/q5/date_router/controller.py - Q5 Stats Worker
src/q5/stats_worker/controller.py - Q5 Average Reducer
src/q5/average_reducer/controller.py - Q5 Comparator
src/q5/comparator/controller.py - Result Sink
src/result_sink/controller.py
Other components
- User
Connections 35
- User → Client: starts run
- Parquet Reader → Client: streams articles
src/client/client.py - Client → Gateway: sends articles
- Client → External Protocol: encodes requests
src/client/client.py - Gateway → External Protocol: decodes requests
src/gateway/handler.py - Gateway → Input Batcher: routes input
- Input Batcher → Preprocessor: publishes batches
- Preprocessor → Query Router: publishes normalized articles
src/preprocessor/controller.py - Query Router → Until-2020 Filter: fans out Q1/Q2
src/query_router/controller.py - Query Router → Word Extractor: routes text
src/query_router/controller.py - Until-2020 Filter → Q1 Results: publishes Q1 candidates
src/until_2020_filter/controller.py - Until-2020 Filter → Biography Filter: publishes Q2 candidates
src/until_2020_filter/controller.py - Biography Filter → Image Fetcher: fetches images
- Word Extractor → Q3 Word Projector: publishes tokens
src/word_extractor/controller.py - Q3 Word Projector → Q3 Aggregator: partitions word presence
- Word Extractor → Q4/Q5 Projector: publishes tokens
src/word_extractor/controller.py - Q4/Q5 Projector → Q4 Word Counter: routes Q4 words
src/q4_q5_word_projector/controller.py - Q4 Word Counter → Q4 Top-20 Reducer: reduces partial counts
- Q4/Q5 Projector → Q5 Date Router: publishes Q5 counts
src/q4_q5_word_projector/controller.py - Q5 Date Router → Q5 Stats Worker: routes historical counts
- Q5 Stats Worker → Q5 Average Reducer: publishes partial statistics
- Q5 Date Router → Q5 Comparator: routes candidates
- Q5 Average Reducer → Q5 Comparator: provides average
- Q1 Results → Result Sink: publishes results
- Image Fetcher → Result Sink: publishes results
- Q3 Aggregator → Result Sink: publishes results
- Q4 Top-20 Reducer → Result Sink: publishes results
- Q5 Comparator → Result Sink: publishes results
- Result Sink → Gateway: delivers run results
- Gateway → Client: returns results
- Preprocessor → Internal Messages: serializes batches
src/preprocessor/controller.py - Query Router → Internal Messages: dispatches typed batches
src/query_router/controller.py - Input Batcher → RabbitMQ Middleware: publishes messages
- Gateway → RabbitMQ Middleware: publishes messages
- Result Sink → RabbitMQ Middleware: consumes messages
src/result_sink/controller.py