# francobossi/tp-final-distribuidos architecture

> Architecture diagram of the GitHub repository francobossi/tp-final-distribuidos, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/francobossi/tp-final-distribuidos
- Repository: https://github.com/francobossi/tp-final-distribuidos

## Overview

Wikipedia Article Analysis streams English and French Wikipedia articles through a RabbitMQ-backed distributed pipeline to compute five query results. The main flow is client input → Gateway → byte-limited batching → normalization and query fan-out → query-specific filters, projections, and aggregations → ResultSink → client output. Shared protocol and middleware support messaging across stages. The README documents the pipeline and its capabilities; some query-stage connections below rely on that documentation because their source was not excerpted. Fault-tolerance features are described as planned, so they are not represented as implemented components.

Main entry points: [`src/result_sink/controller.py`](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/result_sink/controller.py) (Result Sink), [`src/gateway/handler.py`](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/gateway/handler.py) (Gateway), [`src/client/client.py`](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/client/client.py) (Client), [`src/query_router/controller.py`](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/query_router/controller.py) (Query Router)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingress["Input and Gateway"]
  node_client["Client<br/>[client.py]"]
  node_dataset["Parquet Reader<br/>[parquet_reader.py]"]
  node_gateway["Gateway<br/>[handler.py]"]
  node_batcher["Input Batcher<br/>[batcher.py]"]
end

subgraph group_shared["Shared Pipeline"]
  node_external_protocol["External Protocol<br/>[external.py]"]
  node_messages["Internal Messages<br/>[message.py]"]
  node_middleware["RabbitMQ Middleware<br/>[middleware.py]"]
  node_preprocessor["Preprocessor<br/>[controller.py]"]
  node_query_router["Query Router<br/>[controller.py]"]
end

subgraph group_q12["Queries 1 and 2"]
  node_until_filter["Until-2020 Filter<br/>[controller.py]"]
  node_q1["Q1 Results<br/>[controller.py]"]
  node_bio_filter["Biography Filter<br/>[controller.py]"]
  node_image_fetcher["Image Fetcher<br/>[controller.py]"]
end

subgraph group_q34["Queries 3 and 4"]
  node_word_extractor["Word Extractor<br/>[controller.py]"]
  node_q3_projector["Q3 Word Projector<br/>[controller.py]"]
  node_q3_aggregator["Q3 Aggregator<br/>[controller.py]"]
  node_q45_projector["Q4/Q5 Projector<br/>[controller.py]"]
  node_q4_counter["Q4 Word Counter<br/>[controller.py]"]
  node_q4_reducer["Q4 Top-20 Reducer<br/>[controller.py]"]
end

subgraph group_q5results["Query 5 and Results"]
  node_q5_router["Q5 Date Router<br/>[controller.py]"]
  node_q5_stats["Q5 Stats Worker<br/>[controller.py]"]
  node_q5_average["Q5 Average Reducer<br/>[controller.py]"]
  node_q5_comparator["Q5 Comparator<br/>[controller.py]"]
  node_result_sink["Result Sink<br/>[controller.py]"]
end

node_user(("User"))

node_user -->|"starts run"| node_client
node_dataset -->|"streams articles"| node_client
node_client -->|"sends articles"| node_gateway
node_client -->|"encodes requests"| node_external_protocol
node_gateway -->|"decodes requests"| node_external_protocol
node_gateway -->|"routes input"| node_batcher
node_batcher -->|"publishes batches"| node_preprocessor
node_preprocessor -->|"publishes normalized articles"| node_query_router
node_query_router -->|"fans out Q1/Q2"| node_until_filter
node_query_router -->|"routes text"| node_word_extractor
node_until_filter -->|"publishes Q1 candidates"| node_q1
node_until_filter -->|"publishes Q2 candidates"| node_bio_filter
node_bio_filter -->|"fetches images"| node_image_fetcher
node_word_extractor -->|"publishes tokens"| node_q3_projector
node_q3_projector -->|"partitions word presence"| node_q3_aggregator
node_word_extractor -->|"publishes tokens"| node_q45_projector
node_q45_projector -->|"routes Q4 words"| node_q4_counter
node_q4_counter -->|"reduces partial counts"| node_q4_reducer
node_q45_projector -->|"publishes Q5 counts"| node_q5_router
node_q5_router -->|"routes historical counts"| node_q5_stats
node_q5_stats -->|"publishes partial statistics"| node_q5_average
node_q5_router -->|"routes candidates"| node_q5_comparator
node_q5_average -->|"provides average"| node_q5_comparator
node_q1 -->|"publishes results"| node_result_sink
node_image_fetcher -->|"publishes results"| node_result_sink
node_q3_aggregator -->|"publishes results"| node_result_sink
node_q4_reducer -->|"publishes results"| node_result_sink
node_q5_comparator -->|"publishes results"| node_result_sink
node_result_sink -->|"delivers run results"| node_gateway
node_gateway -->|"returns results"| node_client
node_preprocessor -->|"serializes batches"| node_messages
node_query_router -->|"dispatches typed batches"| node_messages
node_batcher -->|"publishes messages"| node_middleware
node_gateway -->|"publishes messages"| node_middleware
node_result_sink -->|"consumes messages"| node_middleware

click node_client "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/client/client.py"
click node_dataset "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/dataset/parquet_reader.py"
click node_gateway "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/gateway/handler.py"
click node_batcher "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/input_batcher/batcher.py"
click node_external_protocol "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/common/message_protocol/external.py"
click node_messages "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/common/message_protocol/message.py"
click node_middleware "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/common/middleware/middleware.py"
click node_preprocessor "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/preprocessor/controller.py"
click node_query_router "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/query_router/controller.py"
click node_until_filter "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/until_2020_filter/controller.py"
click node_q1 "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q1/controller.py"
click node_bio_filter "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q2/biography_filter/controller.py"
click node_image_fetcher "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q2/image_fetcher/controller.py"
click node_word_extractor "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/word_extractor/controller.py"
click node_q3_projector "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q3/word_projector/controller.py"
click node_q3_aggregator "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q3/word_aggregator/controller.py"
click node_q45_projector "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q4_q5_word_projector/controller.py"
click node_q4_counter "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q4/word_counter/controller.py"
click node_q4_reducer "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q4/top20_reducer/controller.py"
click node_q5_router "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q5/date_router/controller.py"
click node_q5_stats "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q5/stats_worker/controller.py"
click node_q5_average "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q5/average_reducer/controller.py"
click node_q5_comparator "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q5/comparator/controller.py"
click node_result_sink "https://github.com/francobossi/tp-final-distribuidos/blob/main/src/result_sink/controller.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client,node_dataset,node_gateway,node_batcher,node_user toneBlue
class node_external_protocol,node_messages,node_middleware,node_preprocessor,node_query_router toneAmber
class node_until_filter,node_q1,node_bio_filter,node_image_fetcher toneMint
class node_word_extractor,node_q3_projector,node_q3_aggregator,node_q45_projector,node_q4_counter,node_q4_reducer toneRose
class node_q5_router,node_q5_stats,node_q5_average,node_q5_comparator,node_result_sink toneIndigo
```

## Components

### Input and Gateway

- [Client](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/client/client.py): `src/client/client.py`
- [Parquet Reader](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/dataset/parquet_reader.py): `src/dataset/parquet_reader.py`
- [Gateway](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/gateway/handler.py): `src/gateway/handler.py`
- [Input Batcher](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/input_batcher/batcher.py): `src/input_batcher/batcher.py`

### Shared Pipeline

- [External Protocol](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/common/message_protocol/external.py): `src/common/message_protocol/external.py`
- [Internal Messages](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/common/message_protocol/message.py): `src/common/message_protocol/message.py`
- [RabbitMQ Middleware](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/common/middleware/middleware.py): `src/common/middleware/middleware.py`
- [Preprocessor](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/preprocessor/controller.py): `src/preprocessor/controller.py`
- [Query Router](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/query_router/controller.py): `src/query_router/controller.py`

### Queries 1 and 2

- [Until-2020 Filter](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/until_2020_filter/controller.py): `src/until_2020_filter/controller.py`
- [Q1 Results](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q1/controller.py): `src/q1/controller.py`
- [Biography Filter](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q2/biography_filter/controller.py): `src/q2/biography_filter/controller.py`
- [Image Fetcher](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q2/image_fetcher/controller.py): `src/q2/image_fetcher/controller.py`

### Queries 3 and 4

- [Word Extractor](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/word_extractor/controller.py): `src/word_extractor/controller.py`
- [Q3 Word Projector](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q3/word_projector/controller.py): `src/q3/word_projector/controller.py`
- [Q3 Aggregator](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q3/word_aggregator/controller.py): `src/q3/word_aggregator/controller.py`
- [Q4/Q5 Projector](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q4_q5_word_projector/controller.py): `src/q4_q5_word_projector/controller.py`
- [Q4 Word Counter](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q4/word_counter/controller.py): `src/q4/word_counter/controller.py`
- [Q4 Top-20 Reducer](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q4/top20_reducer/controller.py): `src/q4/top20_reducer/controller.py`

### Query 5 and Results

- [Q5 Date Router](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q5/date_router/controller.py): `src/q5/date_router/controller.py`
- [Q5 Stats Worker](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q5/stats_worker/controller.py): `src/q5/stats_worker/controller.py`
- [Q5 Average Reducer](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q5/average_reducer/controller.py): `src/q5/average_reducer/controller.py`
- [Q5 Comparator](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/q5/comparator/controller.py): `src/q5/comparator/controller.py`
- [Result Sink](https://github.com/francobossi/tp-final-distribuidos/blob/main/src/result_sink/controller.py): `src/result_sink/controller.py`

### Other components

- **User**

## Connections

- User → Client: starts run
- Parquet Reader → Client: streams articles (evidence: `src/client/client.py`)
- Client → Gateway: sends articles
- Client → External Protocol: encodes requests (evidence: `src/client/client.py`)
- Gateway → External Protocol: decodes requests (evidence: `src/gateway/handler.py`)
- Gateway → Input Batcher: routes input
- Input Batcher → Preprocessor: publishes batches
- Preprocessor → Query Router: publishes normalized articles (evidence: `src/preprocessor/controller.py`)
- Query Router → Until-2020 Filter: fans out Q1/Q2 (evidence: `src/query_router/controller.py`)
- Query Router → Word Extractor: routes text (evidence: `src/query_router/controller.py`)
- Until-2020 Filter → Q1 Results: publishes Q1 candidates (evidence: `src/until_2020_filter/controller.py`)
- Until-2020 Filter → Biography Filter: publishes Q2 candidates (evidence: `src/until_2020_filter/controller.py`)
- Biography Filter → Image Fetcher: fetches images
- Word Extractor → Q3 Word Projector: publishes tokens (evidence: `src/word_extractor/controller.py`)
- Q3 Word Projector → Q3 Aggregator: partitions word presence
- Word Extractor → Q4/Q5 Projector: publishes tokens (evidence: `src/word_extractor/controller.py`)
- Q4/Q5 Projector → Q4 Word Counter: routes Q4 words (evidence: `src/q4_q5_word_projector/controller.py`)
- Q4 Word Counter → Q4 Top-20 Reducer: reduces partial counts
- Q4/Q5 Projector → Q5 Date Router: publishes Q5 counts (evidence: `src/q4_q5_word_projector/controller.py`)
- Q5 Date Router → Q5 Stats Worker: routes historical counts
- Q5 Stats Worker → Q5 Average Reducer: publishes partial statistics
- Q5 Date Router → Q5 Comparator: routes candidates
- Q5 Average Reducer → Q5 Comparator: provides average
- Q1 Results → Result Sink: publishes results
- Image Fetcher → Result Sink: publishes results
- Q3 Aggregator → Result Sink: publishes results
- Q4 Top-20 Reducer → Result Sink: publishes results
- Q5 Comparator → Result Sink: publishes results
- Result Sink → Gateway: delivers run results
- Gateway → Client: returns results
- Preprocessor → Internal Messages: serializes batches (evidence: `src/preprocessor/controller.py`)
- Query Router → Internal Messages: dispatches typed batches (evidence: `src/query_router/controller.py`)
- Input Batcher → RabbitMQ Middleware: publishes messages
- Gateway → RabbitMQ Middleware: publishes messages
- Result Sink → RabbitMQ Middleware: consumes messages (evidence: `src/result_sink/controller.py`)
