# senainspace/news-retrieval-engine architecture

> Architecture diagram of the GitHub repository senainspace/news-retrieval-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/senainspace/news-retrieval-engine
- Repository: https://github.com/senainspace/news-retrieval-engine

## Overview

This is a Java news-retrieval engine: it loads article records and text-cleaning resources, builds a word-to-article index, then supports ID lookup and ranked multi-term search. Its custom hash map underpins article storage and indexing; the README also documents a benchmark comparing hash and collision strategies. A separate debugger implementation is present. The graph follows the supplied excerpts and README; the CNN dataset is not in the tree, and internal details not shown are represented only where documented.

Main entry points: [`src/PreProcessing/PreProcesser.java`](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java) (Article processor), [`src/SearchEngine - run this for searching/SearchEngine.java`](https://github.com/senainspace/news-retrieval-engine/blob/main/src/SearchEngine%20-%20run%20this%20for%20searching/SearchEngine.java) (Interactive search), [`src/HashMaps/HashMapCustom.java`](https://github.com/senainspace/news-retrieval-engine/blob/main/src/HashMaps/HashMapCustom.java) (Custom hash map), [`src/Debuggers/PreProcesserDebugger.java`](https://github.com/senainspace/news-retrieval-engine/blob/main/src/Debuggers/PreProcesserDebugger.java) (Debug processor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_search["Search workflow"]
  node_engine["Interactive search<br/>[SearchEngine.java]"]
  node_processor["Article processor<br/>[PreProcesser.java]"]
  node_cleaning["Text filtering<br/>[PreProcesser.java]"]
  node_ingest["Article ingestion<br/>[PreProcesser.java]"]
  node_article["Article record<br/>[Article.java]"]
  node_dataset["CNN articles"]
  node_stopwords["Stop words<br/>[stop_words_en.txt]"]
  node_delimiters["Delimiters<br/>[delimiters.txt]"]
  node_lookup["ID lookup<br/>[PreProcesser.java]"]
  node_ranking["Query ranking<br/>[PreProcesser.java]"]
  node_results["Ranked articles<br/>[PreProcesser.java]"]
end

subgraph group_index["Index and storage"]
  node_indexing["Word indexing<br/>[PreProcesser.java]"]
  node_postings[("Word postings<br/>[PreProcesser.java]")]
  node_hashmap["Custom hash map<br/>[HashMapCustom.java]"]
  node_entry["Map entry<br/>[Entry.java]"]
end

subgraph group_benchmark["Benchmarking"]
  node_benchmark["Performance matrix"]
  node_benchprocessor["Benchmark processor"]
  node_queries["Search workload<br/>[search.txt]"]
end

subgraph group_debug["Debug tooling"]
  node_debugger["Debugger runner<br/>[Debugger.java]"]
  node_debugprocessor["Debug processor"]
end

node_user(("Search user"))

node_user -->|"enters query"| node_engine
node_engine -->|"loads and searches"| node_processor
node_engine -->|"loads"| node_stopwords
node_engine -->|"loads"| node_delimiters
node_engine -->|"loads articles"| node_dataset
node_processor -->|"loads records"| node_ingest
node_ingest -->|"creates records"| node_article
node_cleaning -->|"filters with"| node_stopwords
node_cleaning -->|"splits with"| node_delimiters
node_processor -->|"indexes articles"| node_indexing
node_indexing -->|"builds postings"| node_postings
node_processor -->|"retrieves by ID"| node_lookup
node_processor -->|"searches query"| node_ranking
node_ranking -->|"reads postings"| node_postings
node_ranking -->|"reads fields"| node_article
node_ranking -->|"ranks and emits"| node_results
node_processor -->|"uses maps"| node_hashmap
node_hashmap -->|"stores entries"| node_entry
node_benchmark -->|"configures runs"| node_benchprocessor
node_benchmark -->|"reads workload"| node_queries
node_benchprocessor -->|"configures map"| node_hashmap
node_debugger -->|"runs searches"| node_debugprocessor
node_debugprocessor -->|"uses maps"| node_hashmap
node_debugprocessor -->|"creates records"| node_article

click node_engine "https://github.com/senainspace/news-retrieval-engine/blob/main/src/SearchEngine%20-%20run%20this%20for%20searching/SearchEngine.java"
click node_processor "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java"
click node_cleaning "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java"
click node_ingest "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java"
click node_article "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/Article.java"
click node_stopwords "https://github.com/senainspace/news-retrieval-engine/blob/main/src/Data/stop_words_en.txt"
click node_delimiters "https://github.com/senainspace/news-retrieval-engine/blob/main/src/Data/delimiters.txt"
click node_indexing "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java"
click node_postings "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java"
click node_hashmap "https://github.com/senainspace/news-retrieval-engine/blob/main/src/HashMaps/HashMapCustom.java"
click node_entry "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/Entry.java"
click node_lookup "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java"
click node_ranking "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java"
click node_results "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java"
click node_benchmark "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PerformanceTesting/PerformanceTesting.java"
click node_benchprocessor "https://github.com/senainspace/news-retrieval-engine/blob/main/src/PerformanceTesting/PreProcesserPerformanceTesting.java"
click node_queries "https://github.com/senainspace/news-retrieval-engine/blob/main/src/Data/search.txt"
click node_debugger "https://github.com/senainspace/news-retrieval-engine/blob/main/src/Debuggers/Debugger.java"
click node_debugprocessor "https://github.com/senainspace/news-retrieval-engine/blob/main/src/Debuggers/PreProcesserDebugger.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_engine,node_processor,node_cleaning,node_ingest,node_article,node_dataset,node_stopwords,node_delimiters,node_lookup,node_ranking,node_results,node_user toneBlue
class node_indexing,node_postings,node_hashmap,node_entry toneAmber
class node_benchmark,node_benchprocessor,node_queries toneMint
class node_debugger,node_debugprocessor toneRose
```

## Components

### Search workflow

- [Interactive search](https://github.com/senainspace/news-retrieval-engine/blob/main/src/SearchEngine%20-%20run%20this%20for%20searching/SearchEngine.java): `src/SearchEngine - run this for searching/SearchEngine.java`
- [Article processor](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java): `src/PreProcessing/PreProcesser.java`
- [Text filtering](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java): `src/PreProcessing/PreProcesser.java`
- [Article ingestion](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java): `src/PreProcessing/PreProcesser.java`
- [Article record](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/Article.java): `src/PreProcessing/Article.java`
- **CNN articles**
- [Stop words](https://github.com/senainspace/news-retrieval-engine/blob/main/src/Data/stop_words_en.txt): `src/Data/stop_words_en.txt`
- [Delimiters](https://github.com/senainspace/news-retrieval-engine/blob/main/src/Data/delimiters.txt): `src/Data/delimiters.txt`
- [ID lookup](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java): `src/PreProcessing/PreProcesser.java`
- [Query ranking](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java): `src/PreProcessing/PreProcesser.java`
- [Ranked articles](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java): `src/PreProcessing/PreProcesser.java`

### Index and storage

- [Word indexing](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java): `src/PreProcessing/PreProcesser.java`
- [Word postings](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/PreProcesser.java): `src/PreProcessing/PreProcesser.java`
- [Custom hash map](https://github.com/senainspace/news-retrieval-engine/blob/main/src/HashMaps/HashMapCustom.java): `src/HashMaps/HashMapCustom.java`
- [Map entry](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PreProcessing/Entry.java): `src/PreProcessing/Entry.java`

### Benchmarking

- [Performance matrix](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PerformanceTesting/PerformanceTesting.java): `src/PerformanceTesting/PerformanceTesting.java`
- [Benchmark processor](https://github.com/senainspace/news-retrieval-engine/blob/main/src/PerformanceTesting/PreProcesserPerformanceTesting.java): `src/PerformanceTesting/PreProcesserPerformanceTesting.java`
- [Search workload](https://github.com/senainspace/news-retrieval-engine/blob/main/src/Data/search.txt): `src/Data/search.txt`

### Debug tooling

- [Debugger runner](https://github.com/senainspace/news-retrieval-engine/blob/main/src/Debuggers/Debugger.java): `src/Debuggers/Debugger.java`
- [Debug processor](https://github.com/senainspace/news-retrieval-engine/blob/main/src/Debuggers/PreProcesserDebugger.java): `src/Debuggers/PreProcesserDebugger.java`

### Other components

- **Search user**

## Connections

- Search user → Interactive search: enters query (evidence: `src/SearchEngine - run this for searching/SearchEngine.java`)
- Interactive search → Article processor: loads and searches (evidence: `src/SearchEngine - run this for searching/SearchEngine.java`)
- Interactive search → Stop words: loads (evidence: `src/SearchEngine - run this for searching/SearchEngine.java`)
- Interactive search → Delimiters: loads (evidence: `src/SearchEngine - run this for searching/SearchEngine.java`)
- Interactive search → CNN articles: loads articles (evidence: `src/SearchEngine - run this for searching/SearchEngine.java`)
- Article processor → Article ingestion: loads records (evidence: `README.md`)
- Article ingestion → Article record: creates records (evidence: `src/PreProcessing/PreProcesser.java`)
- Text filtering → Stop words: filters with (evidence: `README.md`)
- Text filtering → Delimiters: splits with (evidence: `README.md`)
- Article processor → Word indexing: indexes articles (evidence: `src/SearchEngine - run this for searching/SearchEngine.java`)
- Word indexing → Word postings: builds postings (evidence: `src/PreProcessing/PreProcesser.java`)
- Article processor → ID lookup: retrieves by ID (evidence: `src/SearchEngine - run this for searching/SearchEngine.java`)
- Article processor → Query ranking: searches query (evidence: `src/SearchEngine - run this for searching/SearchEngine.java`)
- Query ranking → Word postings: reads postings (evidence: `src/PreProcessing/PreProcesser.java`)
- Query ranking → Article record: reads fields (evidence: `src/PreProcessing/PreProcesser.java`)
- Query ranking → Ranked articles: ranks and emits (evidence: `README.md`)
- Article processor → Custom hash map: uses maps (evidence: `src/PreProcessing/PreProcesser.java`)
- Custom hash map → Map entry: stores entries (evidence: `src/HashMaps/HashMapCustom.java`)
- Performance matrix → Benchmark processor: configures runs (evidence: `src/PerformanceTesting/PerformanceTesting.java`)
- Performance matrix → Search workload: reads workload (evidence: `src/PerformanceTesting/PerformanceTesting.java`)
- Benchmark processor → Custom hash map: configures map (evidence: `src/PerformanceTesting/PreProcesserPerformanceTesting.java`)
- Debugger runner → Debug processor: runs searches (evidence: `src/Debuggers/Debugger.java`)
- Debug processor → Custom hash map: uses maps (evidence: `src/Debuggers/PreProcesserDebugger.java`)
- Debug processor → Article record: creates records (evidence: `src/Debuggers/PreProcesserDebugger.java`)
