# rosyidal/gocrawl architecture

> Architecture diagram of the GitHub repository rosyidal/gocrawl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rosyidal/gocrawl
- Repository: https://github.com/rosyidal/gocrawl

## Overview

GoCrawl crawls configured OJS journals, extracts and cleans article metadata, filters and deduplicates records, then writes per-journal CSV/JSON output. Its web dashboard starts and monitors crawls and supports journal management, article search, analytics, citations, and exports. The sampled Java excerpts show the core crawl and metadata flow; some dashboard endpoints and implementation details are only documented in the README, so those links rely on that documentation.

Main entry points: [`src/main/java/id/gocrawl/Main.java`](https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/Main.java) (Crawl orchestrator), [`web/public/app.js`](https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js) (Browser dashboard), [`web/server.mjs`](https://github.com/rosyidal/gocrawl/blob/main/web/server.mjs) (Crawl control API), [`src/main/java/id/gocrawl/MetadataExtractor.java`](https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/MetadataExtractor.java) (Metadata extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_dashboard["Web dashboard"]
  node_browser["Browser dashboard<br/>[app.js]"]
  node_crawl_ui["Crawl controls<br/>[app.js]"]
  node_live_ui["Live monitor<br/>[app.js]"]
  node_article_ui["Article explorer<br/>[app.js]"]
  node_journal_ui["Journal manager<br/>[app.js]"]
  node_stats_ui["Statistics and export<br/>[app.js]"]
end

subgraph group_webapi["Dashboard services"]
  node_web_server["Web API server<br/>[server.mjs]"]
  node_crawl_control["Crawl control API<br/>[server.mjs]"]
  node_article_service["Article and analytics API<br/>[server.mjs]"]
  node_journal_service["Journal targets API<br/>[server.mjs]"]
end

subgraph group_crawl["Crawl engine"]
  node_java_crawler["Crawl orchestrator<br/>[Main.java]"]
end

subgraph group_metadata["Metadata processing"]
  node_url_filter["URL filtering<br/>[UrlFilter.java]"]
  node_extractor["Metadata extraction"]
  node_sanitizer["Text sanitization<br/>[TextSanitizer.java]"]
  node_quality["Quality gates<br/>[QualityGate.java]"]
  node_dedup["Deduplication"]
  node_article_model["Article record<br/>[Article.java]"]
end

node_user(("User"))
node_journal_file["Journal targets<br/>[journal.txt]"]
node_article_files["Crawl output files"]
node_ojs["OJS journals"]

node_user -->|"operates"| node_browser
node_browser -->|"selects"| node_crawl_ui
node_browser -->|"monitors"| node_live_ui
node_browser -->|"explores"| node_article_ui
node_browser -->|"manages"| node_journal_ui
node_browser -->|"views"| node_stats_ui
node_crawl_ui -.->|"starts crawl"| node_crawl_control
node_live_ui -->|"receives events"| node_crawl_control
node_article_ui -->|"queries"| node_article_service
node_journal_ui -.->|"manages targets"| node_journal_service
node_stats_ui -.->|"requests data"| node_article_service
node_web_server -->|"routes requests"| node_crawl_control
node_web_server -->|"routes requests"| node_article_service
node_web_server -->|"routes requests"| node_journal_service
node_crawl_control -->|"spawns"| node_java_crawler
node_article_service -->|"reads"| node_article_files
node_journal_service -->|"reads and writes"| node_journal_file
node_java_crawler -->|"reads targets"| node_journal_file
node_java_crawler -->|"fetches pages"| node_ojs
node_java_crawler -->|"filters links"| node_url_filter
node_java_crawler -->|"extracts metadata"| node_extractor
node_extractor -->|"cleans fields"| node_sanitizer
node_extractor -->|"creates record"| node_article_model
node_java_crawler -->|"validates records"| node_quality
node_java_crawler -->|"registers identities"| node_dedup
node_dedup -->|"normalizes values"| node_sanitizer
node_java_crawler -->|"writes results"| node_article_files

click node_browser "https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js"
click node_crawl_ui "https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js"
click node_live_ui "https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js"
click node_article_ui "https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js"
click node_journal_ui "https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js"
click node_stats_ui "https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js"
click node_web_server "https://github.com/rosyidal/gocrawl/blob/main/web/server.mjs"
click node_crawl_control "https://github.com/rosyidal/gocrawl/blob/main/web/server.mjs"
click node_article_service "https://github.com/rosyidal/gocrawl/blob/main/web/server.mjs"
click node_journal_service "https://github.com/rosyidal/gocrawl/blob/main/web/server.mjs"
click node_journal_file "https://github.com/rosyidal/gocrawl/blob/main/journal.txt"
click node_java_crawler "https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/Main.java"
click node_url_filter "https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/UrlFilter.java"
click node_extractor "https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/MetadataExtractor.java"
click node_sanitizer "https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/TextSanitizer.java"
click node_quality "https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/QualityGate.java"
click node_dedup "https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/DeduplicationEngine.java"
click node_article_model "https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/Article.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser,node_crawl_ui,node_live_ui,node_article_ui,node_journal_ui,node_stats_ui,node_user toneBlue
class node_web_server,node_crawl_control,node_article_service,node_journal_service toneAmber
class node_java_crawler toneMint
class node_url_filter,node_extractor,node_sanitizer,node_quality,node_dedup,node_article_model toneRose
class node_journal_file toneTeal
class node_article_files,node_ojs toneIndigo
```

## Components

### Web dashboard

- [Browser dashboard](https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js): `web/public/app.js`
- [Crawl controls](https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js): `web/public/app.js`
- [Live monitor](https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js): `web/public/app.js`
- [Article explorer](https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js): `web/public/app.js`
- [Journal manager](https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js): `web/public/app.js`
- [Statistics and export](https://github.com/rosyidal/gocrawl/blob/main/web/public/app.js): `web/public/app.js`

### Dashboard services

- [Web API server](https://github.com/rosyidal/gocrawl/blob/main/web/server.mjs): `web/server.mjs`
- [Crawl control API](https://github.com/rosyidal/gocrawl/blob/main/web/server.mjs): `web/server.mjs`
- [Article and analytics API](https://github.com/rosyidal/gocrawl/blob/main/web/server.mjs): `web/server.mjs`
- [Journal targets API](https://github.com/rosyidal/gocrawl/blob/main/web/server.mjs): `web/server.mjs`

### Crawl engine

- [Crawl orchestrator](https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/Main.java): `src/main/java/id/gocrawl/Main.java`

### Metadata processing

- [URL filtering](https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/UrlFilter.java): `src/main/java/id/gocrawl/UrlFilter.java`
- [Metadata extraction](https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/MetadataExtractor.java): `src/main/java/id/gocrawl/MetadataExtractor.java`
- [Text sanitization](https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/TextSanitizer.java): `src/main/java/id/gocrawl/TextSanitizer.java`
- [Quality gates](https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/QualityGate.java): `src/main/java/id/gocrawl/QualityGate.java`
- [Deduplication](https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/DeduplicationEngine.java): `src/main/java/id/gocrawl/DeduplicationEngine.java`
- [Article record](https://github.com/rosyidal/gocrawl/blob/main/src/main/java/id/gocrawl/Article.java): `src/main/java/id/gocrawl/Article.java`

### Other components

- **User**
- [Journal targets](https://github.com/rosyidal/gocrawl/blob/main/journal.txt): `journal.txt`
- **Crawl output files**
- **OJS journals**

## Connections

- User → Browser dashboard: operates (evidence: `README.md`)
- Browser dashboard → Crawl controls: selects (evidence: `web/public/app.js`)
- Browser dashboard → Live monitor: monitors (evidence: `web/public/app.js`)
- Browser dashboard → Article explorer: explores (evidence: `README.md`)
- Browser dashboard → Journal manager: manages (evidence: `README.md`)
- Browser dashboard → Statistics and export: views (evidence: `README.md`)
- Crawl controls → Crawl control API: starts crawl
- Live monitor → Crawl control API: receives events (evidence: `web/public/app.js`)
- Article explorer → Article and analytics API: queries (evidence: `web/public/app.js`)
- Journal manager → Journal targets API: manages targets
- Statistics and export → Article and analytics API: requests data
- Web API server → Crawl control API: routes requests (evidence: `web/server.mjs`)
- Web API server → Article and analytics API: routes requests (evidence: `web/server.mjs`)
- Web API server → Journal targets API: routes requests (evidence: `web/server.mjs`)
- Crawl control API → Crawl orchestrator: spawns (evidence: `web/server.mjs`)
- Article and analytics API → Crawl output files: reads (evidence: `web/server.mjs`)
- Journal targets API → Journal targets: reads and writes (evidence: `web/server.mjs`)
- Crawl orchestrator → Journal targets: reads targets (evidence: `README.md`)
- Crawl orchestrator → OJS journals: fetches pages (evidence: `src/main/java/id/gocrawl/Main.java`)
- Crawl orchestrator → URL filtering: filters links (evidence: `src/main/java/id/gocrawl/Main.java`)
- Crawl orchestrator → Metadata extraction: extracts metadata (evidence: `src/main/java/id/gocrawl/Main.java`)
- Metadata extraction → Text sanitization: cleans fields (evidence: `src/main/java/id/gocrawl/MetadataExtractor.java`)
- Metadata extraction → Article record: creates record (evidence: `src/main/java/id/gocrawl/MetadataExtractor.java`)
- Crawl orchestrator → Quality gates: validates records (evidence: `src/main/java/id/gocrawl/Main.java`)
- Crawl orchestrator → Deduplication: registers identities (evidence: `src/main/java/id/gocrawl/Main.java`)
- Deduplication → Text sanitization: normalizes values (evidence: `src/main/java/id/gocrawl/DeduplicationEngine.java`)
- Crawl orchestrator → Crawl output files: writes results (evidence: `README.md`)