How rosyidal/gocrawl works
GoCrawl crawls configured OJS journals, extracts and cleans article metadata, filters and deduplicates records, then writes per-journal CSV/JSON output. Its web dashboard starts and monitors crawls and supports journal management, article search, analytics, citations, and exports. The sampled Java excerpts show the core crawl and metadata flow; some dashboard endpoints and implementation details are only documented in the README, so those links rely on that documentation.
Main entry points: Main.java Crawl orchestrator · app.js Browser dashboard · server.mjs Crawl control API · MetadataExtractor.java Metadata extraction
Components 21
Web dashboard
- Browser dashboard
web/public/app.js - Crawl controls
web/public/app.js - Live monitor
web/public/app.js - Article explorer
web/public/app.js - Journal manager
web/public/app.js - Statistics and export
web/public/app.js
Dashboard services
- Web API server
web/server.mjs - Crawl control API
web/server.mjs - Article and analytics API
web/server.mjs - Journal targets API
web/server.mjs
Crawl engine
- Crawl orchestrator
src/main/java/id/gocrawl/Main.java
Metadata processing
- URL filtering
src/main/java/id/gocrawl/UrlFilter.java - Metadata extraction
src/main/java/id/gocrawl/MetadataExtractor.java - Text sanitization
src/main/java/id/gocrawl/TextSanitizer.java - Quality gates
src/main/java/id/gocrawl/QualityGate.java - Deduplication
src/main/java/id/gocrawl/DeduplicationEngine.java - Article record
src/main/java/id/gocrawl/Article.java
Other components
- User
- Journal targets
journal.txt - Crawl output files
- OJS journals
Connections 27
- User → Browser dashboard: operates
README.md - Browser dashboard → Crawl controls: selects
web/public/app.js - Browser dashboard → Live monitor: monitors
web/public/app.js - Browser dashboard → Article explorer: explores
README.md - Browser dashboard → Journal manager: manages
README.md - Browser dashboard → Statistics and export: views
README.md - Crawl controls → Crawl control API: starts crawl
- Live monitor → Crawl control API: receives events
web/public/app.js - Article explorer → Article and analytics API: queries
web/public/app.js - Journal manager → Journal targets API: manages targets
- Statistics and export → Article and analytics API: requests data
- Web API server → Crawl control API: routes requests
web/server.mjs - Web API server → Article and analytics API: routes requests
web/server.mjs - Web API server → Journal targets API: routes requests
web/server.mjs - Crawl control API → Crawl orchestrator: spawns
web/server.mjs - Article and analytics API → Crawl output files: reads
web/server.mjs - Journal targets API → Journal targets: reads and writes
web/server.mjs - Crawl orchestrator → Journal targets: reads targets
README.md - Crawl orchestrator → OJS journals: fetches pages
src/main/java/id/gocrawl/Main.java - Crawl orchestrator → URL filtering: filters links
src/main/java/id/gocrawl/Main.java - Crawl orchestrator → Metadata extraction: extracts metadata
src/main/java/id/gocrawl/Main.java - Metadata extraction → Text sanitization: cleans fields
src/main/java/id/gocrawl/MetadataExtractor.java - Metadata extraction → Article record: creates record
src/main/java/id/gocrawl/MetadataExtractor.java - Crawl orchestrator → Quality gates: validates records
src/main/java/id/gocrawl/Main.java - Crawl orchestrator → Deduplication: registers identities
src/main/java/id/gocrawl/Main.java - Deduplication → Text sanitization: normalizes values
src/main/java/id/gocrawl/DeduplicationEngine.java - Crawl orchestrator → Crawl output files: writes results
README.md