# ahmdngi/shipcrawler architecture

> Architecture diagram of the GitHub repository ahmdngi/shipcrawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahmdngi/shipcrawler
- Repository: https://github.com/ahmdngi/shipcrawler

## Overview

ShipCrawler is a Flask dashboard and background-worker platform for agent-driven maritime and people OSINT investigations. Users submit a target and optional agent profile; the API queues a task, and the worker runs the documented investigation phases through Hermes, streaming progress for the dashboard while producing reports for later viewing and history lookup. The README documents vessel identity, AIS, exposure, vulnerability, threat-intelligence, and report stages, plus analyst, red-team, and detection outputs. Person research and structured report parsing are also represented. Some subsystem links rely on README descriptions because the relevant source excerpts are incomplete.

Main entry points: [`routes/api.py`](https://github.com/ahmdngi/shipcrawler/blob/main/routes/api.py) (API routes), [`worker.py`](https://github.com/ahmdngi/shipcrawler/blob/main/worker.py) (Queue worker), [`static/js/shipcrawler-core.js`](https://github.com/ahmdngi/shipcrawler/blob/main/static/js/shipcrawler-core.js) (Dashboard client), [`static/js/shipcrawler-sse.js`](https://github.com/ahmdngi/shipcrawler/blob/main/static/js/shipcrawler-sse.js) (SSE client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_web["Dashboard and API"]
  node_browser["Dashboard client"]
  node_sse["SSE client<br/>[shipcrawler-sse.js]"]
  node_flask["Flask app<br/>[app.py]"]
  node_api["API routes<br/>[api.py]"]
end

subgraph group_jobs["Task execution"]
  node_worker["Queue worker<br/>[worker.py]"]
  node_hermes["Hermes agent"]
end

subgraph group_intel["Investigation and reports"]
  node_skill["OSINT methodology<br/>[SKILL.md]"]
  node_phases["Investigation phases<br/>[worker.py]"]
  node_analysis["Analysis and tiers<br/>[analysis.py]"]
  node_reports["Report templates"]
  node_parser["Report parser<br/>[parsers.py]"]
end

subgraph group_state["Progress and results"]
  node_queue["Task queue<br/>[worker.py]"]
  node_progress["Progress logger<br/>[worker_progress.py]"]
  node_progresslog[("Progress logs<br/>[worker_progress.py]")]
  node_reportstore[("Report archive")]
end

node_user(("Analyst"))
node_maritime["Maritime OSINT sources"]

node_user -->|"submits target"| node_browser
node_browser -->|"requests scan"| node_api
node_flask -->|"registers routes"| node_api
node_api -->|"writes task"| node_queue
node_worker -->|"claims task"| node_queue
node_worker -->|"runs profile"| node_hermes
node_worker -->|"loads methodology"| node_skill
node_hermes -->|"executes investigation"| node_phases
node_phases -.->|"researches sources"| node_maritime
node_worker -->|"writes events"| node_progress
node_progress -->|"appends JSONL"| node_progresslog
node_api -->|"reads events"| node_progress
node_sse -->|"opens stream"| node_api
node_api -->|"streams events"| node_sse
node_browser -->|"connects updates"| node_sse
node_worker -->|"uses analysis"| node_analysis
node_worker -->|"generates outputs"| node_reports
node_worker -->|"saves reports"| node_reportstore
node_api -->|"loads history"| node_reportstore
node_api -->|"parses report"| node_parser
node_api -->|"returns report"| node_browser

click node_browser "https://github.com/ahmdngi/shipcrawler/blob/main/static/js/shipcrawler-core.js"
click node_sse "https://github.com/ahmdngi/shipcrawler/blob/main/static/js/shipcrawler-sse.js"
click node_flask "https://github.com/ahmdngi/shipcrawler/blob/main/app.py"
click node_api "https://github.com/ahmdngi/shipcrawler/blob/main/routes/api.py"
click node_queue "https://github.com/ahmdngi/shipcrawler/blob/main/worker.py"
click node_worker "https://github.com/ahmdngi/shipcrawler/blob/main/worker.py"
click node_skill "https://github.com/ahmdngi/shipcrawler/blob/main/skill/SKILL.md"
click node_phases "https://github.com/ahmdngi/shipcrawler/blob/main/worker.py"
click node_analysis "https://github.com/ahmdngi/shipcrawler/blob/main/services/analysis.py"
click node_reports "https://github.com/ahmdngi/shipcrawler/tree/main/templates"
click node_parser "https://github.com/ahmdngi/shipcrawler/blob/main/parsers.py"
click node_progress "https://github.com/ahmdngi/shipcrawler/blob/main/worker_progress.py"
click node_progresslog "https://github.com/ahmdngi/shipcrawler/blob/main/worker_progress.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser,node_sse,node_flask,node_api toneBlue
class node_worker,node_hermes toneAmber
class node_skill,node_phases,node_analysis,node_reports,node_parser toneMint
class node_queue,node_progress,node_progresslog,node_reportstore toneRose
class node_user,node_maritime toneIndigo
```

## Components

### Dashboard and API

- [Dashboard client](https://github.com/ahmdngi/shipcrawler/blob/main/static/js/shipcrawler-core.js): `static/js/shipcrawler-core.js`
- [SSE client](https://github.com/ahmdngi/shipcrawler/blob/main/static/js/shipcrawler-sse.js): `static/js/shipcrawler-sse.js`
- [Flask app](https://github.com/ahmdngi/shipcrawler/blob/main/app.py): `app.py`
- [API routes](https://github.com/ahmdngi/shipcrawler/blob/main/routes/api.py): `routes/api.py`

### Task execution

- [Queue worker](https://github.com/ahmdngi/shipcrawler/blob/main/worker.py): `worker.py`
- **Hermes agent**

### Investigation and reports

- [OSINT methodology](https://github.com/ahmdngi/shipcrawler/blob/main/skill/SKILL.md): `skill/SKILL.md`
- [Investigation phases](https://github.com/ahmdngi/shipcrawler/blob/main/worker.py): `worker.py`
- [Analysis and tiers](https://github.com/ahmdngi/shipcrawler/blob/main/services/analysis.py): `services/analysis.py`
- [Report templates](https://github.com/ahmdngi/shipcrawler/tree/main/templates): `templates`
- [Report parser](https://github.com/ahmdngi/shipcrawler/blob/main/parsers.py): `parsers.py`

### Progress and results

- [Task queue](https://github.com/ahmdngi/shipcrawler/blob/main/worker.py): `worker.py`
- [Progress logger](https://github.com/ahmdngi/shipcrawler/blob/main/worker_progress.py): `worker_progress.py`
- [Progress logs](https://github.com/ahmdngi/shipcrawler/blob/main/worker_progress.py): `worker_progress.py`
- **Report archive**

### Other components

- **Analyst**
- **Maritime OSINT sources**

## Connections

- Analyst → Dashboard client: submits target
- Dashboard client → API routes: requests scan
- Flask app → API routes: registers routes
- API routes → Task queue: writes task
- Queue worker → Task queue: claims task
- Queue worker → Hermes agent: runs profile
- Queue worker → OSINT methodology: loads methodology
- Hermes agent → Investigation phases: executes investigation
- Investigation phases → Maritime OSINT sources: researches sources
- Queue worker → Progress logger: writes events
- Progress logger → Progress logs: appends JSONL
- API routes → Progress logger: reads events
- SSE client → API routes: opens stream
- API routes → SSE client: streams events
- Dashboard client → SSE client: connects updates
- Queue worker → Analysis and tiers: uses analysis
- Queue worker → Report templates: generates outputs
- Queue worker → Report archive: saves reports
- API routes → Report archive: loads history
- API routes → Report parser: parses report
- API routes → Dashboard client: returns report