How apurvsinghgautam/robin works
1. What this repository is
Robin is a single-process Streamlit application for dark-web OSINT investigations. The app is not split into separate backend/frontend services; instead, ui.py acts as the top-level orchestrator and calls into search, scraping, and LLM modules.
2. User-facing entry point
The main runtime entrypoint is the Streamlit UI launched with streamlit run ui.py. In Docker, entrypoint.sh and the Dockerfile package that same app into a container exposing port 8501.
3. Core application flow
The user provides an investigation query in the Streamlit UI. That query is refined by an LLM, sent to dark-web search engines, the resulting pages are scraped, and the extracted content is summarized back through the LLM. The final output is shown in the UI and can be saved as an investigation artifact.
4. LLM subsystem
llm.py and llm_utils.py encapsulate model access and prompt/response handling. The README indicates support for multiple providers: OpenAI, Claude, Gemini, Ollama, and other OpenAI-compatible APIs. This layer is the main abstraction boundary for model choice, so the rest of the app can remain provider-agnostic.
5. Search subsystem
search.py is responsible for querying dark-web search engines. This is the part most tightly coupled to network access and Tor. The architecture implies search is an independent step before scraping, rather than directly crawling arbitrary sites from the UI.
6. Scraping subsystem
scrape.py handles retrieving page content from search results or investigation targets. This sits between the search stage and the LLM summary stage, turning raw web pages into text that can be analyzed. It is the main content-ingestion boundary.
7. Configuration and environment
config.py centralizes runtime settings such as API keys, provider URLs, and other app options. .env.example documents the expected environment variables. This repo is designed so secrets and provider settings come from environment variables rather than hardcoded values.
8. Operational dependencies
Tor is a material dependency for the search flow; the README explicitly requires it for dark-web lookups. The app also depends on third-party LLM APIs or local model servers, so external connectivity is part of the runtime architecture, not just a deployment detail.
9. Persistence and investigation history
The README describes saved investigations and a “Past Investigations” panel. That means the app has a local persistence layer for investigation outputs, likely filesystem-based and mounted into the container when using Docker. This is a key boundary between ephemeral UI sessions and reusable case history.
10. Health and deployment support
health.py suggests a lightweight container health-check or runtime readiness probe. Together with the Dockerfile and entrypoint.sh, this indicates the project is deployment-friendly and optimized for a single-container runtime.
11. Streamlit-specific presentation layer
.streamlit/config.toml configures the Streamlit app’s appearance/behavior. This is not a separate subsystem, but it affects the deployment/runtime experience and confirms that the UI is the primary interaction surface.
12. Release and distribution pipeline
The GitHub Actions workflows under .github/workflows handle release automation and Docker image publication. So the repo has a clear build-and-publish boundary: source code is packaged into a versioned container image and GitHub release artifacts.
13. Architectural summary for diagramming
A good diagram would show: user → Streamlit UI (ui.py) → orchestration layer → LLM provider (llm.py/llm_utils.py) and Tor-backed search (search.py) → scraper (scrape.py) → summarized results back to UI, with config injected from environment and optional local file persistence for saved investigations.
Main entry points: ui.py Streamlit UI · llm.py LLM core · search.py Search engine · scrape.py Scraper
Components 17
Interface
User-facing Streamlit entrypoint and presentation runtime.
- Investigator · Human analyst running an OSINT investigation.
- Streamlit UI · Top-level orchestrator for query intake, investigation flow, and result presentation.
ui.py - Streamlit runtime · Streamlit execution environment launched with the app.
- Streamlit config · Streamlit appearance and behavior configuration.
.streamlit/config.toml
Core flow
Orchestration, model access, search, scraping, and persisted case data.
- App config · Central runtime settings and environment-driven configuration.
config.py - LLM core · Provider-agnostic model access and response handling.
llm.py - Prompt utils · Prompt shaping and LLM request/response utilities.
llm_utils.py - Search engine · Queries dark-web search sources for investigation leads.
search.py - Scraper · Retrieves and extracts page content from result targets.
scrape.py - Case history · Saved investigations and prior output reused across sessions.
External deps
Network and service dependencies used during investigations.
- Tor network · Required network layer for dark-web lookups and related traffic.
- LLM providers · OpenAI, Claude, Gemini, Ollama, and compatible model endpoints.
- Search targets · Search engines and reachable pages returned for investigation.
Operations
Containerization, health, and release automation.
- Docker image · Container packaging for the single-process app.
Dockerfile - Entrypoint · Container start command that launches the Streamlit app.
entrypoint.sh - Health check · Lightweight runtime or container health validation.
health.py - Release CI · Automated release and Docker image publication workflows.
.github/workflows
Connections 17
- Investigator → Streamlit UI: uses · The investigator interacts with the Streamlit app to start and inspect cases.
- Streamlit UI → App config: reads · UI behavior and provider settings are sourced from central configuration.
- Streamlit UI → LLM core: orchestrates · UI calls into the model layer for query refinement and summarization.
- LLM core → Prompt utils: uses · Model access relies on prompt/response utilities.
- Streamlit UI → Search engine: triggers · Investigation queries are sent to dark-web search.
- Search engine → Tor network: routes via · Search depends on Tor-backed network access.
- Search engine → Search targets: queries · Search engines return candidate pages and leads.
- Streamlit UI → Scraper: passes leads to · Search results are handed to the scraper for content extraction.
- Scraper → Search targets: fetches · Scraper retrieves the underlying pages and artifacts.
- Scraper → LLM core: summarizes with · Extracted content is condensed and interpreted by the LLM layer.
- LLM core → LLM providers: calls · The model layer talks to configurable provider endpoints.
- Streamlit UI → Case history: saves to · Final investigation artifacts can be stored for later review.
- Streamlit UI → Streamlit runtime: runs in · The UI executes inside the Streamlit runtime.
- Docker image → Entrypoint: starts · The container image launches the app through the entrypoint script.
- Docker image → Health check: exposes · Containerized deployment includes a health/readiness hook.
- Release CI → Docker image: builds · Release workflows package and publish the container image.
- Streamlit UI → Streamlit config: styled by · Streamlit presentation settings are applied to the UI.