# janavrana/osint architecture

> Architecture diagram of the GitHub repository janavrana/osint, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/janavrana/osint
- Repository: https://github.com/janavrana/osint

## Overview

#### Purpose and Deployment Shape
OSINT is an intelligence-aggregation system centered on investigations of identifiers, collecting registry data, normalizing it into facts, and deriving correlation, graph, timeline, and report views. It has a Python backend, a separate React client, and an optional graph-service package. Container deployment is represented by `docker-compose.yml`, `backend/Dockerfile`, and `graph/docker-compose-additions.yml`.

#### Entry Points and API Boundary
The backend application starts at `backend/app/main.py`. Its HTTP surface is assembled through `backend/app/api/routes.py`, with health checking at `backend/app/api/health.py` and investigation-oriented versioned routes at `backend/app/api/v1/investigations.py`. API schemas isolate transport contracts from domain persistence in `backend/app/schemas/investigation.py` and `backend/app/schemas/timeline.py`.

The frontend enters through `frontend/src/main.tsx` and `frontend/src/App.tsx`; its backend boundary is centralized in `frontend/src/services/api.ts`. Existing pages include investigation, graph, report, and timeline screens, though the README identifies backend integration and these visualizations as incomplete.

#### Investigation Execution Pipeline
`backend/app/services/investigation_service.py` coordinates investigation lifecycle operations backed by `backend/app/repositories/investigation_repository.py` and the persisted `backend/app/models/investigation.py` model.

Connector execution is separated into `backend/app/services/connector_execution_service.py`. It selects registered implementations through `backend/app/connectors/manager.py` and `backend/app/connectors/registry.py`, using a common connector contract in `backend/app/connectors/base.py`. Current external intelligence sources are WHOIS (`backend/app/connectors/whois/connector.py`) and RDAP (`backend/app/connectors/rdap/connector.py`); raw outcomes are represented and stored as connector results via `backend/app/models/connector_result.py` and `backend/app/repositories/connector_result_repository.py`.

#### Normalized Fact Boundary
Source-specific responses are converted into a source-independent fact layer by `backend/app/normalizers/manager.py` and `backend/app/normalizers/base.py`. WHOIS and RDAP transformations live in `backend/app/normalizers/whois/normalizer.py` and `backend/app/normalizers/rdap/normalizer.py`.

Normalized facts are a material persistence boundary: `backend/app/models/normalized_fact.py`, `backend/app/repositories/normalized_fact_repository.py`, and `backend/app/services/normalized_fact_service.py`. The Alembic migration `backend/alembic/versions/708401118290_add_normalized_facts_table.py` indicates this store is intended to support downstream analysis independent of raw connector payloads.

#### Analysis Products
Correlation derives relationships or findings from normalized data through `backend/app/correlation/engine.py` and its typed contracts in `backend/app/correlation/types.py`.

Graph generation/query functionality is layered under `backend/app/graph/`: generation and mapping (`generator.py`, `mapper.py`), domain models (`models.py`), and query-facing service logic (`query_service.py`, `service.py`). A distinct graph runtime integration is suggested by `graph/service.py`, `graph/connection.py`, and `graph/config.py`, keeping graph infrastructure separate from the primary backend.

Timeline construction is handled by `backend/app/timeline/service.py` over `backend/app/timeline/models.py`. Reporting composes durable report records and generated sections via `backend/app/reporting/report_generator.py`, `backend/app/reporting/sections.py`, and `backend/app/services/report_service.py`.

#### Persistence, Runtime, and Observability
The backend uses a database session boundary at `backend/app/db/session.py`, SQL-model-style domain entities under `backend/app/models/`, repositories for persistence access, and Alembic under `backend/alembic/` for schema evolution. Configuration, logging, and metrics are centralized in `backend/app/core/config.py`, `backend/app/core/logging.py`, and `backend/app/core/metrics.py`.

Material external dependencies are WHOIS and RDAP services. The frontend is TypeScript/React, while backend and graph packages are Python-based; backend dependencies are declared in `backend/requirements.txt`, with graph-specific additions in `graph/requirements-additions.txt`.

Main entry points: [`backend/app/models/normalized_fact.py`](https://github.com/janavrana/osint/blob/main/backend/app/models/normalized_fact.py) (Normalized facts), [`backend/app/db/session.py`](https://github.com/janavrana/osint/blob/main/backend/app/db/session.py) (Database session and models), [`backend/app/services/investigation_service.py`](https://github.com/janavrana/osint/blob/main/backend/app/services/investigation_service.py) (Investigation lifecycle), [`backend/app/models/connector_result.py`](https://github.com/janavrana/osint/blob/main/backend/app/models/connector_result.py) (Raw connector results)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["React client"]
  node_client["React application<br/>TypeScript/React<br/>[App.tsx]"]
  node_client_api["Backend API client<br/>HTTP client<br/>[api.ts]"]
end

subgraph group_api["Backend API"]
  node_backend_app["Backend application<br/>Python app<br/>[main.py]"]
  node_routes["Routes and schemas<br/>HTTP API<br/>[routes.py]"]
end

subgraph group_pipeline["Investigation pipeline"]
  node_investigations["Investigation lifecycle"]
  node_execution["Connector execution"]
  node_connectors["Connector registry<br/>plugin registry<br/>[manager.py]"]
  node_normalization["Fact normalization<br/>normalizer manager<br/>[manager.py]"]
end

subgraph group_analysis["Analysis products"]
  node_correlation["Correlation engine<br/>analysis engine<br/>[engine.py]"]
  node_graph_backend["Graph generation and queries<br/>graph service layer<br/>[service.py]"]
  node_graph_runtime{{"Graph runtime integration<br/>Python graph package<br/>[service.py]"}}
  node_timeline["Timeline builder<br/>timeline service<br/>[service.py]"]
  node_reporting["Report generation<br/>report service"]
end

subgraph group_data["Persistence &amp; runtime"]
  node_connector_results[("Raw connector results<br/>persisted model")]
  node_facts[("Normalized facts<br/>persisted fact store<br/>[normalized_fact.py]")]
  node_database[("Database session and models<br/>SQL persistence<br/>[session.py]")]
  node_migrations["Schema migrations<br/>Alembic"]
  node_deployment["Container deployment<br/>Docker Compose<br/>[docker-compose.yml]"]
end

subgraph group_external["External intelligence"]
  node_sources{{"WHOIS and RDAP<br/>connectors<br/>[connector.py]"}}
end

node_client -->|"uses"| node_client_api
node_client_api -->|"HTTP requests"| node_routes
node_backend_app -->|"assembles"| node_routes
node_routes -->|"investigation operations"| node_investigations
node_investigations -->|"runs collection"| node_execution
node_investigations -->|"persists investigations"| node_database
node_execution -->|"selects connector"| node_connectors
node_connectors -->|"dispatches"| node_sources
node_sources -->|"raw outcomes"| node_connector_results
node_connector_results -->|"source payloads"| node_normalization
node_normalization -->|"normalized facts"| node_facts
node_facts -->|"analyzes"| node_correlation
node_facts -->|"maps"| node_graph_backend
node_facts -->|"orders events"| node_timeline
node_facts -->|"report inputs"| node_reporting
node_correlation -->|"relationships"| node_graph_backend
node_graph_backend -.->|"graph infrastructure"| node_graph_runtime
node_timeline -->|"timeline section"| node_reporting
node_connector_results -->|"stored through repository"| node_database
node_facts -->|"stored through repository"| node_database
node_migrations -->|"evolves schema"| node_database
node_deployment -->|"runs"| node_backend_app
node_deployment -.->|"optional additions"| node_graph_runtime

click node_client "https://github.com/janavrana/osint/blob/main/frontend/src/App.tsx"
click node_client_api "https://github.com/janavrana/osint/blob/main/frontend/src/services/api.ts"
click node_backend_app "https://github.com/janavrana/osint/blob/main/backend/app/main.py"
click node_routes "https://github.com/janavrana/osint/blob/main/backend/app/api/routes.py"
click node_investigations "https://github.com/janavrana/osint/blob/main/backend/app/services/investigation_service.py"
click node_execution "https://github.com/janavrana/osint/blob/main/backend/app/services/connector_execution_service.py"
click node_connectors "https://github.com/janavrana/osint/blob/main/backend/app/connectors/manager.py"
click node_sources "https://github.com/janavrana/osint/blob/main/backend/app/connectors/whois/connector.py"
click node_connector_results "https://github.com/janavrana/osint/blob/main/backend/app/models/connector_result.py"
click node_normalization "https://github.com/janavrana/osint/blob/main/backend/app/normalizers/manager.py"
click node_facts "https://github.com/janavrana/osint/blob/main/backend/app/models/normalized_fact.py"
click node_correlation "https://github.com/janavrana/osint/blob/main/backend/app/correlation/engine.py"
click node_graph_backend "https://github.com/janavrana/osint/blob/main/backend/app/graph/service.py"
click node_graph_runtime "https://github.com/janavrana/osint/blob/main/graph/service.py"
click node_timeline "https://github.com/janavrana/osint/blob/main/backend/app/timeline/service.py"
click node_reporting "https://github.com/janavrana/osint/blob/main/backend/app/reporting/report_generator.py"
click node_database "https://github.com/janavrana/osint/blob/main/backend/app/db/session.py"
click node_migrations "https://github.com/janavrana/osint/blob/main/backend/alembic/versions/708401118290_add_normalized_facts_table.py"
click node_deployment "https://github.com/janavrana/osint/blob/main/docker-compose.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client,node_client_api toneBlue
class node_backend_app,node_routes toneAmber
class node_investigations,node_execution,node_connectors,node_normalization toneMint
class node_correlation,node_graph_backend,node_graph_runtime,node_timeline,node_reporting toneRose
class node_connector_results,node_facts,node_database,node_migrations,node_deployment toneIndigo
class node_sources toneTeal
```

## Components

### React client

- [React application](https://github.com/janavrana/osint/blob/main/frontend/src/App.tsx): TypeScript/React · `frontend/src/App.tsx`
- [Backend API client](https://github.com/janavrana/osint/blob/main/frontend/src/services/api.ts): HTTP client · `frontend/src/services/api.ts`

### Backend API

- [Backend application](https://github.com/janavrana/osint/blob/main/backend/app/main.py): Python app · `backend/app/main.py`
- [Routes and schemas](https://github.com/janavrana/osint/blob/main/backend/app/api/routes.py): HTTP API · Health and versioned investigation endpoints use transport schemas. · `backend/app/api/routes.py`

### Investigation pipeline

- [Investigation lifecycle](https://github.com/janavrana/osint/blob/main/backend/app/services/investigation_service.py): service · `backend/app/services/investigation_service.py`
- [Connector execution](https://github.com/janavrana/osint/blob/main/backend/app/services/connector_execution_service.py): service · `backend/app/services/connector_execution_service.py`
- [Connector registry](https://github.com/janavrana/osint/blob/main/backend/app/connectors/manager.py): plugin registry · Selects implementations behind the common connector contract. · `backend/app/connectors/manager.py`
- [Fact normalization](https://github.com/janavrana/osint/blob/main/backend/app/normalizers/manager.py): normalizer manager · Converts source-specific payloads into source-independent facts. · `backend/app/normalizers/manager.py`

### Analysis products

- [Correlation engine](https://github.com/janavrana/osint/blob/main/backend/app/correlation/engine.py): analysis engine · `backend/app/correlation/engine.py`
- [Graph generation and queries](https://github.com/janavrana/osint/blob/main/backend/app/graph/service.py): graph service layer · `backend/app/graph/service.py`
- [Graph runtime integration](https://github.com/janavrana/osint/blob/main/graph/service.py): Python graph package · Separate graph infrastructure integration. · `graph/service.py`
- [Timeline builder](https://github.com/janavrana/osint/blob/main/backend/app/timeline/service.py): timeline service · `backend/app/timeline/service.py`
- [Report generation](https://github.com/janavrana/osint/blob/main/backend/app/reporting/report_generator.py): report service · Composes durable reports and generated sections. · `backend/app/reporting/report_generator.py`

### Persistence & runtime

- [Raw connector results](https://github.com/janavrana/osint/blob/main/backend/app/models/connector_result.py): persisted model · `backend/app/models/connector_result.py`
- [Normalized facts](https://github.com/janavrana/osint/blob/main/backend/app/models/normalized_fact.py): persisted fact store · Durable analysis boundary independent of raw payloads. · `backend/app/models/normalized_fact.py`
- [Database session and models](https://github.com/janavrana/osint/blob/main/backend/app/db/session.py): SQL persistence · Repositories access domain entities through the session boundary. · `backend/app/db/session.py`
- [Schema migrations](https://github.com/janavrana/osint/blob/main/backend/alembic/versions/708401118290_add_normalized_facts_table.py): Alembic · `backend/alembic/versions/708401118290_add_normalized_facts_table.py`
- [Container deployment](https://github.com/janavrana/osint/blob/main/docker-compose.yml): Docker Compose · Runs backend with optional graph-service additions. · `docker-compose.yml`

### External intelligence

- [WHOIS and RDAP](https://github.com/janavrana/osint/blob/main/backend/app/connectors/whois/connector.py): connectors · `backend/app/connectors/whois/connector.py`

## Connections

- React application → Backend API client: uses
- Backend API client → Routes and schemas: HTTP requests
- Backend application → Routes and schemas: assembles
- Routes and schemas → Investigation lifecycle: investigation operations
- Investigation lifecycle → Connector execution: runs collection
- Investigation lifecycle → Database session and models: persists investigations
- Connector execution → Connector registry: selects connector
- Connector registry → WHOIS and RDAP: dispatches
- WHOIS and RDAP → Raw connector results: raw outcomes
- Raw connector results → Fact normalization: source payloads
- Fact normalization → Normalized facts: normalized facts
- Normalized facts → Correlation engine: analyzes
- Normalized facts → Graph generation and queries: maps
- Normalized facts → Timeline builder: orders events
- Normalized facts → Report generation: report inputs
- Correlation engine → Graph generation and queries: relationships
- Graph generation and queries → Graph runtime integration: graph infrastructure
- Timeline builder → Report generation: timeline section
- Raw connector results → Database session and models: stored through repository
- Normalized facts → Database session and models: stored through repository
- Schema migrations → Database session and models: evolves schema
- Container deployment → Backend application: runs
- Container deployment → Graph runtime integration: optional additions