# rithesh24/autonomous-osint-agent architecture

> Architecture diagram of the GitHub repository rithesh24/autonomous-osint-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rithesh24/autonomous-osint-agent
- Repository: https://github.com/rithesh24/autonomous-osint-agent

## Overview

1. **Repository shape**
   - The repo is split into a **Next.js frontend** and a **Python backend**. The backend is the real system core: it owns orchestration, agents, storage, scheduling, and APIs. The frontend is a thin dashboard layer that reads backend endpoints and renders system state.

2. **Frontend boundary**
   - `aosia-frontend/aosia-frontend` is a Next.js 14 App Router app with pages for **activity, discover, ecosystems, graph, health, hypotheses, and investigations**.
   - Its job is mostly presentation: it calls the backend API through `lib/api.ts`, then shows live research state and system health. It does not contain business logic for discovery or analysis.

3. **Backend API surface**
   - `backend/app/api/routes/*` exposes the user-facing contract: discovery triggers, activity feed, investigation listings, hypothesis listings, ecosystems, graph data, health, validation, and scheduler status.
   - These routes are the main bridge between UI and the autonomous system. They query persistent stores and workspace services rather than directly invoking agents.

4. **Event-driven agent core**
   - The backend is organized around a **custom event bus** (`backend/app/events/bus`) and a set of **specialized agents** under `backend/app/agents`.
   - Agents do not call each other directly. They publish and subscribe to events such as `repository_discovered`, `investigation_candidate`, `investigation_completed`, `hypothesis_created`, and `memory_stored`. This event boundary is the main coupling mechanism in the architecture.

5. **Main multi-agent pipeline**
   - The pipeline flows through six functional stages:
     - **Collector Agent**: queries GitHub and emits repository discovery events.
     - **Prioritization Agent**: scores candidates and filters noise.
     - **Investigation Agent**: fetches repo evidence, analyzes README and technologies, and produces a structured investigation result.
     - **Memory Agent**: persists completed investigations into long-term memory.
     - **Hypothesis Agent**: turns evidence into ecosystem-level hypotheses and tracks confidence over time.
     - **Planner Agent**: chooses the next research objective so the loop keeps running autonomously.

6. **Investigation workflow boundary**
   - The Investigation Agent is not a free-form prompt loop; it uses **LangGraph** (`backend/app/langgraph`) as a deterministic state machine.
   - The graph moves through fetch → analyze → extract → confidence/evaluation → optional additional evidence → complete.
   - That separation matters for architecture diagrams: the investigation is a **workflow subsystem** inside the agent layer, not just a single service call.

7. **Parallel intelligence enrichment**
   - Separate from the investigation workflow, the `backend/app/intelligence` package performs broader ecosystem analysis:
     - dependency extraction and graph building,
     - contributor influence analysis,
     - ecosystem detection and clustering,
     - trend and priority scoring.
   - This is a support pipeline that enriches the system’s understanding, but it is not the primary gate for repository investigation.

8. **Memory and persistence model**
   - The system uses **three persistence layers** with different responsibilities:
     - **PostgreSQL** (`backend/app/storage/postgres`) for structured records: repositories, investigations, hypotheses, activities, ecosystems, jobs, events.
     - **ChromaDB** (`backend/app/memory/chromadb` and vector memory) for semantic recall of investigation content.
     - **Neo4j** (`backend/app/memory/neo4j` and graph memory) for relationships between repositories, technologies, and entities.
   - This is a dual-memory-plus-relational design, with Postgres as the system of record and the vector/graph stores as specialized intelligence stores.

9. **Autonomy loop**
   - The autonomy layer (`backend/app/autonomy`, `backend/app/planning`, `backend/app/scheduling`) closes the loop:
     - research objectives are generated,
     - prioritized,
     - scheduled,
     - investigated,
     - fed back into hypothesis revision.
   - APScheduler jobs trigger discovery, planning, and reevaluation cycles automatically, so the system behaves like a persistent research process rather than a request/response tool.

10. **Workspace/read model layer**
   - `backend/app/workspace/*` shapes backend data into dashboard-oriented views: activity stream, ecosystem dashboard, hypothesis dashboard, and investigation feed.
   - This layer sits between raw storage/events and the frontend, giving the UI stable read models without exposing internal agent details.

11. **Infrastructure, startup, and health**
   - Startup and service wiring live under `backend/app/core/startup`, `core/infrastructure`, and `core/config`.
   - The system also includes explicit health and validation paths: database, memory, agent, pipeline, and startup checks are surfaced through health routes and startup validation modules.
   - Redis is used for coordination/caching, and Alembic manages Postgres schema migrations.

12. **Operational concerns**
   - Observability is first-class: there are metrics, investigation history tracking, activity publication, and failure/recovery modules.
   - The design emphasizes visibility and resilience: structured logging, event persistence, and health reporting make the autonomous loop inspectable from the UI and API.

13. **Key diagram boundary to preserve**
   - A good architecture diagram should show:
     - **Frontend dashboard** consuming backend APIs,
     - **API / workspace layer** reading from stores,
     - **event bus** connecting agents,
     - **LangGraph investigation workflow** inside the Investigation Agent,
     - **Postgres / ChromaDB / Neo4j / Redis** as distinct infrastructure dependencies,
     - **scheduler + autonomy** feeding the next cycle back into discovery and planning.

Main entry points: [`backend/app/events/bus`](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/events/bus) (Event Bus), [`backend/app/agents/memory_agent`](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/memory_agent) (Memory Agent), [`backend/app/validation`](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/validation) (Health & Validation), [`backend/app/api/routes`](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/api/routes) (API Routes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Frontend"]
  node_frontend_app["Dashboard<br/>Next.js App Router UI"]
  node_frontend_api["API Client<br/>frontend data client<br/>[api.ts]"]
end

subgraph group_backend["Backend"]
  node_api_routes["API Routes<br/>FastAPI route layer"]
  node_workspace_layer["Workspace Views<br/>read models"]
  node_event_bus(("Event Bus<br/>event-driven core"))
  node_collector_agent["Collector<br/>repo discovery agent"]
  node_prioritization_agent["Prioritizer<br/>candidate scoring agent"]
  node_investigation_agent["Investigator<br/>evidence agent"]
  node_investigation_graph{{"Investigation Graph<br/>LangGraph workflow"}}
  node_memory_agent["Memory Agent<br/>persistence agent"]
  node_hypothesis_agent["Hypothesis Agent"]
  node_planner_agent["Planner Agent<br/>objective planner"]
  node_autonomy_loop["Autonomy Loop<br/>research control"]
  node_scheduler["Scheduler"]
  node_intelligence_pipeline["Intelligence<br/>ecosystem analysis"]
  node_health_validation["Health & Validation<br/>startup checks"]
end

subgraph group_infra["Infrastructure"]
  node_postgres[("Postgres<br/>system of record")]
  node_chromadb[("ChromaDB<br/>vector memory")]
  node_neo4j[("Neo4j<br/>graph memory")]
  node_redis[("Redis<br/>coordination cache")]
end

node_frontend_app -->|"reads via"| node_frontend_api
node_frontend_api -->|"calls"| node_api_routes
node_api_routes -->|"serves"| node_workspace_layer
node_workspace_layer -->|"reads"| node_postgres
node_workspace_layer -.->|"recalls"| node_chromadb
node_workspace_layer -.->|"explores"| node_neo4j
node_api_routes -->|"exposes"| node_health_validation
node_api_routes -.->|"observes"| node_event_bus
node_event_bus -->|"publishes to"| node_collector_agent
node_event_bus -->|"publishes to"| node_prioritization_agent
node_event_bus -->|"publishes to"| node_investigation_agent
node_event_bus -->|"publishes to"| node_memory_agent
node_event_bus -->|"publishes to"| node_hypothesis_agent
node_event_bus -->|"publishes to"| node_planner_agent
node_collector_agent -->|"emits"| node_prioritization_agent
node_prioritization_agent -->|"promotes"| node_investigation_agent
node_investigation_agent -->|"executes"| node_investigation_graph
node_investigation_graph -->|"completes into"| node_memory_agent
node_memory_agent -->|"stores"| node_postgres
node_memory_agent -->|"indexes"| node_chromadb
node_memory_agent -->|"links"| node_neo4j
node_memory_agent -->|"publishes"| node_event_bus
node_hypothesis_agent -->|"stores"| node_postgres
node_planner_agent -->|"feeds"| node_autonomy_loop
node_autonomy_loop -->|"schedules"| node_scheduler
node_scheduler -->|"triggers"| node_collector_agent
node_scheduler -->|"triggers"| node_planner_agent
node_scheduler -->|"runs"| node_autonomy_loop
node_intelligence_pipeline -.->|"enriches"| node_autonomy_loop
node_health_validation -.->|"checks"| node_postgres
node_health_validation -.->|"checks"| node_chromadb
node_health_validation -.->|"checks"| node_neo4j
node_health_validation -.->|"checks"| node_redis

click node_frontend_app "https://github.com/rithesh24/autonomous-osint-agent/tree/main/aosia-frontend/aosia-frontend/app"
click node_frontend_api "https://github.com/rithesh24/autonomous-osint-agent/blob/main/aosia-frontend/aosia-frontend/lib/api.ts"
click node_api_routes "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/api/routes"
click node_workspace_layer "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/workspace"
click node_event_bus "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/events/bus"
click node_collector_agent "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/collector_agent"
click node_prioritization_agent "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/prioritization_agent"
click node_investigation_agent "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/investigation_agent"
click node_investigation_graph "https://github.com/rithesh24/autonomous-osint-agent/blob/main/backend/app/langgraph/graphs/investigation_graph.py"
click node_memory_agent "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/memory_agent"
click node_hypothesis_agent "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/hypothesis_agent"
click node_planner_agent "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/planner_agent"
click node_autonomy_loop "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/autonomy"
click node_scheduler "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/scheduling"
click node_intelligence_pipeline "https://github.com/rithesh24/autonomous-osint-agent/blob/main/backend/app/intelligence/intelligence_pipeline.py"
click node_postgres "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/storage/postgres"
click node_chromadb "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/memory/chromadb"
click node_neo4j "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/memory/neo4j"
click node_redis "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/persistence/redis"
click node_health_validation "https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/validation"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend_app,node_frontend_api toneBlue
class node_api_routes,node_workspace_layer,node_event_bus,node_collector_agent,node_prioritization_agent,node_investigation_agent,node_investigation_graph,node_memory_agent,node_hypothesis_agent,node_planner_agent,node_autonomy_loop,node_scheduler,node_intelligence_pipeline,node_health_validation toneAmber
class node_postgres,node_chromadb,node_neo4j,node_redis toneMint
```

## Components

### Frontend

Next.js dashboard that presents system state and triggers backend actions.

- [Dashboard](https://github.com/rithesh24/autonomous-osint-agent/tree/main/aosia-frontend/aosia-frontend/app): Next.js App Router UI · Pages for activity, discover, ecosystems, graph, health, hypotheses, and investigations. · `aosia-frontend/aosia-frontend/app`
- [API Client](https://github.com/rithesh24/autonomous-osint-agent/blob/main/aosia-frontend/aosia-frontend/lib/api.ts): frontend data client · Thin client layer that calls backend endpoints and supplies data to pages. · `aosia-frontend/aosia-frontend/lib/api.ts`

### Backend

Autonomous research core with APIs, agents, memory, scheduling, and storage.

- [API Routes](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/api/routes): FastAPI route layer · User-facing contract for discovery, activity, investigations, hypotheses, ecosystems, graph, health, validation, and system status. · `backend/app/api/routes`
- [Workspace Views](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/workspace): read models · Dashboard-oriented services that shape raw backend data into stable UI views. · `backend/app/workspace`
- [Event Bus](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/events/bus): event-driven core · Custom publish/subscribe boundary that couples agents through events rather than direct calls. · `backend/app/events/bus`
- [Collector](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/collector_agent): repo discovery agent · Queries GitHub and emits repository discovery events. · `backend/app/agents/collector_agent`
- [Prioritizer](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/prioritization_agent): candidate scoring agent · Scores and filters discovered repositories before investigation. · `backend/app/agents/prioritization_agent`
- [Investigator](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/investigation_agent): evidence agent · Orchestrates repository analysis and produces structured investigation results. · `backend/app/agents/investigation_agent`
- [Investigation Graph](https://github.com/rithesh24/autonomous-osint-agent/blob/main/backend/app/langgraph/graphs/investigation_graph.py): LangGraph workflow · Deterministic workflow for fetch, analyze, extract, assess confidence, optionally gather more evidence, and complete. · `backend/app/langgraph/graphs/investigation_graph.py`
- [Memory Agent](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/memory_agent): persistence agent · Persists completed investigations and builds graph memory from results. · `backend/app/agents/memory_agent`
- [Hypothesis Agent](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/hypothesis_agent): hypothesis agent · Turns evidence into ecosystem hypotheses and manages confidence over time. · `backend/app/agents/hypothesis_agent`
- [Planner Agent](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/agents/planner_agent): objective planner · Selects the next research objective to keep the autonomy loop moving. · `backend/app/agents/planner_agent`
- [Autonomy Loop](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/autonomy): research control · Objective generation, prioritization, scheduling, feedback, and cycle management. · `backend/app/autonomy`
- [Scheduler](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/scheduling): APS scheduler · Runs discovery, planning, reevaluation, and autonomy cycle jobs automatically. · `backend/app/scheduling`
- [Intelligence](https://github.com/rithesh24/autonomous-osint-agent/blob/main/backend/app/intelligence/intelligence_pipeline.py): ecosystem analysis · Dependency, contributor, ecosystem, trend, and priority enrichment pipeline. · `backend/app/intelligence/intelligence_pipeline.py`
- [Health & Validation](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/validation): startup checks · Database, memory, agent, pipeline, and startup validation paths exposed to operators. · `backend/app/validation`

### Infrastructure

External state and coordination services used by the backend.

- [Postgres](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/storage/postgres): system of record · Structured storage for repositories, investigations, hypotheses, activities, ecosystems, jobs, and events. · `backend/app/storage/postgres`
- [ChromaDB](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/memory/chromadb): vector memory · Semantic recall store for investigation content and memory search. · `backend/app/memory/chromadb`
- [Neo4j](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/memory/neo4j): graph memory · Relationship store for repositories, technologies, and entities. · `backend/app/memory/neo4j`
- [Redis](https://github.com/rithesh24/autonomous-osint-agent/tree/main/backend/app/persistence/redis): coordination cache · Used for coordination and caching in the backend runtime. · `backend/app/persistence/redis`

## Connections

- Dashboard → API Client: reads via · Pages load dashboard data through the API client.
- API Client → API Routes: calls · Frontend requests backend status and research data.
- API Routes → Workspace Views: serves · Routes read dashboard-shaped views from workspace services.
- Workspace Views → Postgres: reads · Workspace views are backed by persistent structured records.
- Workspace Views → ChromaDB: recalls · Some views enrich results with semantic memory.
- Workspace Views → Neo4j: explores · Graph-oriented views use relationship memory.
- API Routes → Health & Validation: exposes · Health and validation endpoints surface runtime checks.
- API Routes → Event Bus: observes · API surfaces can reflect event and activity state.
- Event Bus → Collector: publishes to · Discovery events reach the collector stage.
- Event Bus → Prioritizer: publishes to · Discovery candidates are scored and filtered.
- Event Bus → Investigator: publishes to · Investigation candidates trigger repository analysis.
- Event Bus → Memory Agent: publishes to · Completed investigations trigger memory persistence.
- Event Bus → Hypothesis Agent: publishes to · Investigation completions feed hypothesis creation.
- Event Bus → Planner Agent: publishes to · Hypothesis events inform next-objective planning.
- Collector → Prioritizer: emits · Discovery output becomes prioritization input.
- Prioritizer → Investigator: promotes · High-value candidates advance into investigation.
- Investigator → Investigation Graph: executes · The investigation workflow runs as a deterministic state machine.
- Investigation Graph → Memory Agent: completes into · Finished investigations are handed to memory.
- Memory Agent → Postgres: stores · Structured investigations and related records are persisted.
- Memory Agent → ChromaDB: indexes · Investigation content is stored for semantic retrieval.
- Memory Agent → Neo4j: links · Entities and relationships are written to graph memory.
- Memory Agent → Event Bus: publishes · Persistence events continue the autonomous pipeline.
- Hypothesis Agent → Postgres: stores · Hypotheses and snapshots are recorded in the system of record.
- Planner Agent → Autonomy Loop: feeds · Planning decisions update the research control loop.
- Autonomy Loop → Scheduler: schedules · Autonomy decisions are turned into scheduled jobs.
- Scheduler → Collector: triggers · Scheduled discovery starts new repository collection cycles.
- Scheduler → Planner Agent: triggers · Planning and reevaluation jobs keep objectives current.
- Scheduler → Autonomy Loop: runs · Cycle jobs drive the persistent research process.
- Intelligence → Autonomy Loop: enriches · Ecosystem intelligence informs prioritization and planning.
- Health & Validation → Postgres: checks · Validation includes database state and migrations.
- Health & Validation → ChromaDB: checks · Validation includes vector memory availability.
- Health & Validation → Neo4j: checks · Validation includes graph memory availability.
- Health & Validation → Redis: checks · Validation includes coordination infrastructure.