# tanayprobo/byconn_deeptech_project architecture

> Architecture diagram of the GitHub repository tanayprobo/byconn_deeptech_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tanayprobo/byconn_deeptech_project
- Repository: https://github.com/tanayprobo/byconn_deeptech_project

## Overview

BYCONN-X turns a requested URL, prompt, and JSON Schema into crawl results with page-grounded citations. The principal flow is request intake, frontier expansion and browser crawling, page cleaning, concurrent extraction and indexing, then stored results and live operator views. Separate documented capabilities include API-traffic capture, a visual browser agent, Android tooling, a public-API catalogue, and extraction evaluation. Edges follow the README’s architecture and capability descriptions or shown source references; unsampled subsystem internals are represented without speculative connections.

Main entry points: [`byconn/pipeline/cleaning.py`](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/cleaning.py) (Page cleaner), [`byconn/core/crawler.py`](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/crawler.py) (Crawler workers), [`byconn/storage/adapters.py`](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/storage/adapters.py) (Storage adapters), [`byconn/pipeline/llm_extractor.py`](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/llm_extractor.py) (Knowledge extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_crawl["Crawl runtime"]
  node_frontier["Request queue<br/>[request_queue.py]"]
  node_discovery["Link discovery<br/>[link_discovery.py]"]
  node_crawler["Crawler workers<br/>[crawler.py]"]
  node_browser["Browser pool<br/>[browser_pool.py]"]
  node_sessions["Session pool<br/>[session_pool.py]"]
end

subgraph group_extract["Extraction and indexing"]
  node_cleaner["Page cleaner<br/>[cleaning.py]"]
  node_embedder["Embedding indexer<br/>[embedder.py]"]
  node_vectors[("Qdrant")]
  node_knowledge["Knowledge extraction<br/>[llm_extractor.py]"]
  node_structured["Schema extraction<br/>[structured.py]"]
  node_entities["Entity extraction"]
end

subgraph group_serve["Service and results"]
  node_service["REST API<br/>[server.py]"]
  node_storage["Storage adapters<br/>[adapters.py]"]
  node_postgres[("PostgreSQL")]
  node_neo4j[("Neo4j graph")]
  node_results["Job results<br/>[server.py]"]
  node_dashboard["Operator console<br/>[app.js]"]
end

subgraph group_integrations["Optional integrations"]
  node_sniffer["API traffic sniffer<br/>[proxy_sniffer.py]"]
  node_openapi["OpenAPI generator"]
  node_visual["Visual agent<br/>[agent_loop.py]"]
  node_planner["Interaction planner<br/>[planner.py]"]
  node_dom["DOM parser<br/>[dom_parser.py]"]
  node_api_catalog[("Public API catalogue<br/>[api_registry.py]")]
  node_api_health["API health monitor<br/>[health_monitor.py]"]
  node_api_docs["API auto-documentation<br/>[auto_doc.py]"]
  node_api_schema["API schema detector<br/>[schema_detector.py]"]
  node_connectors["Connector generator"]
  node_android["ADB controls<br/>[adb_client.py]"]
  node_inspector["Accessibility inspector<br/>[inspector.py]"]
  node_frames["Frame decoder<br/>[frame_decoder.py]"]
end

subgraph group_tools["Evaluation"]
  node_eval_runner["Extraction evaluator<br/>[runner.py]"]
  node_eval_scores["Field scoring<br/>[scoring.py]"]
end

node_user(("Crawl requester"))

node_user -->|"submits crawl"| node_service
node_service -->|"queues requests"| node_frontier
node_frontier -->|"dispatches work"| node_crawler
node_crawler -->|"drives pages"| node_browser
node_crawler -->|"uses sessions"| node_sessions
node_crawler -->|"expands links"| node_discovery
node_browser -->|"captures traffic"| node_sniffer
node_sniffer -->|"supplies captures"| node_openapi
node_crawler -->|"cleans pages"| node_cleaner
node_cleaner -->|"saves pages"| node_storage
node_cleaner -->|"provides text"| node_embedder
node_embedder -->|"indexes vectors"| node_vectors
node_cleaner -->|"provides text"| node_knowledge
node_cleaner -->|"provides text"| node_structured
node_knowledge -->|"extracts entities"| node_entities
node_knowledge -->|"writes knowledge"| node_storage
node_structured -->|"returns cited data"| node_results
node_knowledge -->|"returns relations"| node_results
node_storage -->|"persists records"| node_postgres
node_storage -->|"writes graph"| node_neo4j
node_results -->|"stores results"| node_storage
node_service -->|"serves results"| node_results
node_service -->|"streams progress"| node_dashboard
node_visual -->|"plans actions"| node_planner
node_visual -->|"parses page"| node_dom
node_api_health -->|"checks APIs"| node_api_catalog
node_inspector -->|"uses ADB"| node_android
node_eval_runner -->|"cleans fixtures"| node_cleaner
node_eval_runner -->|"evaluates extraction"| node_structured
node_eval_runner -->|"scores fields"| node_eval_scores

click node_service "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/server.py"
click node_frontier "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/request_queue.py"
click node_discovery "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/link_discovery.py"
click node_crawler "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/crawler.py"
click node_browser "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/browser_pool.py"
click node_sessions "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/session_pool.py"
click node_sniffer "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/api_intelligence/proxy_sniffer.py"
click node_openapi "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/api_intelligence/schema_generator.py"
click node_cleaner "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/cleaning.py"
click node_embedder "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/embedder.py"
click node_knowledge "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/llm_extractor.py"
click node_structured "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/structured.py"
click node_entities "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/entity_extractor.py"
click node_storage "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/storage/adapters.py"
click node_results "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/server.py"
click node_dashboard "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/dashboard/app.js"
click node_visual "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/visual_agent/agent_loop.py"
click node_planner "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/visual_agent/planner.py"
click node_dom "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/visual_agent/dom_parser.py"
click node_api_catalog "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/free_api_integration/api_registry.py"
click node_api_health "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/free_api_integration/health_monitor.py"
click node_api_docs "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/free_api_integration/auto_doc.py"
click node_api_schema "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/free_api_integration/schema_detector.py"
click node_connectors "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/free_api_integration/connector_generator.py"
click node_android "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/mobile/adb_client.py"
click node_inspector "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/mobile/inspector.py"
click node_frames "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/mobile/frame_decoder.py"
click node_eval_runner "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/eval/runner.py"
click node_eval_scores "https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/eval/scoring.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontier,node_discovery,node_crawler,node_browser,node_sessions toneBlue
class node_cleaner,node_embedder,node_vectors,node_knowledge,node_structured,node_entities toneAmber
class node_service,node_storage,node_postgres,node_neo4j,node_results,node_dashboard,node_user toneMint
class node_sniffer,node_openapi,node_visual,node_planner,node_dom,node_api_catalog,node_api_health,node_api_docs,node_api_schema,node_connectors,node_android,node_inspector,node_frames toneRose
class node_eval_runner,node_eval_scores toneIndigo
```

## Components

### Crawl runtime

- [Request queue](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/request_queue.py): `byconn/core/request_queue.py`
- [Link discovery](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/link_discovery.py): `byconn/core/link_discovery.py`
- [Crawler workers](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/crawler.py): `byconn/core/crawler.py`
- [Browser pool](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/browser_pool.py): `byconn/core/browser_pool.py`
- [Session pool](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/core/session_pool.py): `byconn/core/session_pool.py`

### Extraction and indexing

- [Page cleaner](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/cleaning.py): `byconn/pipeline/cleaning.py`
- [Embedding indexer](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/embedder.py): `byconn/pipeline/embedder.py`
- **Qdrant**
- [Knowledge extraction](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/llm_extractor.py): `byconn/pipeline/llm_extractor.py`
- [Schema extraction](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/structured.py): `byconn/pipeline/structured.py`
- [Entity extraction](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/pipeline/entity_extractor.py): `byconn/pipeline/entity_extractor.py`

### Service and results

- [REST API](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/server.py): `server.py`
- [Storage adapters](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/storage/adapters.py): `byconn/storage/adapters.py`
- **PostgreSQL**
- **Neo4j graph**
- [Job results](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/server.py): `server.py`
- [Operator console](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/dashboard/app.js): `byconn/dashboard/app.js`

### Optional integrations

- [API traffic sniffer](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/api_intelligence/proxy_sniffer.py): `byconn/api_intelligence/proxy_sniffer.py`
- [OpenAPI generator](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/api_intelligence/schema_generator.py): `byconn/api_intelligence/schema_generator.py`
- [Visual agent](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/visual_agent/agent_loop.py): `byconn/visual_agent/agent_loop.py`
- [Interaction planner](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/visual_agent/planner.py): `byconn/visual_agent/planner.py`
- [DOM parser](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/visual_agent/dom_parser.py): `byconn/visual_agent/dom_parser.py`
- [Public API catalogue](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/free_api_integration/api_registry.py): `byconn/free_api_integration/api_registry.py`
- [API health monitor](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/free_api_integration/health_monitor.py): `byconn/free_api_integration/health_monitor.py`
- [API auto-documentation](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/free_api_integration/auto_doc.py): `byconn/free_api_integration/auto_doc.py`
- [API schema detector](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/free_api_integration/schema_detector.py): `byconn/free_api_integration/schema_detector.py`
- [Connector generator](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/free_api_integration/connector_generator.py): `byconn/free_api_integration/connector_generator.py`
- [ADB controls](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/mobile/adb_client.py): `byconn/mobile/adb_client.py`
- [Accessibility inspector](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/mobile/inspector.py): `byconn/mobile/inspector.py`
- [Frame decoder](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/mobile/frame_decoder.py): `byconn/mobile/frame_decoder.py`

### Evaluation

- [Extraction evaluator](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/eval/runner.py): `byconn/eval/runner.py`
- [Field scoring](https://github.com/tanayprobo/byconn_deeptech_project/blob/main/byconn/eval/scoring.py): `byconn/eval/scoring.py`

### Other components

- **Crawl requester**

## Connections

- Crawl requester → REST API: submits crawl (evidence: `README.md`)
- REST API → Request queue: queues requests (evidence: `README.md`)
- Request queue → Crawler workers: dispatches work (evidence: `README.md`)
- Crawler workers → Browser pool: drives pages (evidence: `byconn/core/crawler.py`)
- Crawler workers → Session pool: uses sessions (evidence: `byconn/core/crawler.py`)
- Crawler workers → Link discovery: expands links (evidence: `README.md`)
- Browser pool → API traffic sniffer: captures traffic (evidence: `byconn/core/browser_pool.py`)
- API traffic sniffer → OpenAPI generator: supplies captures (evidence: `byconn/api_intelligence/schema_generator.py`)
- Crawler workers → Page cleaner: cleans pages (evidence: `README.md`)
- Page cleaner → Storage adapters: saves pages (evidence: `README.md`)
- Page cleaner → Embedding indexer: provides text (evidence: `README.md`)
- Embedding indexer → Qdrant: indexes vectors (evidence: `README.md`)
- Page cleaner → Knowledge extraction: provides text (evidence: `README.md`)
- Page cleaner → Schema extraction: provides text (evidence: `README.md`)
- Knowledge extraction → Entity extraction: extracts entities (evidence: `byconn/pipeline/llm_extractor.py`)
- Knowledge extraction → Storage adapters: writes knowledge (evidence: `README.md`)
- Schema extraction → Job results: returns cited data (evidence: `README.md`)
- Knowledge extraction → Job results: returns relations (evidence: `README.md`)
- Storage adapters → PostgreSQL: persists records (evidence: `README.md`)
- Storage adapters → Neo4j graph: writes graph (evidence: `README.md`)
- Job results → Storage adapters: stores results (evidence: `README.md`)
- REST API → Job results: serves results (evidence: `README.md`)
- REST API → Operator console: streams progress (evidence: `README.md`)
- Visual agent → Interaction planner: plans actions (evidence: `byconn/visual_agent/agent_loop.py`)
- Visual agent → DOM parser: parses page (evidence: `byconn/visual_agent/agent_loop.py`)
- API health monitor → Public API catalogue: checks APIs (evidence: `byconn/free_api_integration/health_monitor.py`)
- Accessibility inspector → ADB controls: uses ADB (evidence: `byconn/mobile/inspector.py`)
- Extraction evaluator → Page cleaner: cleans fixtures (evidence: `byconn/eval/runner.py`)
- Extraction evaluator → Schema extraction: evaluates extraction (evidence: `byconn/eval/runner.py`)
- Extraction evaluator → Field scoring: scores fields (evidence: `byconn/eval/runner.py`)
