How tanayprobo/byconn_deeptech_project works
BYCONN-X turns a requested URL, prompt, and JSON Schema into crawl results with page-grounded citations. The principal flow is request intake, frontier expansion and browser crawling, page cleaning, concurrent extraction and indexing, then stored results and live operator views. Separate documented capabilities include API-traffic capture, a visual browser agent, Android tooling, a public-API catalogue, and extraction evaluation. Edges follow the README’s architecture and capability descriptions or shown source references; unsampled subsystem internals are represented without speculative connections.
Main entry points: cleaning.py Page cleaner · crawler.py Crawler workers · adapters.py Storage adapters · llm_extractor.py Knowledge extraction
Components 33
Crawl runtime
- Request queue
byconn/core/request_queue.py - Link discovery
byconn/core/link_discovery.py - Crawler workers
byconn/core/crawler.py - Browser pool
byconn/core/browser_pool.py - Session pool
byconn/core/session_pool.py
Extraction and indexing
- Page cleaner
byconn/pipeline/cleaning.py - Embedding indexer
byconn/pipeline/embedder.py - Qdrant
- Knowledge extraction
byconn/pipeline/llm_extractor.py - Schema extraction
byconn/pipeline/structured.py - Entity extraction
byconn/pipeline/entity_extractor.py
Service and results
- REST API
server.py - Storage adapters
byconn/storage/adapters.py - PostgreSQL
- Neo4j graph
- Job results
server.py - Operator console
byconn/dashboard/app.js
Optional integrations
- API traffic sniffer
byconn/api_intelligence/proxy_sniffer.py - OpenAPI generator
byconn/api_intelligence/schema_generator.py - Visual agent
byconn/visual_agent/agent_loop.py - Interaction planner
byconn/visual_agent/planner.py - DOM parser
byconn/visual_agent/dom_parser.py - Public API catalogue
byconn/free_api_integration/api_registry.py - API health monitor
byconn/free_api_integration/health_monitor.py - API auto-documentation
byconn/free_api_integration/auto_doc.py - API schema detector
byconn/free_api_integration/schema_detector.py - Connector generator
byconn/free_api_integration/connector_generator.py - ADB controls
byconn/mobile/adb_client.py - Accessibility inspector
byconn/mobile/inspector.py - Frame decoder
byconn/mobile/frame_decoder.py
Evaluation
- Extraction evaluator
byconn/eval/runner.py - Field scoring
byconn/eval/scoring.py
Other components
- Crawl requester
Connections 30
- Crawl requester → REST API: submits crawl
README.md - REST API → Request queue: queues requests
README.md - Request queue → Crawler workers: dispatches work
README.md - Crawler workers → Browser pool: drives pages
byconn/core/crawler.py - Crawler workers → Session pool: uses sessions
byconn/core/crawler.py - Crawler workers → Link discovery: expands links
README.md - Browser pool → API traffic sniffer: captures traffic
byconn/core/browser_pool.py - API traffic sniffer → OpenAPI generator: supplies captures
byconn/api_intelligence/schema_generator.py - Crawler workers → Page cleaner: cleans pages
README.md - Page cleaner → Storage adapters: saves pages
README.md - Page cleaner → Embedding indexer: provides text
README.md - Embedding indexer → Qdrant: indexes vectors
README.md - Page cleaner → Knowledge extraction: provides text
README.md - Page cleaner → Schema extraction: provides text
README.md - Knowledge extraction → Entity extraction: extracts entities
byconn/pipeline/llm_extractor.py - Knowledge extraction → Storage adapters: writes knowledge
README.md - Schema extraction → Job results: returns cited data
README.md - Knowledge extraction → Job results: returns relations
README.md - Storage adapters → PostgreSQL: persists records
README.md - Storage adapters → Neo4j graph: writes graph
README.md - Job results → Storage adapters: stores results
README.md - REST API → Job results: serves results
README.md - REST API → Operator console: streams progress
README.md - Visual agent → Interaction planner: plans actions
byconn/visual_agent/agent_loop.py - Visual agent → DOM parser: parses page
byconn/visual_agent/agent_loop.py - API health monitor → Public API catalogue: checks APIs
byconn/free_api_integration/health_monitor.py - Accessibility inspector → ADB controls: uses ADB
byconn/mobile/inspector.py - Extraction evaluator → Page cleaner: cleans fixtures
byconn/eval/runner.py - Extraction evaluator → Schema extraction: evaluates extraction
byconn/eval/runner.py - Extraction evaluator → Field scoring: scores fields
byconn/eval/runner.py