How outpoot/vyntr works
Main Components
• Genesis (Rust crawler & analyzer)
– Crawls web pages, fingerprints, parses HTML, generates metrics
– Sources: genesis/src
• Genesis Tools (Python & JS scripts)
– Dataset cleaning (content_cleaner.js, remove_largest.js)
– Embedding generation & benchmarks (generate_embeddings.py, benchmark.py)
– Database init & migration (init.sql, migration.sql)
• Storage
– AWS S3 (or S3-compatible) for JSONL data partitions
– PostgreSQL with pgvector for metadata, embeddings, user data
• Pulse (Rust indexing service)
– Reads cleaned JSONL/embeddings from S3
– Builds & serves full-text search indices via Tantivy
– CLI binaries: pulse/src/bin/search.rs, server.rs
• Lexicon (Bun/Node WordNet service)
– Loads WordNet into DB
– Provides dictionary lookup API
– Files: lexicon/tools/src
• Website (SvelteKit frontend & server)
– UI components under website/src/lib/components/...
– Server routes under website/src/routes/api/...
– Auth, search, chat, domain-management, user preferences
• External Services
– AWS S3, PostgreSQL, (optionally) Redis or monitoring
Architectural Patterns
• Microservices: discrete services for crawling, processing, indexing, dictionary, frontend
• Event-driven pipeline: S3 serves as the event bus between ingestion & processing
• CQRS: separate write path (Genesis & Tools) from read path (Pulse & Website)
• Polyglot stack: Rust for high-performance services, Python/JS for tooling, SvelteKit for UI
Key Technologies & Frameworks
• Rust & Cargo (Genesis, Pulse)
• Python (Genesis dataset & embedding scripts)
• Bun/Node (Lexicon service)
• SvelteKit, TypeScript (Website)
• Docker & Docker Compose (local environments)
• PostgreSQL + pgvector, Tantivy (search index)
• AWS S3 (storage), optional Redis/metrics