How miroshyaroslav/airulescraper works
AiRuleScraper is a two-stage pipeline: crawl sites to enqueue URLs, then mine queued pages into deduplicated, technology-tagged code-review rules. The crawler and miner are separate entry points; the miner owns task status updates, extraction orchestration, similarity checks, and persistence. The README is empty, so capabilities and wiring are based on the supplied tree and sampled source; model and schema roles are inferred from their paths and imports.
Main entry points: run_miner.py Miner worker · run_crawler.py Crawler runner · ai_miner.py Rule extraction · crawler.py Internal link crawler
Components 16
URL collection
- Crawler runner
run_crawler.py - Internal link crawler
app/services/crawler.py
Rule mining
- Miner worker
run_miner.py - Article scraper
app/services/scraper.py - Rule extraction
app/services/ai_miner.py - Rule validation schema
app/schemas/rule.py - Embedding generation
app/services/ai_miner.py
Persistence
- Scraping task queue
app/models/scraping_task.py - Review rules
app/models/review_rule.py - Technology tags
app/models/technology.py - Database sessions
app/db/database.py
Configuration
- Runtime settings
app/core/config.py
Other components
- Operator
- Target websites
- Groq API
- Gemini API
Connections 21
- Operator → Crawler runner: starts crawl
- Operator → Miner worker: starts worker
- Crawler runner → Internal link crawler: requests links
- Internal link crawler → Target websites: fetches pages
- Internal link crawler → Crawler runner: returns URLs
- Crawler runner → Database sessions: opens session
- Crawler runner → Scraping task queue: enqueues URLs
- Miner worker → Database sessions: opens session
- Miner worker → Scraping task queue: claims and updates
- Miner worker → Article scraper: fetches article text
- Article scraper → Target websites: fetches page
- Miner worker → Rule extraction: extracts rules
- Rule extraction → Groq API: requests extraction
- Rule extraction → Rule validation schema: validates output
- Miner worker → Embedding generation: generates embeddings
- Embedding generation → Gemini API: requests embedding
- Miner worker → Review rules: checks similarity; saves rules
- Miner worker → Technology tags: looks up and saves tags
- Database sessions → Runtime settings: reads database URL
- Rule extraction → Runtime settings: reads API keys
- Embedding generation → Runtime settings: reads API keys