# miroshyaroslav/airulescraper architecture

> Architecture diagram of the GitHub repository miroshyaroslav/airulescraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/miroshyaroslav/airulescraper
- Repository: https://github.com/miroshyaroslav/airulescraper

## Overview

AiRuleScraper is a two-stage pipeline: crawl sites to enqueue URLs, then mine queued pages into deduplicated, technology-tagged code-review rules. The crawler and miner are separate entry points; the miner owns task status updates, extraction orchestration, similarity checks, and persistence. The README is empty, so capabilities and wiring are based on the supplied tree and sampled source; model and schema roles are inferred from their paths and imports.

Main entry points: [`run_miner.py`](https://github.com/miroshyaroslav/airulescraper/blob/main/run_miner.py) (Miner worker), [`run_crawler.py`](https://github.com/miroshyaroslav/airulescraper/blob/main/run_crawler.py) (Crawler runner), [`app/services/ai_miner.py`](https://github.com/miroshyaroslav/airulescraper/blob/main/app/services/ai_miner.py) (Rule extraction), [`app/services/crawler.py`](https://github.com/miroshyaroslav/airulescraper/blob/main/app/services/crawler.py) (Internal link crawler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["URL collection"]
  node_crawler_runner["Crawler runner<br/>[run_crawler.py]"]
  node_link_crawler["Internal link crawler<br/>[crawler.py]"]
end

subgraph group_mining["Rule mining"]
  node_miner_runner["Miner worker<br/>[run_miner.py]"]
  node_article_scraper["Article scraper<br/>[scraper.py]"]
  node_rule_extractor["Rule extraction<br/>[ai_miner.py]"]
  node_rule_schema["Rule validation schema<br/>[rule.py]"]
  node_embedding_service["Embedding generation<br/>[ai_miner.py]"]
end

subgraph group_storage["Persistence"]
  node_task_queue["Scraping task queue<br/>[scraping_task.py]"]
  node_rule_store[("Review rules<br/>[review_rule.py]")]
  node_technology_store[("Technology tags<br/>[technology.py]")]
  node_database_session["Database sessions<br/>[database.py]"]
end

subgraph group_config["Configuration"]
  node_settings["Runtime settings<br/>[config.py]"]
end

node_operator(("Operator"))
node_sites["Target websites"]
node_groq["Groq API"]
node_gemini["Gemini API"]

node_operator -->|"starts crawl"| node_crawler_runner
node_operator -->|"starts worker"| node_miner_runner
node_crawler_runner -->|"requests links"| node_link_crawler
node_link_crawler -->|"fetches pages"| node_sites
node_link_crawler -->|"returns URLs"| node_crawler_runner
node_crawler_runner -->|"opens session"| node_database_session
node_crawler_runner -->|"enqueues URLs"| node_task_queue
node_miner_runner -->|"opens session"| node_database_session
node_miner_runner -->|"claims and updates"| node_task_queue
node_miner_runner -->|"fetches article text"| node_article_scraper
node_article_scraper -->|"fetches page"| node_sites
node_miner_runner -->|"extracts rules"| node_rule_extractor
node_rule_extractor -->|"requests extraction"| node_groq
node_rule_extractor -->|"validates output"| node_rule_schema
node_miner_runner -->|"generates embeddings"| node_embedding_service
node_embedding_service -->|"requests embedding"| node_gemini
node_miner_runner -->|"checks similarity; saves rules"| node_rule_store
node_miner_runner -->|"looks up and saves tags"| node_technology_store
node_database_session -->|"reads database URL"| node_settings
node_rule_extractor -->|"reads API keys"| node_settings
node_embedding_service -->|"reads API keys"| node_settings

click node_crawler_runner "https://github.com/miroshyaroslav/airulescraper/blob/main/run_crawler.py"
click node_link_crawler "https://github.com/miroshyaroslav/airulescraper/blob/main/app/services/crawler.py"
click node_task_queue "https://github.com/miroshyaroslav/airulescraper/blob/main/app/models/scraping_task.py"
click node_miner_runner "https://github.com/miroshyaroslav/airulescraper/blob/main/run_miner.py"
click node_article_scraper "https://github.com/miroshyaroslav/airulescraper/blob/main/app/services/scraper.py"
click node_rule_extractor "https://github.com/miroshyaroslav/airulescraper/blob/main/app/services/ai_miner.py"
click node_rule_schema "https://github.com/miroshyaroslav/airulescraper/blob/main/app/schemas/rule.py"
click node_embedding_service "https://github.com/miroshyaroslav/airulescraper/blob/main/app/services/ai_miner.py"
click node_rule_store "https://github.com/miroshyaroslav/airulescraper/blob/main/app/models/review_rule.py"
click node_technology_store "https://github.com/miroshyaroslav/airulescraper/blob/main/app/models/technology.py"
click node_database_session "https://github.com/miroshyaroslav/airulescraper/blob/main/app/db/database.py"
click node_settings "https://github.com/miroshyaroslav/airulescraper/blob/main/app/core/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_crawler_runner,node_link_crawler toneBlue
class node_miner_runner,node_article_scraper,node_rule_extractor,node_rule_schema,node_embedding_service toneAmber
class node_task_queue,node_rule_store,node_technology_store,node_database_session,node_groq,node_gemini toneMint
class node_settings toneRose
class node_operator,node_sites toneIndigo
```

## Components

### URL collection

- [Crawler runner](https://github.com/miroshyaroslav/airulescraper/blob/main/run_crawler.py): `run_crawler.py`
- [Internal link crawler](https://github.com/miroshyaroslav/airulescraper/blob/main/app/services/crawler.py): `app/services/crawler.py`

### Rule mining

- [Miner worker](https://github.com/miroshyaroslav/airulescraper/blob/main/run_miner.py): `run_miner.py`
- [Article scraper](https://github.com/miroshyaroslav/airulescraper/blob/main/app/services/scraper.py): `app/services/scraper.py`
- [Rule extraction](https://github.com/miroshyaroslav/airulescraper/blob/main/app/services/ai_miner.py): `app/services/ai_miner.py`
- [Rule validation schema](https://github.com/miroshyaroslav/airulescraper/blob/main/app/schemas/rule.py): `app/schemas/rule.py`
- [Embedding generation](https://github.com/miroshyaroslav/airulescraper/blob/main/app/services/ai_miner.py): `app/services/ai_miner.py`

### Persistence

- [Scraping task queue](https://github.com/miroshyaroslav/airulescraper/blob/main/app/models/scraping_task.py): `app/models/scraping_task.py`
- [Review rules](https://github.com/miroshyaroslav/airulescraper/blob/main/app/models/review_rule.py): `app/models/review_rule.py`
- [Technology tags](https://github.com/miroshyaroslav/airulescraper/blob/main/app/models/technology.py): `app/models/technology.py`
- [Database sessions](https://github.com/miroshyaroslav/airulescraper/blob/main/app/db/database.py): `app/db/database.py`

### Configuration

- [Runtime settings](https://github.com/miroshyaroslav/airulescraper/blob/main/app/core/config.py): `app/core/config.py`

### Other components

- **Operator**
- **Target websites**
- **Groq API**
- **Gemini API**

## Connections

- Operator → Crawler runner: starts crawl
- Operator → Miner worker: starts worker
- Crawler runner → Internal link crawler: requests links
- Internal link crawler → Target websites: fetches pages
- Internal link crawler → Crawler runner: returns URLs
- Crawler runner → Database sessions: opens session
- Crawler runner → Scraping task queue: enqueues URLs
- Miner worker → Database sessions: opens session
- Miner worker → Scraping task queue: claims and updates
- Miner worker → Article scraper: fetches article text
- Article scraper → Target websites: fetches page
- Miner worker → Rule extraction: extracts rules
- Rule extraction → Groq API: requests extraction
- Rule extraction → Rule validation schema: validates output
- Miner worker → Embedding generation: generates embeddings
- Embedding generation → Gemini API: requests embedding
- Miner worker → Review rules: checks similarity; saves rules
- Miner worker → Technology tags: looks up and saves tags
- Database sessions → Runtime settings: reads database URL
- Rule extraction → Runtime settings: reads API keys
- Embedding generation → Runtime settings: reads API keys