# kmu-aws-capstone-team-4/scraping architecture

> Architecture diagram of the GitHub repository kmu-aws-capstone-team-4/scraping, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kmu-aws-capstone-team-4/scraping
- Repository: https://github.com/kmu-aws-capstone-team-4/scraping

## Overview

This is a Korean job-posting scraper that turns a posting URL into normalized job data, with CLI JSON output or asynchronous database persistence. The main workflow is URL entry → plugin-aware HTTP/Playwright collection → HTML and optional image OCR → LLM structuring and normalization → result delivery. The README documents platform-specific scraping, anti-bot handling, and Celery/Redis support. Several relationships are supported by README statements or the source index; unsampled implementation details are not expanded into additional flows.

Main entry points: [`pipeline.py`](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/pipeline.py) (Collection pipeline), [`main.py`](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/main.py) (CLI runner), [`plugins/registry.py`](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/registry.py) (Plugin registry), [`extractors/llm_extractor.py`](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/extractors/llm_extractor.py) (Job field extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry and orchestration"]
  node_cli["CLI runner<br/>[main.py]"]
  node_scraper["Scraping entry<br/>[scraper.py]"]
  node_pipeline["Collection pipeline<br/>[pipeline.py]"]
end

subgraph group_collection["Page collection"]
  node_browser["Browser manager<br/>[browser.py]"]
  node_anti["Anti-bot handling<br/>[anti_bot.py]"]
  node_plugins["Plugin registry<br/>[registry.py]"]
  node_pluginbase["Plugin interface<br/>[base.py]"]
  node_default["Generic scraper<br/>[default.py]"]
  node_jobkorea["JobKorea scraper<br/>[jobkorea.py]"]
  node_jobplanet["JobPlanet scraper<br/>[jobplanet.py]"]
end

subgraph group_extraction["Content extraction"]
  node_htmltext["HTML text cleanup<br/>[llm_extractor.py]"]
  node_vision["Image OCR<br/>[llm_extractor.py]"]
  node_llm["Job field extraction<br/>[llm_extractor.py]"]
  node_normalize["Result normalization<br/>[schema.py]"]
  node_posting["Posting data model<br/>[job_posting.py]"]
  node_openai["OpenAI models"]
end

subgraph group_delivery["Async delivery and storage"]
  node_celery["Celery worker app<br/>[celery_app.py]"]
  node_tasks["Scrape task<br/>[tasks.py]"]
  node_redis["Redis broker"]
  node_dbsession["Database sessions<br/>[connection.py]"]
  node_jobdb[("PostgreSQL postings<br/>[job_description.py]")]
  node_json["JSON output"]
end

node_user(("URL submitter"))

node_user -->|"provides URL"| node_cli
node_cli -->|"runs scraping"| node_scraper
node_cli -->|"creates browser"| node_browser
node_scraper -->|"calls"| node_pipeline
node_pipeline -->|"selects plugin"| node_plugins
node_plugins -->|"resolves"| node_default
node_plugins -->|"resolves"| node_jobkorea
node_plugins -->|"resolves"| node_jobplanet
node_pipeline -->|"uses interface"| node_pluginbase
node_pipeline -->|"renders page"| node_browser
node_pipeline -->|"checks blocks"| node_anti
node_browser -->|"applies stealth"| node_anti
node_pipeline -->|"cleans HTML"| node_htmltext
node_pipeline -->|"sends images"| node_vision
node_htmltext -->|"supplies text"| node_llm
node_vision -->|"passes OCR text"| node_llm
node_llm -->|"requests extraction"| node_openai
node_pipeline -->|"extracts fields"| node_llm
node_pipeline -->|"normalizes result"| node_normalize
node_normalize -.->|"builds posting"| node_posting
node_cli -->|"writes result"| node_json
node_user -.->|"backend submits task"| node_redis
node_celery -.->|"uses broker"| node_redis
node_tasks -->|"registers task"| node_celery
node_tasks -->|"runs scraping"| node_scraper
node_tasks -->|"updates records"| node_dbsession
node_dbsession -->|"persists postings"| node_jobdb

click node_cli "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/main.py"
click node_scraper "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/scraper.py"
click node_pipeline "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/pipeline.py"
click node_browser "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/utils/browser.py"
click node_anti "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/utils/anti_bot.py"
click node_plugins "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/registry.py"
click node_pluginbase "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/base.py"
click node_default "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/default.py"
click node_jobkorea "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/jobkorea.py"
click node_jobplanet "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/jobplanet.py"
click node_htmltext "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/extractors/llm_extractor.py"
click node_vision "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/extractors/llm_extractor.py"
click node_llm "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/extractors/llm_extractor.py"
click node_normalize "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/extractors/schema.py"
click node_posting "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/schemas/job_posting.py"
click node_celery "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/celery_app.py"
click node_tasks "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/tasks.py"
click node_dbsession "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/db/connection.py"
click node_jobdb "https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/schemas/job_description.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_scraper,node_pipeline toneBlue
class node_browser,node_anti,node_plugins,node_pluginbase,node_default,node_jobkorea,node_jobplanet toneAmber
class node_htmltext,node_vision,node_llm,node_normalize,node_posting,node_openai toneMint
class node_celery,node_tasks,node_redis,node_dbsession,node_jobdb,node_json toneRose
class node_user toneIndigo
```

## Components

### Entry and orchestration

- [CLI runner](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/main.py): `main.py`
- [Scraping entry](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/scraper.py): `scraper.py`
- [Collection pipeline](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/pipeline.py): `pipeline.py`

### Page collection

- [Browser manager](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/utils/browser.py): `utils/browser.py`
- [Anti-bot handling](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/utils/anti_bot.py): `utils/anti_bot.py`
- [Plugin registry](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/registry.py): `plugins/registry.py`
- [Plugin interface](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/base.py): `plugins/base.py`
- [Generic scraper](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/default.py): `plugins/default.py`
- [JobKorea scraper](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/jobkorea.py): `plugins/jobkorea.py`
- [JobPlanet scraper](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/plugins/jobplanet.py): `plugins/jobplanet.py`

### Content extraction

- [HTML text cleanup](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/extractors/llm_extractor.py): `extractors/llm_extractor.py`
- [Image OCR](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/extractors/llm_extractor.py): `extractors/llm_extractor.py`
- [Job field extraction](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/extractors/llm_extractor.py): `extractors/llm_extractor.py`
- [Result normalization](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/extractors/schema.py): `extractors/schema.py`
- [Posting data model](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/schemas/job_posting.py): `schemas/job_posting.py`
- **OpenAI models**

### Async delivery and storage

- [Celery worker app](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/celery_app.py): `celery_app.py`
- [Scrape task](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/tasks.py): `tasks.py`
- **Redis broker**
- [Database sessions](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/db/connection.py): `db/connection.py`
- [PostgreSQL postings](https://github.com/kmu-aws-capstone-team-4/scraping/blob/main/schemas/job_description.py): `schemas/job_description.py`
- **JSON output**

### Other components

- **URL submitter**

## Connections

- URL submitter → CLI runner: provides URL (evidence: `README.md`)
- CLI runner → Scraping entry: runs scraping (evidence: `main.py`)
- CLI runner → Browser manager: creates browser (evidence: `main.py`)
- Scraping entry → Collection pipeline: calls (evidence: `scraper.py`)
- Collection pipeline → Plugin registry: selects plugin (evidence: `pipeline.py`)
- Plugin registry → Generic scraper: resolves (evidence: `plugins/registry.py`)
- Plugin registry → JobKorea scraper: resolves (evidence: `plugins/registry.py`)
- Plugin registry → JobPlanet scraper: resolves (evidence: `plugins/registry.py`)
- Collection pipeline → Plugin interface: uses interface (evidence: `pipeline.py`)
- Collection pipeline → Browser manager: renders page (evidence: `pipeline.py`)
- Collection pipeline → Anti-bot handling: checks blocks (evidence: `pipeline.py`)
- Browser manager → Anti-bot handling: applies stealth (evidence: `utils/browser.py`)
- Collection pipeline → HTML text cleanup: cleans HTML (evidence: `README.md`)
- Collection pipeline → Image OCR: sends images (evidence: `README.md`)
- HTML text cleanup → Job field extraction: supplies text (evidence: `extractors/llm_extractor.py`)
- Image OCR → Job field extraction: passes OCR text (evidence: `extractors/llm_extractor.py`)
- Job field extraction → OpenAI models: requests extraction (evidence: `README.md`)
- Collection pipeline → Job field extraction: extracts fields (evidence: `pipeline.py`)
- Collection pipeline → Result normalization: normalizes result (evidence: `pipeline.py`)
- Result normalization → Posting data model: builds posting
- CLI runner → JSON output: writes result (evidence: `main.py`)
- URL submitter → Redis broker: backend submits task
- Celery worker app → Redis broker: uses broker (evidence: `README.md`)
- Scrape task → Celery worker app: registers task (evidence: `tasks.py`)
- Scrape task → Scraping entry: runs scraping (evidence: `tasks.py`)
- Scrape task → Database sessions: updates records (evidence: `tasks.py`)
- Database sessions → PostgreSQL postings: persists postings
