# mende237/smart-bible-scraping architecture

> Architecture diagram of the GitHub repository mende237/smart-bible-scraping, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mende237/smart-bible-scraping
- Repository: https://github.com/mende237/smart-bible-scraping

## Overview

This is a Bible.com scraping and dataset-preparation toolkit. Its main workflow downloads chapter text and audio, organizes and cleans the material into verse-level data, then verifies transcriptions; the separate synchronization workflow gates Google Drive transfers on verification. README documentation establishes the preprocessing and verification capabilities, while sampled source confirms the scraper dispatch, verifier structure, and synchronization calls. Some preprocessing and utility relationships are documented rather than visible in excerpts, so those edges cite the README; unsampled implementation details are not inferred.

Main entry points: [`scraping/src/scrapping.js`](https://github.com/mende237/smart-bible-scraping/blob/main/scraping/src/scrapping.js) (Local dataset), [`data-synchronisation/synchronizer.py`](https://github.com/mende237/smart-bible-scraping/blob/main/data-synchronisation/synchronizer.py) (Data synchronizer), [`data-synchronisation/synchronise_data.py`](https://github.com/mende237/smart-bible-scraping/blob/main/data-synchronisation/synchronise_data.py) (Synchronization CLI), [`data-verification/verify_data.py`](https://github.com/mende237/smart-bible-scraping/blob/main/data-verification/verify_data.py) (Verification CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_scraping["Bible scraping"]
  node_scraper["Chapter scraper<br/>[scrapping.js]"]
  node_text["Text downloader<br/>[textDownloader.js]"]
  node_audio["Audio downloader<br/>[audioDownloader.js]"]
  node_dataset[("Local dataset<br/>[scrapping.js]")]
end

subgraph group_preprocessing["Data preparation"]
  node_clean["Verse text cleaner"]
  node_versefolders["Verse folders"]
  node_utterances["Utterance file monitor"]
end

subgraph group_verification["Data verification"]
  node_validator["Transcription validator<br/>[validator.py]"]
  node_validation_helpers["Validation helpers<br/>[validator_utils.py]"]
  node_verify_cli["Verification CLI<br/>[verify_data.py]"]
end

subgraph group_sync["Cloud synchronization"]
  node_sync_cli["Synchronization CLI"]
  node_synchronizer["Data synchronizer<br/>[synchronizer.py]"]
  node_auth["Drive authentication<br/>[auth.py]"]
  node_config["Drive configuration<br/>[config.py]"]
end

subgraph group_utilities["Dataset utilities"]
  node_cli_args["Shared CLI arguments<br/>[cli_args.py]"]
  node_logging["Logging setup<br/>[logging_config.py]"]
  node_statistics["Dataset statistics<br/>[data_statistics.py]"]
  node_assignment["Work assignment"]
  node_aglc["AGLC conversion<br/>[convert_to_aglc.py]"]
end

node_user(("Operator"))
node_bible(("Bible.com"))
node_drive[("Google Drive")]

node_user -->|"configures and starts"| node_scraper
node_scraper -->|"scrapes chapters"| node_bible
node_scraper -->|"calls"| node_text
node_scraper -->|"calls"| node_audio
node_text -->|"writes text"| node_dataset
node_audio -->|"downloads audio"| node_dataset
node_user -->|"runs preprocessing"| node_clean
node_clean -->|"cleans chapter text"| node_dataset
node_user -->|"runs preprocessing"| node_versefolders
node_versefolders -->|"creates verse folders"| node_dataset
node_user -->|"starts monitor"| node_utterances
node_utterances -->|"creates transcript files"| node_dataset
node_user -->|"selects verification scope"| node_verify_cli
node_verify_cli -->|"dispatches checks"| node_validator
node_validator -->|"uses helpers"| node_validation_helpers
node_validator -->|"reads transcripts and references"| node_dataset
node_validation_helpers -->|"fetches missing reference text"| node_bible
node_user -->|"selects sync operation"| node_sync_cli
node_sync_cli -->|"dispatches transfer"| node_synchronizer
node_sync_cli -->|"requests verification gate"| node_synchronizer
node_synchronizer -->|"runs verification"| node_validator
node_synchronizer -->|"gets Drive service"| node_auth
node_auth -->|"loads credentials settings"| node_config
node_synchronizer -->|"uploads and downloads"| node_drive
node_synchronizer -->|"reads and writes files"| node_dataset
node_user -->|"requests analysis"| node_statistics
node_statistics -->|"analyzes dataset files"| node_dataset
node_assignment -->|"uses statistics"| node_statistics
node_user -->|"assigns workloads"| node_assignment
node_user -->|"runs conversion"| node_aglc
node_scraper -.->|"uses shared arguments"| node_cli_args
node_verify_cli -->|"uses shared arguments"| node_cli_args
node_sync_cli -->|"uses shared arguments"| node_cli_args
node_verify_cli -->|"configures logging"| node_logging
node_sync_cli -->|"configures logging"| node_logging

click node_scraper "https://github.com/mende237/smart-bible-scraping/blob/main/scraping/src/scrapping.js"
click node_text "https://github.com/mende237/smart-bible-scraping/blob/main/scraping/src/textDownloader.js"
click node_audio "https://github.com/mende237/smart-bible-scraping/blob/main/scraping/src/audioDownloader.js"
click node_dataset "https://github.com/mende237/smart-bible-scraping/blob/main/scraping/src/scrapping.js"
click node_clean "https://github.com/mende237/smart-bible-scraping/blob/main/data-pre-processing/pre_process_verses.py"
click node_versefolders "https://github.com/mende237/smart-bible-scraping/blob/main/data-pre-processing/generate_verses_folder.py"
click node_utterances "https://github.com/mende237/smart-bible-scraping/blob/main/data-pre-processing/generate_utterance_file.py"
click node_validator "https://github.com/mende237/smart-bible-scraping/blob/main/data-verification/validator.py"
click node_validation_helpers "https://github.com/mende237/smart-bible-scraping/blob/main/data-verification/validator_utils.py"
click node_verify_cli "https://github.com/mende237/smart-bible-scraping/blob/main/data-verification/verify_data.py"
click node_sync_cli "https://github.com/mende237/smart-bible-scraping/blob/main/data-synchronisation/synchronise_data.py"
click node_synchronizer "https://github.com/mende237/smart-bible-scraping/blob/main/data-synchronisation/synchronizer.py"
click node_auth "https://github.com/mende237/smart-bible-scraping/blob/main/data-synchronisation/auth.py"
click node_config "https://github.com/mende237/smart-bible-scraping/blob/main/data-synchronisation/config.py"
click node_cli_args "https://github.com/mende237/smart-bible-scraping/blob/main/utils/cli_args.py"
click node_logging "https://github.com/mende237/smart-bible-scraping/blob/main/utils/logging_config.py"
click node_statistics "https://github.com/mende237/smart-bible-scraping/blob/main/utils/data_statistics.py"
click node_assignment "https://github.com/mende237/smart-bible-scraping/blob/main/utils/assigning_data_to_pre_pocessors.py"
click node_aglc "https://github.com/mende237/smart-bible-scraping/blob/main/utils/convert_to_aglc.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scraper,node_text,node_audio,node_dataset toneBlue
class node_clean,node_versefolders,node_utterances,node_drive toneAmber
class node_validator,node_validation_helpers,node_verify_cli toneMint
class node_sync_cli,node_synchronizer,node_auth,node_config toneRose
class node_cli_args,node_logging,node_statistics,node_assignment,node_aglc,node_user,node_bible toneIndigo
```

## Components

### Bible scraping

- [Chapter scraper](https://github.com/mende237/smart-bible-scraping/blob/main/scraping/src/scrapping.js): `scraping/src/scrapping.js`
- [Text downloader](https://github.com/mende237/smart-bible-scraping/blob/main/scraping/src/textDownloader.js): `scraping/src/textDownloader.js`
- [Audio downloader](https://github.com/mende237/smart-bible-scraping/blob/main/scraping/src/audioDownloader.js): `scraping/src/audioDownloader.js`
- [Local dataset](https://github.com/mende237/smart-bible-scraping/blob/main/scraping/src/scrapping.js): `scraping/src/scrapping.js`

### Data preparation

- [Verse text cleaner](https://github.com/mende237/smart-bible-scraping/blob/main/data-pre-processing/pre_process_verses.py): `data-pre-processing/pre_process_verses.py`
- [Verse folders](https://github.com/mende237/smart-bible-scraping/blob/main/data-pre-processing/generate_verses_folder.py): `data-pre-processing/generate_verses_folder.py`
- [Utterance file monitor](https://github.com/mende237/smart-bible-scraping/blob/main/data-pre-processing/generate_utterance_file.py): `data-pre-processing/generate_utterance_file.py`

### Data verification

- [Transcription validator](https://github.com/mende237/smart-bible-scraping/blob/main/data-verification/validator.py): `data-verification/validator.py`
- [Validation helpers](https://github.com/mende237/smart-bible-scraping/blob/main/data-verification/validator_utils.py): `data-verification/validator_utils.py`
- [Verification CLI](https://github.com/mende237/smart-bible-scraping/blob/main/data-verification/verify_data.py): `data-verification/verify_data.py`

### Cloud synchronization

- [Synchronization CLI](https://github.com/mende237/smart-bible-scraping/blob/main/data-synchronisation/synchronise_data.py): `data-synchronisation/synchronise_data.py`
- [Data synchronizer](https://github.com/mende237/smart-bible-scraping/blob/main/data-synchronisation/synchronizer.py): `data-synchronisation/synchronizer.py`
- [Drive authentication](https://github.com/mende237/smart-bible-scraping/blob/main/data-synchronisation/auth.py): `data-synchronisation/auth.py`
- [Drive configuration](https://github.com/mende237/smart-bible-scraping/blob/main/data-synchronisation/config.py): `data-synchronisation/config.py`

### Dataset utilities

- [Shared CLI arguments](https://github.com/mende237/smart-bible-scraping/blob/main/utils/cli_args.py): `utils/cli_args.py`
- [Logging setup](https://github.com/mende237/smart-bible-scraping/blob/main/utils/logging_config.py): `utils/logging_config.py`
- [Dataset statistics](https://github.com/mende237/smart-bible-scraping/blob/main/utils/data_statistics.py): `utils/data_statistics.py`
- [Work assignment](https://github.com/mende237/smart-bible-scraping/blob/main/utils/assigning_data_to_pre_pocessors.py): `utils/assigning_data_to_pre_pocessors.py`
- [AGLC conversion](https://github.com/mende237/smart-bible-scraping/blob/main/utils/convert_to_aglc.py): `utils/convert_to_aglc.py`

### Other components

- **Operator**
- **Bible.com**
- **Google Drive**

## Connections

- Operator → Chapter scraper: configures and starts (evidence: `README.md`)
- Chapter scraper → Bible.com: scrapes chapters (evidence: `README.md`)
- Chapter scraper → Text downloader: calls (evidence: `scraping/src/scrapping.js`)
- Chapter scraper → Audio downloader: calls (evidence: `scraping/src/scrapping.js`)
- Text downloader → Local dataset: writes text (evidence: `scraping/src/textDownloader.js`)
- Audio downloader → Local dataset: downloads audio (evidence: `README.md`)
- Operator → Verse text cleaner: runs preprocessing (evidence: `README.md`)
- Verse text cleaner → Local dataset: cleans chapter text (evidence: `README.md`)
- Operator → Verse folders: runs preprocessing (evidence: `README.md`)
- Verse folders → Local dataset: creates verse folders (evidence: `README.md`)
- Operator → Utterance file monitor: starts monitor (evidence: `README.md`)
- Utterance file monitor → Local dataset: creates transcript files (evidence: `README.md`)
- Operator → Verification CLI: selects verification scope (evidence: `README.md`)
- Verification CLI → Transcription validator: dispatches checks (evidence: `data-verification/verify_data.py`)
- Transcription validator → Validation helpers: uses helpers (evidence: `data-verification/validator.py`)
- Transcription validator → Local dataset: reads transcripts and references (evidence: `README.md`)
- Validation helpers → Bible.com: fetches missing reference text (evidence: `README.md`)
- Operator → Synchronization CLI: selects sync operation (evidence: `README.md`)
- Synchronization CLI → Data synchronizer: dispatches transfer (evidence: `data-synchronisation/synchronise_data.py`)
- Synchronization CLI → Data synchronizer: requests verification gate (evidence: `data-synchronisation/synchronise_data.py`)
- Data synchronizer → Transcription validator: runs verification (evidence: `data-synchronisation/synchronizer.py`)
- Data synchronizer → Drive authentication: gets Drive service (evidence: `data-synchronisation/synchronizer.py`)
- Drive authentication → Drive configuration: loads credentials settings (evidence: `data-synchronisation/auth.py`)
- Data synchronizer → Google Drive: uploads and downloads (evidence: `data-synchronisation/synchronizer.py`)
- Data synchronizer → Local dataset: reads and writes files (evidence: `data-synchronisation/synchronizer.py`)
- Operator → Dataset statistics: requests analysis (evidence: `README.md`)
- Dataset statistics → Local dataset: analyzes dataset files (evidence: `utils/data_statistics.py`)
- Work assignment → Dataset statistics: uses statistics (evidence: `utils/assigning_data_to_pre_pocessors.py`)
- Operator → Work assignment: assigns workloads (evidence: `README.md`)
- Operator → AGLC conversion: runs conversion (evidence: `README.md`)
- Chapter scraper → Shared CLI arguments: uses shared arguments
- Verification CLI → Shared CLI arguments: uses shared arguments (evidence: `data-verification/verify_data.py`)
- Synchronization CLI → Shared CLI arguments: uses shared arguments (evidence: `data-synchronisation/synchronise_data.py`)
- Verification CLI → Logging setup: configures logging (evidence: `data-verification/verify_data.py`)
- Synchronization CLI → Logging setup: configures logging (evidence: `data-synchronisation/synchronise_data.py`)