How mende237/smart-bible-scraping works
This is a Bible.com scraping and dataset-preparation toolkit. Its main workflow downloads chapter text and audio, organizes and cleans the material into verse-level data, then verifies transcriptions; the separate synchronization workflow gates Google Drive transfers on verification. README documentation establishes the preprocessing and verification capabilities, while sampled source confirms the scraper dispatch, verifier structure, and synchronization calls. Some preprocessing and utility relationships are documented rather than visible in excerpts, so those edges cite the README; unsampled implementation details are not inferred.
Main entry points: scrapping.js Local dataset · synchronizer.py Data synchronizer · synchronise_data.py Synchronization CLI · verify_data.py Verification CLI
Components 22
Bible scraping
- Chapter scraper
scraping/src/scrapping.js - Text downloader
scraping/src/textDownloader.js - Audio downloader
scraping/src/audioDownloader.js - Local dataset
scraping/src/scrapping.js
Data preparation
- Verse text cleaner
data-pre-processing/pre_process_verses.py - Verse folders
data-pre-processing/generate_verses_folder.py - Utterance file monitor
data-pre-processing/generate_utterance_file.py
Data verification
- Transcription validator
data-verification/validator.py - Validation helpers
data-verification/validator_utils.py - Verification CLI
data-verification/verify_data.py
Cloud synchronization
- Synchronization CLI
data-synchronisation/synchronise_data.py - Data synchronizer
data-synchronisation/synchronizer.py - Drive authentication
data-synchronisation/auth.py - Drive configuration
data-synchronisation/config.py
Dataset utilities
- Shared CLI arguments
utils/cli_args.py - Logging setup
utils/logging_config.py - Dataset statistics
utils/data_statistics.py - Work assignment
utils/assigning_data_to_pre_pocessors.py - AGLC conversion
utils/convert_to_aglc.py
Other components
- Operator
- Bible.com
- Google Drive
Connections 35
- Operator → Chapter scraper: configures and starts
README.md - Chapter scraper → Bible.com: scrapes chapters
README.md - Chapter scraper → Text downloader: calls
scraping/src/scrapping.js - Chapter scraper → Audio downloader: calls
scraping/src/scrapping.js - Text downloader → Local dataset: writes text
scraping/src/textDownloader.js - Audio downloader → Local dataset: downloads audio
README.md - Operator → Verse text cleaner: runs preprocessing
README.md - Verse text cleaner → Local dataset: cleans chapter text
README.md - Operator → Verse folders: runs preprocessing
README.md - Verse folders → Local dataset: creates verse folders
README.md - Operator → Utterance file monitor: starts monitor
README.md - Utterance file monitor → Local dataset: creates transcript files
README.md - Operator → Verification CLI: selects verification scope
README.md - Verification CLI → Transcription validator: dispatches checks
data-verification/verify_data.py - Transcription validator → Validation helpers: uses helpers
data-verification/validator.py - Transcription validator → Local dataset: reads transcripts and references
README.md - Validation helpers → Bible.com: fetches missing reference text
README.md - Operator → Synchronization CLI: selects sync operation
README.md - Synchronization CLI → Data synchronizer: dispatches transfer
data-synchronisation/synchronise_data.py - Synchronization CLI → Data synchronizer: requests verification gate
data-synchronisation/synchronise_data.py - Data synchronizer → Transcription validator: runs verification
data-synchronisation/synchronizer.py - Data synchronizer → Drive authentication: gets Drive service
data-synchronisation/synchronizer.py - Drive authentication → Drive configuration: loads credentials settings
data-synchronisation/auth.py - Data synchronizer → Google Drive: uploads and downloads
data-synchronisation/synchronizer.py - Data synchronizer → Local dataset: reads and writes files
data-synchronisation/synchronizer.py - Operator → Dataset statistics: requests analysis
README.md - Dataset statistics → Local dataset: analyzes dataset files
utils/data_statistics.py - Work assignment → Dataset statistics: uses statistics
utils/assigning_data_to_pre_pocessors.py - Operator → Work assignment: assigns workloads
README.md - Operator → AGLC conversion: runs conversion
README.md - Chapter scraper → Shared CLI arguments: uses shared arguments
- Verification CLI → Shared CLI arguments: uses shared arguments
data-verification/verify_data.py - Synchronization CLI → Shared CLI arguments: uses shared arguments
data-synchronisation/synchronise_data.py - Verification CLI → Logging setup: configures logging
data-verification/verify_data.py - Synchronization CLI → Logging setup: configures logging
data-synchronisation/synchronise_data.py