How hum-ae-n/claudecode works
Project Type and Purpose
• This is a full-stack Python application combining:
• A modular web-scraper engine (CLI-driven and programmatic API)
• A Flask-based web interface (dashboard, product listing, export)
• A local SQLite database for storage
• Purpose: scrape e-commerce product data with rate limiting, retry/circuit breaker logic, expose a CLI and real-time web dashboard, and allow data export.
File Structure
• Top-level
• run_web_app.py: entry point for Flask UI
• example_usage.py, test_demo.py: sample usage and tests
• requirements.txt, setup.py: dependencies and packaging
• scraper/ (core package)
• cli.py: command-line interface commands
• config.py: configuration loader (env vars & JSON)
• database.py: SQLite manager (connect, save, query, export)
• models.py: Product data model
• scraper.py: scraping engine (URL extraction, product scraping)
• utils.py: rate limiter, retry logic, circuit breaker
• web_app.py: Flask app instantiation, route definitions
• templates/: Jinja2 HTML views (dashboard, product list, detail, scrape form)
• Data files and sample databases at top level for demo.
README for Additional Insights
• Core components map to modules in scraper/
• Architecture section in README confirms modular design and highlights:
• Token bucket rate limiter
• Exponential backoff retry with jitter
• Circuit breaker states
• SQLite storage with indexing
• Flask UI routes and templates
Detail Without Overthinking
• Every module as a separate component inside appropriate cluster