How aosm26/case-study works
Project Type and Purpose
• This is a backend-only data-collection pipeline (“The Dyrt Kamp Alanı Veri Toplayıcı”) written in Python.
• It periodically fetches campground data from The Dyrt API, resolves addresses via OpenStreetMap Nominatim, validates and persists records in PostgreSQL, and writes detailed logs.
File Structure
• src/api_client: modules for fetching, threading, geocoding, saving.
• src/models: Pydantic/SQLAlchemy definitions.
• src/scheduler: a task scheduler wrapper (uses schedule library).
• src/utils: shared logger setup and custom exceptions.
• main.py: CLI entry point that dispatches one-time run or scheduled runs.
• Dockerfile & docker-compose.yml: containerization for the app and Postgres.
• logs/: rotating log files per module.
README for Insights
• Lists core features (multi-threaded fetch, rate-limit handling, batch DB writes, customizable via CLI flags).
• Describes dependencies (Requests, Threading, SQLAlchemy, Pydantic, Schedule).
• Mentions Docker & Compose, environment variables, and schedule intervals.
Main Components
a. CLI / Orchestrator
• main.py (parses flags, triggers one-off or scheduled pipeline).
b. Scheduler
• task_scheduler.py (wraps schedule, triggers at configured intervals).
c. API Clients
• fetch_campgrounds(_threaded).py (downloads pages in parallel).
• geocode_addresses.py (calls Nominatim).
• save_to_db.py (invokes SQLAlchemy session to write).
d. Data Models & Validation
• Pydantic models + SQLAlchemy ORM (campground_db.py).
e. Logging Subsystem
• logger.py + rotating FileHandlers (one file per module).
f. Database
• PostgreSQL container (psycopg2 driver).
g. External Services
• The Dyrt REST API
• OpenStreetMap Nominatim API
Relationships & Data Flow
• CLI → decides “run-once” or “schedule”.
• Scheduler → at each tick invokes the download pipeline.
• Download Pipeline:
1. fetch_campgrounds_threaded → retrieves raw JSON pages from The Dyrt API
2. geocode_addresses → transforms coordinates into address data
3. save_to_db → batches and writes via SQLAlchemy to PostgreSQL
4. Errors/exceptions bubble up to exceptions.py handlers, all events logged to module-specific log files in logs/.
• All inter-module calls occur within the same Python process.
Architectural Patterns & Design Principles
• Layered Architecture: Separation between API-Clients, Models, Scheduler, Utils.
• Modular Pipeline: Each stage (fetch, geocode, save) is its own module.
• Thread-pool Concurrency: fetch_campgrounds_threaded uses concurrent.futures.ThreadPoolExecutor.
• Dependency Injection: Configurable parameters (page size, max_workers, batch size, schedule interval) passed via CLI and environment.
• Containerized Deployment: Docker Compose defines two services: app and postgres.
Elements
• Component Labels: e.g., “fetch_campgrounds_threaded.py”, “task_scheduler.py”.