# aosm26/case-study architecture

> Architecture diagram of the GitHub repository aosm26/case-study, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aosm26/case-study
- Repository: https://github.com/aosm26/case-study

## Overview

##### Project Type and Purpose
- This is a backend-only data-collection pipeline (“The Dyrt Kamp Alanı Veri Toplayıcı”) written in Python.
- It periodically fetches campground data from The Dyrt API, resolves addresses via OpenStreetMap Nominatim, validates and persists records in PostgreSQL, and writes detailed logs.

##### File Structure
- src/api_client: modules for fetching, threading, geocoding, saving.
- src/models: Pydantic/SQLAlchemy definitions.
- src/scheduler: a task scheduler wrapper (uses `schedule` library).
- src/utils: shared logger setup and custom exceptions.
- main.py: CLI entry point that dispatches one-time run or scheduled runs.
- Dockerfile & docker-compose.yml: containerization for the app and Postgres.
- logs/: rotating log files per module.

##### README for Insights
- Lists core features (multi-threaded fetch, rate-limit handling, batch DB writes, customizable via CLI flags).
- Describes dependencies (Requests, Threading, SQLAlchemy, Pydantic, Schedule).
- Mentions Docker & Compose, environment variables, and schedule intervals.

##### Main Components
a. **CLI / Orchestrator**
- main.py (parses flags, triggers one-off or scheduled pipeline).
b. **Scheduler**
- task_scheduler.py (wraps `schedule`, triggers at configured intervals).
c. **API Clients**
- fetch_campgrounds(_threaded).py (downloads pages in parallel).
- geocode_addresses.py (calls Nominatim).
- save_to_db.py (invokes SQLAlchemy session to write).
d. **Data Models & Validation**
- Pydantic models + SQLAlchemy ORM (campground_db.py).
e. **Logging Subsystem**
- logger.py + rotating FileHandlers (one file per module).
f. **Database**
- PostgreSQL container (psycopg2 driver).
g. **External Services**
- The Dyrt REST API
- OpenStreetMap Nominatim API

##### Relationships & Data Flow
- **CLI** → decides “run-once” or “schedule”.
- **Scheduler** → at each tick invokes the download pipeline.
- **Download Pipeline**:
1. fetch_campgrounds_threaded → retrieves raw JSON pages from The Dyrt API
2. geocode_addresses → transforms coordinates into address data
3. save_to_db → batches and writes via SQLAlchemy to PostgreSQL
4. Errors/exceptions bubble up to exceptions.py handlers, all events logged to module-specific log files in logs/.
- All inter-module calls occur within the same Python process.

##### Architectural Patterns & Design Principles
- **Layered Architecture**: Separation between API-Clients, Models, Scheduler, Utils.
- **Modular Pipeline**: Each stage (fetch, geocode, save) is its own module.
- **Thread-pool Concurrency**: fetch_campgrounds_threaded uses `concurrent.futures.ThreadPoolExecutor`.
- **Dependency Injection**: Configurable parameters (page size, max_workers, batch size, schedule interval) passed via CLI and environment.
- **Containerized Deployment**: Docker Compose defines two services: app and postgres.

##### Elements
- **Component Labels**: e.g., “fetch_campgrounds_threaded.py”, “task_scheduler.py”.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Title
    title["The Dyrt Kamp Alanı Veri Toplayıcı Architecture"]

    %% External Services
    API1["The Dyrt API"]:::external
    API2["OSM Nominatim"]:::external

    %% Orchestrator & Scheduler Layer
    subgraph "Orchestrator & Scheduler"
        direction TB
        CLI["main.py"]:::orchestrator
        Scheduler["task_scheduler.py"]:::orchestrator
    end

    %% API Client Layer
    subgraph "API Client Modules"
        direction TB
        FetchThread["fetch_campgrounds_threaded.py"]:::api
        FetchSingle["fetch_campgrounds.py"]:::api
        Geocode["geocode_addresses.py"]:::api
        SaveDB["save_to_db.py"]:::api
    end

    %% Models Layer
    subgraph "Data Models & Persistence"
        direction TB
        Pydantic["campground.py"]:::model
        ORM["campground_db.py"]:::model
    end

    %% Utilities Layer
    subgraph "Utilities & Logging"
        direction TB
        Logger["logger.py"]:::util
        Exc["exceptions.py"]:::util
        Logs["logs/"]:::logs
    end

    %% Database
    DB[(PostgreSQL)]:::db

    %% Docker Compose
    subgraph "Docker Compose" 
        direction TB
        DockerApp["App Container"]:::docker
        DockerDB["Postgres Container"]:::docker
    end

    %% Connections
    CLI -->|"run-once or scheduled"| Scheduler
    CLI -->|"one-off pipeline"| FetchThread

    Scheduler -->|"trigger pipeline"| FetchThread

    FetchThread -->|"HTTP GET JSON"| API1
    FetchThread -->|"raw JSON"| Geocode

    Geocode -->|"HTTP GET /reverse"| API2
    Geocode -->|"enriched data"| SaveDB

    SaveDB -->|"Batch INSERT"| DB

    %% Logging and exceptions flow
    CLI -->|logs events| Logs
    Scheduler -->|logs events| Logs
    FetchThread -->|logs events| Logs
    Geocode -->|logs events| Logs
    SaveDB -->|logs events| Logs
    Exc -->|handles errors| Logs

    %% Container orchestration links
    DockerApp -->|runs| CLI
    DockerApp -->|uses| DB
    DockerDB -->|provides| DB

    %% Click Events
    click CLI "https://github.com/aosm26/case-study/blob/main/main.py"
    click Scheduler "https://github.com/aosm26/case-study/blob/main/src/scheduler/task_scheduler.py"
    click FetchSingle "https://github.com/aosm26/case-study/blob/main/src/api_client/fetch_campgrounds.py"
    click FetchThread "https://github.com/aosm26/case-study/blob/main/src/api_client/fetch_campgrounds_threaded.py"
    click Geocode "https://github.com/aosm26/case-study/blob/main/src/api_client/geocode_addresses.py"
    click SaveDB "https://github.com/aosm26/case-study/blob/main/src/api_client/save_to_db.py"
    click Pydantic "https://github.com/aosm26/case-study/blob/main/src/models/campground.py"
    click ORM "https://github.com/aosm26/case-study/blob/main/src/models/campground_db.py"
    click Logger "https://github.com/aosm26/case-study/blob/main/src/utils/logger.py"
    click Exc "https://github.com/aosm26/case-study/blob/main/src/utils/exceptions.py"
    click DockerApp "https://github.com/aosm26/case-study/tree/main/Dockerfile"
    click DockerDB "https://github.com/aosm26/case-study/blob/main/docker-compose.yml"
    click Logs "https://github.com/aosm26/case-study/tree/main/logs/"

    %% Styles
    classDef orchestrator fill:#D0E1F9,stroke:#2E6DA4,color:#2E6DA4;
    classDef api fill:#DFF2BF,stroke:#4F8A10,color:#4F8A10;
    classDef model fill:#FDEBD0,stroke:#A04000,color:#A04000;
    classDef util fill:#E0E0E0,stroke:#555555,color:#555555;
    classDef db fill:#FFD1D1,stroke:#C12B2B,color:#C12B2B,stroke-dasharray: 5 5;
    classDef external fill:#E8EAF6,stroke:#303F9F,color:#303F9F,stroke-dasharray: 2 2;
    classDef docker fill:#D6EAF8,stroke:#154360,color:#154360;
    classDef logs fill:#F2F3F4,stroke:#7F8C8D,color:#7F8C8D,stroke-dasharray: 2 2;
```
