# kannansingaravelu/sbi_forex_rates architecture

> Architecture diagram of the GitHub repository kannansingaravelu/sbi_forex_rates, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kannansingaravelu/sbi_forex_rates
- Repository: https://github.com/kannansingaravelu/sbi_forex_rates

## Overview

##### Project Type and Purpose
- This is a Python‐based ETL/data‐pipeline tool that automatically downloads, parses, and stores daily SBI telegraphic transfer forex rates.
- It fetches PDF rate sheets (from SBI and secondary sources), extracts the 2nd‐page rates, appends them to per‐currency CSV files, and commits both the raw PDFs and updated CSVs to GitHub.

##### Key Components
a. Scheduler (GitHub Actions)
- daily‐download.yml triggers once a day (except weekends/holidays).
b. Data Sources (External Services)
- SBI website PDF endpoint
- Secondary archives (e.g., mksco.in)
c. Ingestion Layer (Downloader Module)
- main.py and mksco.py handle HTTP(S) fetch of PDFs.
d. Processing Layer (Parser Module)
- PDF‐to‐text parsing logic (only page 2)
- Data‐cleaning and transformation into tabular rows
e. Storage Layer (Local Filesystem/Git Repo)
- /pdf_files/YYYY/MM/DD.pdf
- /csv_files/SBI_REFERENCE_RATES_<CUR>.csv
f. Version Control & Publishing
- Git branch commits & pushes (via GitHub Actions)
- GitHub-hosted CSV/PDF browsing

##### Relationships and Data Flow
- Scheduler → spins up a Python runner → invokes main.py
- main.py → downloads new PDF(s) → passes raw PDF to parser
- Parser → extracts date & rates → writes/updates CSVs
- main.py → writes PDF to /pdf_files → appends to log.txt
- On success → Git commit & push to repository
- End users access CSVs directly from GitHub or download for Excel/Sheets

##### Architectural Patterns & Principles
- Modular CLI application – clear separation between download logic and parse logic
- Idempotent daily job – only new days are downloaded/parsed
- File‐based data lake – raw (PDF) + curated (CSV) artifacts in folder hierarchy
- Infrastructure as code – GitHub Actions defines schedule and environment

##### Technologies & Frameworks
- Python 3.x (requests, PyPDF2/pdfminer or similar)
- GitHub Actions for CI/scheduling
- Filesystem folders managed in Git
- CSV for tabular storage, leveraged by Excel/Sheets

##### Essential Details to Capture
- The daily cron‐style trigger in GitHub Actions
- The two ingestion paths (SBI site & mksco archive)
- PDF folder hierarchy by year/month/day
- CSV files separated per currency
- Python modules (main.py vs. mksco.py)
- The final publishing step for user consumption

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External Data Sources"
        SBI["SBI PDF Endpoint"]:::external
        MKS["MKSCO Archive"]:::external
    end

    %% Scheduler
    Sched["GitHub Actions Scheduler"]:::external
    click Sched "https://github.com/kannansingaravelu/sbi_forex_rates/blob/main/.github/workflows/daily-download.yml"
    Props["Scheduler Properties"]:::config
    click Props "https://github.com/kannansingaravelu/sbi_forex_rates/blob/main/.github/daily-download.properties.json"
    Sched -->|"cron trigger"| Runner["Python Runner"]:::compute
    Sched -->|"reads cron config"| Props

    %% Ingestion Layer
    Runner -->|starts| Downloader["Downloader Service\n(main.py)"]:::compute
    click Downloader "https://github.com/kannansingaravelu/sbi_forex_rates/blob/main/main.py"
    Runner -->|starts| SecDownloader["Secondary Downloader\n(mksco.py)"]:::compute
    click SecDownloader "https://github.com/kannansingaravelu/sbi_forex_rates/blob/main/mksco.py"

    SBI -->|"HTTP GET PDF"| Downloader
    MKS -->|"HTTP GET PDF"| SecDownloader

    %% Processing Layer
    Downloader -->|"raw PDF"| Parser["Parser Service\n(page2 extraction)"]:::compute
    SecDownloader -->|"raw PDF"| Parser
    click Parser "https://github.com/kannansingaravelu/sbi_forex_rates/blob/main/main.py"

    %% Storage Layer
    Parser -->|"store PDF"| StoragePDF["Raw PDF Storage\n(pdf_files/)"]:::storage
    click StoragePDF "https://github.com/kannansingaravelu/sbi_forex_rates/tree/main/pdf_files/"
    Parser -->|"append CSV"| StorageCSV["Curated CSV Storage\n(csv_files/)"]:::storage
    click StorageCSV "https://github.com/kannansingaravelu/sbi_forex_rates/tree/main/csv_files/"
    Downloader -->|"write PDF"| StoragePDF
    Downloader -->|"append log"| Log["Download Log\n(log.txt)"]:::storage
    click Log "https://github.com/kannansingaravelu/sbi_forex_rates/blob/main/log.txt"

    %% Version Control & Publishing
    Parser -->|"on success"| Commit["Commit & Push"]:::process
    Commit -->|"push to"| Repo["GitHub Repo"]:::storage

    %% Repository Files
    subgraph "Repository Metadata"
        README["README.md"]:::note
        click README "https://github.com/kannansingaravelu/sbi_forex_rates/blob/main/README.md"
        LICENSE["LICENSE"]:::note
        click LICENSE "https://github.com/kannansingaravelu/sbi_forex_rates/tree/main/LICENSE"
        GITIGNORE[".gitignore"]:::note
        click GITIGNORE "https://github.com/kannansingaravelu/sbi_forex_rates/blob/main/.gitignore"
        REQ["requirements.txt"]:::note
        click REQ "https://github.com/kannansingaravelu/sbi_forex_rates/blob/main/requirements.txt"
    end
    Repo --> README
    Repo --> LICENSE
    Repo --> GITIGNORE
    Repo --> REQ

    %% Styling
    classDef external fill:#D0E6FF,stroke:#0366d6,color:#0366d6
    classDef compute fill:#DFFFE0,stroke:#28a745,color:#28a745
    classDef storage fill:#FFF5BA,stroke:#dbab09,color:#b08800
    classDef process fill:#FFE0E0,stroke:#d73a49,color:#d73a49
    classDef config fill:#E8E8E8,stroke:#6a737d,color:#6a737d
    classDef note fill:#F0F0F0,stroke:#959da5,color:#586069
```
