How kannansingaravelu/sbi_forex_rates works
Project Type and Purpose
• This is a Python‐based ETL/data‐pipeline tool that automatically downloads, parses, and stores daily SBI telegraphic transfer forex rates.
• It fetches PDF rate sheets (from SBI and secondary sources), extracts the 2nd‐page rates, appends them to per‐currency CSV files, and commits both the raw PDFs and updated CSVs to GitHub.
Key Components
a. Scheduler (GitHub Actions)
• daily‐download.yml triggers once a day (except weekends/holidays).
b. Data Sources (External Services)
• SBI website PDF endpoint
• Secondary archives (e.g., mksco.in)
c. Ingestion Layer (Downloader Module)
• main.py and mksco.py handle HTTP(S) fetch of PDFs.
d. Processing Layer (Parser Module)
• PDF‐to‐text parsing logic (only page 2)
• Data‐cleaning and transformation into tabular rows
e. Storage Layer (Local Filesystem/Git Repo)
• /pdf_files/YYYY/MM/DD.pdf
• /csv_files/SBI_REFERENCE_RATES_<CUR>.csv
f. Version Control & Publishing
• Git branch commits & pushes (via GitHub Actions)
• GitHub-hosted CSV/PDF browsing
Relationships and Data Flow
• Scheduler → spins up a Python runner → invokes main.py
• main.py → downloads new PDF(s) → passes raw PDF to parser
• Parser → extracts date & rates → writes/updates CSVs
• main.py → writes PDF to /pdf_files → appends to log.txt
• On success → Git commit & push to repository
• End users access CSVs directly from GitHub or download for Excel/Sheets
Architectural Patterns & Principles
• Modular CLI application – clear separation between download logic and parse logic
• Idempotent daily job – only new days are downloaded/parsed
• File‐based data lake – raw (PDF) + curated (CSV) artifacts in folder hierarchy
• Infrastructure as code – GitHub Actions defines schedule and environment
Technologies & Frameworks
• Python 3.x (requests, PyPDF2/pdfminer or similar)
• GitHub Actions for CI/scheduling
• Filesystem folders managed in Git
• CSV for tabular storage, leveraged by Excel/Sheets
Essential Details to Capture
• The daily cron‐style trigger in GitHub Actions
• The two ingestion paths (SBI site & mksco archive)
• PDF folder hierarchy by year/month/day
• CSV files separated per currency
• Python modules (main.py vs. mksco.py)
• The final publishing step for user consumption