captn3m0/historical-mf-data

1. This repository is primarily a historical data archive, not an application. The core asset is the data/ directory: a long, date-partitioned collection of daily AMFI-style CSV exports for Indian mutual fund NAVs.

2. The data layer is organized by year, month, and day (data/YYYY/MM/DD.csv). That structure is the main storage boundary in the repo: each file represents a snapshot of the day’s fund pricing data, preserved in the source format without cleanup.

3. The repo intentionally keeps AMFI’s original quirks. Invalid ISINs and invalid NAV values are retained as-is rather than normalized away. This means the raw CSV archive is the canonical input, and downstream consumers must tolerate imperfect source data.

4. fetch.sh is the acquisition subsystem. It is responsible for pulling or refreshing the daily CSV inputs from the external AMFI source and populating the data/ tree.

5. generate.py is the transformation/build subsystem. Its job is to take the raw CSV archive and produce the packaged dataset described in the README: a SQLite database containing historical NAVs and fund metadata.

6. The generated database is the main distribution artifact for consumers. The README points to a release asset (funds.db.zst) rather than the repo itself, which means the repo serves as both source archive and build input for a published binary dataset.

7. The logical database schema has three main entities: schemes for scheme metadata, funds for date-indexed NAV history, and securities for ISIN-to-scheme mapping and type. A derived view, nav_by_isin, provides the key join path for querying NAVs by ISIN.

8. The primary data flow is: external AMFI export → fetch.sh → raw CSV archive in data/ → generate.py → SQLite DB → compressed release artifact. This is the central pipeline to diagram.

9. Query access is read-only and consumer-driven. The README’s examples show usage patterns that first resolve a security by ISIN, then look up the latest NAV on or before a target date, or retrieve NAV history for a scheme.

10. Indexes are deliberately omitted from the published database to reduce download size. Consumers are expected to add indexes locally after download, which shifts performance optimization from the build artifact to the user’s environment.

11. GitHub Actions (.github/workflows/update.yml) indicates the repo is maintained by automation. The workflow likely orchestrates periodic fetch/build/release updates, making the CI pipeline part of the data refresh architecture.

12. The repo boundary is therefore: upstream market data provider on one side, and downstream analysts/tools querying SQLite on the other. Everything inside the repo exists to preserve, rebuild, version, and distribute that historical dataset.