# captn3m0/historical-mf-data architecture

> Architecture diagram of the GitHub repository captn3m0/historical-mf-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/captn3m0/historical-mf-data
- Repository: https://github.com/captn3m0/historical-mf-data

## Overview

1. This repository is primarily a historical data archive, not an application. The core asset is the `data/` directory: a long, date-partitioned collection of daily AMFI-style CSV exports for Indian mutual fund NAVs.

2. The data layer is organized by year, month, and day (`data/YYYY/MM/DD.csv`). That structure is the main storage boundary in the repo: each file represents a snapshot of the day’s fund pricing data, preserved in the source format without cleanup.

3. The repo intentionally keeps AMFI’s original quirks. Invalid ISINs and invalid NAV values are retained as-is rather than normalized away. This means the raw CSV archive is the canonical input, and downstream consumers must tolerate imperfect source data.

4. `fetch.sh` is the acquisition subsystem. It is responsible for pulling or refreshing the daily CSV inputs from the external AMFI source and populating the `data/` tree.

5. `generate.py` is the transformation/build subsystem. Its job is to take the raw CSV archive and produce the packaged dataset described in the README: a SQLite database containing historical NAVs and fund metadata.

6. The generated database is the main distribution artifact for consumers. The README points to a release asset (`funds.db.zst`) rather than the repo itself, which means the repo serves as both source archive and build input for a published binary dataset.

7. The logical database schema has three main entities: `schemes` for scheme metadata, `funds` for date-indexed NAV history, and `securities` for ISIN-to-scheme mapping and type. A derived view, `nav_by_isin`, provides the key join path for querying NAVs by ISIN.

8. The primary data flow is: external AMFI export → `fetch.sh` → raw CSV archive in `data/` → `generate.py` → SQLite DB → compressed release artifact. This is the central pipeline to diagram.

9. Query access is read-only and consumer-driven. The README’s examples show usage patterns that first resolve a security by ISIN, then look up the latest NAV on or before a target date, or retrieve NAV history for a scheme.

10. Indexes are deliberately omitted from the published database to reduce download size. Consumers are expected to add indexes locally after download, which shifts performance optimization from the build artifact to the user’s environment.

11. GitHub Actions (`.github/workflows/update.yml`) indicates the repo is maintained by automation. The workflow likely orchestrates periodic fetch/build/release updates, making the CI pipeline part of the data refresh architecture.

12. The repo boundary is therefore: upstream market data provider on one side, and downstream analysts/tools querying SQLite on the other. Everything inside the repo exists to preserve, rebuild, version, and distribute that historical dataset.

Main entry points: [`.github/workflows/update.yml`](https://github.com/captn3m0/historical-mf-data/blob/main/.github/workflows/update.yml) (update.yml), [`fetch.sh`](https://github.com/captn3m0/historical-mf-data/blob/main/fetch.sh) (fetch.sh), [`generate.py`](https://github.com/captn3m0/historical-mf-data/blob/main/generate.py) (generate.py), [`data/2006/04/01.csv`](https://github.com/captn3m0/historical-mf-data/blob/main/data/2006/04/01.csv) (Daily CSV)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_upstream["Upstream"]
  node_amfi(("AMFI export<br/>external source"))
end

subgraph group_archive["Archive"]
  node_data_tree[("Raw archive<br/>csv archive")]
  node_daily_csv["Daily CSV<br/>snapshot file<br/>[01.csv]"]
  node_quirks["Source quirks<br/>raw data policy"]
end

subgraph group_build["Build"]
  node_fetch_sh["fetch.sh<br/>acquisition script<br/>[fetch.sh]"]
  node_generate_py["generate.py<br/>build script<br/>[generate.py]"]
end

subgraph group_distribution["Distribution"]
  node_sqlite_db[("funds.db<br/>sqlite database")]
  node_schemes["schemes<br/>table"]
  node_funds["funds<br/>table"]
  node_securities["securities<br/>table"]
  node_nav_view{{"nav_by_isin<br/>view"}}
  node_no_indexes["No indexes<br/>distribution choice"]
  node_release_zst[("Release asset<br/>compressed artifact")]
end

subgraph group_automation["Automation"]
  node_workflow["update.yml<br/>github actions workflow<br/>[update.yml]"]
end

subgraph group_consumers["Consumers"]
  node_analysts(("Readers<br/>read-only consumers"))
  node_nav_lookup["NAV lookup<br/>query pattern"]
  node_history_query["History query<br/>query pattern"]
end

node_amfi -->|"daily exports"| node_fetch_sh
node_fetch_sh -->|"populate"| node_data_tree
node_data_tree -->|"contains"| node_daily_csv
node_data_tree -->|"preserves"| node_quirks
node_data_tree -->|"input"| node_generate_py
node_generate_py -->|"builds"| node_sqlite_db
node_sqlite_db -->|"stores"| node_schemes
node_sqlite_db -->|"stores"| node_funds
node_sqlite_db -->|"stores"| node_securities
node_schemes -->|"joins via"| node_nav_view
node_funds -->|"joins via"| node_nav_view
node_securities -->|"joins via"| node_nav_view
node_sqlite_db -->|"omits"| node_no_indexes
node_sqlite_db -->|"compresses to"| node_release_zst
node_workflow -->|"orchestrates"| node_fetch_sh
node_workflow -->|"orchestrates"| node_generate_py
node_workflow -->|"publishes"| node_release_zst
node_analysts -->|"use"| node_nav_lookup
node_analysts -->|"use"| node_history_query
node_nav_lookup -->|"reads"| node_nav_view
node_history_query -->|"reads"| node_funds
node_analysts -->|"download"| node_sqlite_db

click node_fetch_sh "https://github.com/captn3m0/historical-mf-data/blob/main/fetch.sh"
click node_daily_csv "https://github.com/captn3m0/historical-mf-data/blob/main/data/2006/04/01.csv"
click node_generate_py "https://github.com/captn3m0/historical-mf-data/blob/main/generate.py"
click node_workflow "https://github.com/captn3m0/historical-mf-data/blob/main/.github/workflows/update.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_amfi toneBlue
class node_data_tree,node_daily_csv,node_quirks toneAmber
class node_fetch_sh,node_generate_py toneMint
class node_sqlite_db,node_schemes,node_funds,node_securities,node_nav_view,node_no_indexes,node_release_zst toneRose
class node_workflow toneIndigo
class node_analysts,node_nav_lookup,node_history_query toneTeal
```

## Components

### Upstream

External AMFI source feeding the archive.

- **AMFI export**: external source · Upstream daily mutual-fund NAV CSV feed.

### Archive

Raw date-partitioned CSV storage.

- **Raw archive**: csv archive · Canonical preserved source data organized by year, month, and day.
- [Daily CSV](https://github.com/captn3m0/historical-mf-data/blob/main/data/2006/04/01.csv): snapshot file · One day of AMFI-style fund pricing data, kept in source form. · `data/2006/04/01.csv`
- **Source quirks**: raw data policy · Retains invalid ISINs and NAV values without normalization.

### Build

Scripts that refresh and package the dataset.

- [fetch.sh](https://github.com/captn3m0/historical-mf-data/blob/main/fetch.sh): acquisition script · Fetches or refreshes daily CSV inputs from AMFI. · `fetch.sh`
- [generate.py](https://github.com/captn3m0/historical-mf-data/blob/main/generate.py): build script · Transforms the raw archive into the packaged SQLite dataset. · `generate.py`

### Distribution

Published SQLite artifact and its schema.

- **funds.db**: sqlite database · Packaged historical NAV and metadata database for consumers.
- **schemes**: table · Scheme metadata and descriptive fund attributes.
- **funds**: table · Date-indexed NAV history for schemes.
- **securities**: table · ISIN-to-scheme mapping and security type.
- **nav_by_isin**: view · Join path for looking up NAVs by ISIN.
- **No indexes**: distribution choice · Indexes are omitted to keep the release artifact small.
- **Release asset**: compressed artifact · Published compressed database asset for download.

### Automation

Scheduled maintenance and release workflow.

- [update.yml](https://github.com/captn3m0/historical-mf-data/blob/main/.github/workflows/update.yml): github actions workflow · Automation that likely schedules refresh, build, and release. · `.github/workflows/update.yml`

### Consumers

Read-only query patterns over the packaged database.

- **Readers**: read-only consumers · Query the SQLite database by ISIN or scheme and add local indexes if needed.
- **NAV lookup**: query pattern · Resolve security by ISIN, then find NAV on or before a target date.
- **History query**: query pattern · Retrieve NAV history for a scheme over time.

## Connections

- AMFI export → fetch.sh: daily exports · Upstream feed is acquired by the fetch script.
- fetch.sh → Raw archive: populate · Acquired CSVs are written into the raw archive tree.
- Raw archive → Daily CSV: contains · Archive is partitioned into per-day snapshot files.
- Raw archive → Source quirks: preserves · Source irregularities are retained as canonical input.
- Raw archive → generate.py: input · The build script reads the raw archive as its source.
- generate.py → funds.db: builds · Transformation produces the packaged SQLite database.
- funds.db → schemes: stores · Database contains scheme metadata.
- funds.db → funds: stores · Database contains historical NAV rows.
- funds.db → securities: stores · Database contains ISIN and security mapping data.
- schemes → nav_by_isin: joins via · Scheme metadata participates in query paths.
- funds → nav_by_isin: joins via · NAV history is exposed through the derived view.
- securities → nav_by_isin: joins via · ISIN mapping anchors the primary lookup view.
- funds.db → No indexes: omits · Published database intentionally excludes indexes.
- funds.db → Release asset: compresses to · Database is distributed as a compressed release asset.
- update.yml → fetch.sh: orchestrates · Automation coordinates data refresh.
- update.yml → generate.py: orchestrates · Automation coordinates dataset generation.
- update.yml → Release asset: publishes · Automation likely updates the release artifact.
- Readers → NAV lookup: use · Consumers query by ISIN for current or historical NAVs.
- Readers → History query: use · Consumers query NAV history by scheme.
- NAV lookup → nav_by_isin: reads · Lookup workflow depends on the ISIN join view.
- History query → funds: reads · History workflow reads the NAV table directly.
- Readers → funds.db: download · Consumers retrieve the packaged database artifact.