# aakashkhanal-10/sharesansar-data-pipeline architecture

> Architecture diagram of the GitHub repository aakashkhanal-10/sharesansar-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aakashkhanal-10/sharesansar-data-pipeline
- Repository: https://github.com/aakashkhanal-10/sharesansar-data-pipeline
- Video tour (about a minute): https://gitdiagram.com/aakashkhanal-10/sharesansar-data-pipeline/video

## Overview

Sharesansar Data Pipeline scrapes Nepalese market data, cleans it, and either prints analytics or persists it for REST access. The documented loader workflow orchestrates extraction and transformation before writing ORM records to SQLite; the API reads stored records and serializes responses. The standalone analyzer calculates gainers, losers, volume leaders, and other summaries. Source excerpts confirm these entry points and API queries; links for documented capabilities without sampled implementation details are omitted.

Main entry points: [`loader.py`](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/loader.py) (Pipeline Loader), [`main.py`](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/main.py) (Market API), [`extractor.py`](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/extractor.py) (Market Scraper), [`transformer.py`](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/transformer.py) (Data Cleaner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data Ingestion"]
  node_extractor["Market Scraper<br/>[extractor.py]"]
  node_transformer["Data Cleaner<br/>[transformer.py]"]
  node_loader["Pipeline Loader<br/>[loader.py]"]
end

subgraph group_analysis["Market Analysis"]
  node_analyzer["Market Analytics<br/>[analyzer.py]"]
end

subgraph group_storage["Persistence"]
  node_database["DB Session Config<br/>[database.py]"]
  node_model["Market Record Model<br/>[models.py]"]
  node_sqlite[("SQLite Store")]
end

subgraph group_api["REST API"]
  node_api["Market API<br/>[main.py]"]
  node_schema["Response Schema<br/>[schemas.py]"]
end

node_sharesansar(("Sharesansar"))
node_consumer(("API Client"))

node_loader -->|"fetches rows"| node_extractor
node_extractor -->|"requests page"| node_sharesansar
node_extractor -->|"returns rows"| node_loader
node_loader -->|"cleans rows"| node_transformer
node_transformer -->|"returns DataFrame"| node_loader
node_loader -->|"opens session"| node_database
node_loader -->|"creates records"| node_model
node_loader -->|"commits records"| node_sqlite
node_database -->|"connects"| node_sqlite
node_analyzer -->|"fetches rows"| node_extractor
node_extractor -->|"returns rows"| node_analyzer
node_analyzer -->|"cleans rows"| node_transformer
node_transformer -->|"returns DataFrame"| node_analyzer
node_consumer -->|"requests data"| node_api
node_api -->|"gets session"| node_database
node_api -->|"queries records"| node_model
node_api -->|"reads records"| node_sqlite
node_api -->|"serializes response"| node_schema

click node_extractor "https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/extractor.py"
click node_transformer "https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/transformer.py"
click node_loader "https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/loader.py"
click node_analyzer "https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/analyzer.py"
click node_database "https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/database.py"
click node_model "https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/models.py"
click node_api "https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/main.py"
click node_schema "https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/schemas.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_extractor,node_transformer,node_loader,node_consumer toneBlue
class node_analyzer toneAmber
class node_database,node_model,node_sqlite toneMint
class node_api,node_schema toneRose
class node_sharesansar toneIndigo
```

## Components

### Data Ingestion

- [Market Scraper](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/extractor.py): `extractor.py`
- [Data Cleaner](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/transformer.py): `transformer.py`
- [Pipeline Loader](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/loader.py): `loader.py`

### Market Analysis

- [Market Analytics](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/analyzer.py): `analyzer.py`

### Persistence

- [DB Session Config](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/database.py): `database.py`
- [Market Record Model](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/models.py): `models.py`
- **SQLite Store**

### REST API

- [Market API](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/main.py): `main.py`
- [Response Schema](https://github.com/aakashkhanal-10/sharesansar-data-pipeline/blob/main/schemas.py): `schemas.py`

### Other components

- **Sharesansar**
- **API Client**

## Connections

- Pipeline Loader → Market Scraper: fetches rows
- Market Scraper → Sharesansar: requests page
- Market Scraper → Pipeline Loader: returns rows
- Pipeline Loader → Data Cleaner: cleans rows
- Data Cleaner → Pipeline Loader: returns DataFrame
- Pipeline Loader → DB Session Config: opens session
- Pipeline Loader → Market Record Model: creates records
- Pipeline Loader → SQLite Store: commits records
- DB Session Config → SQLite Store: connects
- Market Analytics → Market Scraper: fetches rows
- Market Scraper → Market Analytics: returns rows
- Market Analytics → Data Cleaner: cleans rows
- Data Cleaner → Market Analytics: returns DataFrame
- API Client → Market API: requests data
- Market API → DB Session Config: gets session
- Market API → Market Record Model: queries records
- Market API → SQLite Store: reads records
- Market API → Response Schema: serializes response