How aakashkhanal-10/sharesansar-data-pipeline works
Sharesansar Data Pipeline scrapes Nepalese market data, cleans it, and either prints analytics or persists it for REST access. The documented loader workflow orchestrates extraction and transformation before writing ORM records to SQLite; the API reads stored records and serializes responses. The standalone analyzer calculates gainers, losers, volume leaders, and other summaries. Source excerpts confirm these entry points and API queries; links for documented capabilities without sampled implementation details are omitted.
Main entry points: loader.py Pipeline Loader · main.py Market API · extractor.py Market Scraper · transformer.py Data Cleaner
Components 11
Data Ingestion
- Market Scraper
extractor.py - Data Cleaner
transformer.py - Pipeline Loader
loader.py
Market Analysis
- Market Analytics
analyzer.py
Persistence
- DB Session Config
database.py - Market Record Model
models.py - SQLite Store
REST API
- Market API
main.py - Response Schema
schemas.py
Other components
- Sharesansar
- API Client
Connections 18
- Pipeline Loader → Market Scraper: fetches rows
- Market Scraper → Sharesansar: requests page
- Market Scraper → Pipeline Loader: returns rows
- Pipeline Loader → Data Cleaner: cleans rows
- Data Cleaner → Pipeline Loader: returns DataFrame
- Pipeline Loader → DB Session Config: opens session
- Pipeline Loader → Market Record Model: creates records
- Pipeline Loader → SQLite Store: commits records
- DB Session Config → SQLite Store: connects
- Market Analytics → Market Scraper: fetches rows
- Market Scraper → Market Analytics: returns rows
- Market Analytics → Data Cleaner: cleans rows
- Data Cleaner → Market Analytics: returns DataFrame
- API Client → Market API: requests data
- Market API → DB Session Config: gets session
- Market API → Market Record Model: queries records
- Market API → SQLite Store: reads records
- Market API → Response Schema: serializes response