# dhivkrish7/lkfinance-engine architecture

> Architecture diagram of the GitHub repository dhivkrish7/lkfinance-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dhivkrish7/lkfinance-engine
- Repository: https://github.com/dhivkrish7/lkfinance-engine

## Overview

1. **System purpose and shape**
   - `lkfinance` is a Python library plus two operational entry points: a data-ingestion collector engine and a FastAPI read API.
   - The core domain is Colombo Stock Exchange market data: tickers, sectors, indices, market summaries, and trade summaries.

2. **External data source boundary**
   - The only clearly implied upstream dependency is the CSE web/data source.
   - All live market data enters the system through HTTP, then is parsed and normalized before becoming library objects or stored records.

3. **HTTP client layer**
   - `lkfinance.client.CSEClient` is the low-level outbound HTTP boundary.
   - It owns request sessions, retries, timeouts, logging, JSON parsing, and HTTP error handling, so higher layers do not talk to `requests` directly.
   - This layer is reused by collectors and API-facing services.

4. **API contract and parsing boundary**
   - `lkfinance.api` defines endpoint names, response parsing, middleware, and route wiring.
   - `lkfinance.api.services.market_data` sits between route handlers and the lower-level client/data logic.
   - `lkfinance.api.parser`, `schemas.py`, and `routes/*` separate wire-format concerns from domain objects.

5. **Domain model layer**
   - `lkfinance.market` contains the user-facing objects: `Ticker`, `Index`, `Sector`, `Summary`, and `Lookup`.
   - `lkfinance.models` contains immutable dataclass-style data structures such as ticker/sector/summary/index data.
   - This is the main boundary between raw API payloads and stable library consumers.

6. **Normalization, validation, and parsing**
   - `lkfinance.parsers` and `lkfinance.validators` isolate the messy parts of turning CSE responses into consistent values.
   - `lkfinance.parsers.cse` and `normalizers.py` suggest source-specific parsing and cleanup.
   - `lkfinance.exceptions` provides explicit failure modes for invalid or incomplete market data.

7. **Collector subsystem**
   - `lkfinance.collectors` is the ingestion side of the system.
   - It includes collectors for tickers, sectors, indices, market summary, and trade summary, plus a `summary_pipeline` and `engine`.
   - These collectors are reusable jobs that fetch data, transform it, and hand it to storage; they are not tied to the FastAPI layer.

8. **Persistence and storage boundary**
   - `lkfinance.database` owns SQLAlchemy ORM models, session management, repositories, and adapters for SQLite, PostgreSQL, and Redis cache.
   - `lkfinance.storage` appears to be a lighter storage abstraction for collector outputs, with SQLite as the default implementation.
   - The architecture keeps collectors independent from the concrete database engine by routing writes through repository/storage interfaces.

9. **Caching and optional speed layer**
   - `lkfinance.cache` and `lkfinance.database.redis_cache` indicate an optional cache layer, likely used to reduce repeated fetches or reads.
   - Cache is a supporting subsystem, not the primary source of truth; persistence is still centered on the database/storage adapters.

10. **Scheduler and batch execution**
   - `lkfinance.scheduler.CollectorScheduler` uses APScheduler to run collectors on intervals or cron schedules.
   - It includes overlap prevention and graceful shutdown, so collector jobs can run as long-lived background tasks without clashing.
   - This is the orchestration layer for continuous ingestion.

11. **End-to-end data flow**
   - Typical ingestion flow: CSE HTTP response -> `CSEClient` -> parser/validator/normalizer -> market model/dataclass -> collector -> storage/database.
   - Typical read flow for the API: FastAPI route -> service/repository -> database/storage -> serialized response.
   - The library API likely sits on top of the same domain models, exposing convenient constructors like `lk.Ticker(...)` and `lk.Summary(...)`.

12. **Runtime surfaces**
   - `run_engine.py` is a minimal standalone collector prototype that fetches market summary data and writes to SQLite.
   - `uvicorn lkfinance.api.app:app` runs the REST API.
   - `python -m lkfinance.scheduler` runs the scheduled collector process.

13. **Testing and package boundaries**
   - Tests cover the API, collectors, scheduler, sector, summary, and ticker flows, which mirrors the package boundaries well.
   - The repository is packaged as a typed Python distribution (`py.typed`, `pyproject.toml`, `setup.py`), so type boundaries are part of the design rather than an afterthought.

Main entry points: [`lkfinance/api/services/market_data.py`](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/services/market_data.py) (Market service), [`lkfinance/collectors/summary_pipeline.py`](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/collectors/summary_pipeline.py) (Summary pipeline), [`lkfinance/database/repositories.py`](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/database/repositories.py) (Repositories), [`lkfinance/parsers/cse.py`](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/parsers/cse.py) (CSE parser)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API"]
  node_api_app["API app<br/>[app.py]"]
  node_api_routes_market["Market route<br/>HTTP route<br/>[market.py]"]
  node_api_services_market_data["Market service<br/>[market_data.py]"]
  node_api_parser["API parser<br/>response parser<br/>[parser.py]"]
  node_api_schemas["Schemas<br/>schema definitions<br/>[schemas.py]"]
end

subgraph group_ingestion["Ingestion"]
  node_collectors_engine["Collector engine<br/>orchestrator<br/>[engine.py]"]
  node_collectors_pipeline["Summary pipeline"]
  node_collectors_base["Collector base<br/>[base.py]"]
end

subgraph group_domain["Domain"]
  node_parsers_cse["CSE parser<br/>payload parser<br/>[cse.py]"]
  node_parsers_normalizers["Normalizers<br/>data cleanup<br/>[normalizers.py]"]
  node_validators_cse["CSE validators<br/>validation<br/>[cse.py]"]
  node_market_domain["Market objects<br/>domain objects<br/>[__init__.py]"]
  node_models_core["Core models<br/>immutable models<br/>[__init__.py]"]
end

subgraph group_persistence["Persistence"]
  node_storage_base["Storage base<br/>storage interface<br/>[base.py]"]
  node_storage_sqlite[("SQLite storage<br/>storage impl<br/>[sqlite.py]")]
  node_db_repositories["Repositories<br/>repository layer<br/>[repositories.py]"]
  node_db_session["DB session<br/>session mgmt<br/>[session.py]"]
  node_db_models["ORM models<br/>[models.py]"]
  node_db_redis_cache["Redis cache<br/>cache adapter<br/>[redis_cache.py]"]
end

subgraph group_shared["Shared"]
  node_client_cse["CSE client<br/>HTTP client<br/>[client.py]"]
  node_cache_shared["Cache<br/>[cache.py]"]
end

subgraph group_runtime["Runtime"]
  node_scheduler["Scheduler<br/>[scheduler.py]"]
  node_run_engine["Engine runner<br/>entry point<br/>[run_engine.py]"]
end

node_api_app -->|"mounts"| node_api_routes_market
node_api_routes_market -->|"calls"| node_api_services_market_data
node_api_services_market_data -->|"parses"| node_api_parser
node_api_services_market_data -->|"fetches"| node_client_cse
node_api_services_market_data -->|"reads"| node_db_repositories
node_api_parser -->|"maps to"| node_api_schemas
node_client_cse -->|"feeds"| node_parsers_cse
node_parsers_cse -->|"cleans"| node_parsers_normalizers
node_parsers_cse -->|"checks"| node_validators_cse
node_validators_cse -->|"produces"| node_models_core
node_models_core -->|"wraps"| node_market_domain
node_collectors_engine -->|"orchestrates"| node_collectors_base
node_collectors_engine -->|"runs"| node_collectors_pipeline
node_collectors_pipeline -->|"fetches"| node_client_cse
node_collectors_pipeline -->|"emits"| node_models_core
node_collectors_pipeline -->|"writes"| node_storage_base
node_storage_sqlite -->|"implements"| node_storage_base
node_db_repositories -->|"uses"| node_db_session
node_db_repositories -->|"persists"| node_db_models
node_db_repositories -->|"accelerates"| node_db_redis_cache
node_cache_shared -->|"adapts"| node_db_redis_cache
node_scheduler -->|"schedules"| node_collectors_engine
node_run_engine -->|"invokes"| node_collectors_pipeline
node_api_services_market_data -->|"consults"| node_cache_shared

click node_api_app "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/app.py"
click node_api_routes_market "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/routes/market.py"
click node_api_services_market_data "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/services/market_data.py"
click node_api_parser "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/parser.py"
click node_api_schemas "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/schemas.py"
click node_client_cse "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/client.py"
click node_parsers_cse "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/parsers/cse.py"
click node_parsers_normalizers "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/parsers/normalizers.py"
click node_validators_cse "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/validators/cse.py"
click node_market_domain "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/market/__init__.py"
click node_models_core "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/models/__init__.py"
click node_collectors_engine "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/collectors/engine.py"
click node_collectors_pipeline "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/collectors/summary_pipeline.py"
click node_collectors_base "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/collectors/base.py"
click node_storage_base "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/storage/base.py"
click node_storage_sqlite "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/storage/sqlite.py"
click node_db_repositories "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/database/repositories.py"
click node_db_session "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/database/session.py"
click node_db_models "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/database/models.py"
click node_db_redis_cache "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/database/redis_cache.py"
click node_cache_shared "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/cache.py"
click node_scheduler "https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/scheduler.py"
click node_run_engine "https://github.com/dhivkrish7/lkfinance-engine/blob/main/run_engine.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_app,node_api_routes_market,node_api_services_market_data,node_api_parser,node_api_schemas toneBlue
class node_collectors_engine,node_collectors_pipeline,node_collectors_base toneAmber
class node_parsers_cse,node_parsers_normalizers,node_validators_cse,node_market_domain,node_models_core toneMint
class node_storage_base,node_storage_sqlite,node_db_repositories,node_db_session,node_db_models,node_db_redis_cache toneRose
class node_client_cse,node_cache_shared toneIndigo
class node_scheduler,node_run_engine toneTeal
```

## Components

### API

FastAPI read surface, contracts, and service layer.

- [API app](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/app.py): FastAPI app · Application assembly and startup for the read API. · `lkfinance/api/app.py`
- [Market route](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/routes/market.py): HTTP route · Market data endpoint wiring for the API. · `lkfinance/api/routes/market.py`
- [Market service](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/services/market_data.py): service · Bridges HTTP handlers to market-data retrieval and transformation. · `lkfinance/api/services/market_data.py`
- [API parser](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/parser.py): response parser · Wire-format parsing and endpoint response handling. · `lkfinance/api/parser.py`
- [Schemas](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/api/schemas.py): schema definitions · API request/response schemas and serialization shapes. · `lkfinance/api/schemas.py`

### Ingestion

Collectors, pipelines, and scheduler for market data ingestion.

- [Collector engine](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/collectors/engine.py): orchestrator · Coordinates collector execution and job flow. · `lkfinance/collectors/engine.py`
- [Summary pipeline](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/collectors/summary_pipeline.py): pipeline · Transforms market summary data through collector stages. · `lkfinance/collectors/summary_pipeline.py`
- [Collector base](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/collectors/base.py): collector base · Shared collector behavior used by ingestion jobs. · `lkfinance/collectors/base.py`

### Domain

Stable market objects, models, parsers, and validation.

- [CSE parser](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/parsers/cse.py): payload parser · Source-specific parsing of CSE responses. · `lkfinance/parsers/cse.py`
- [Normalizers](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/parsers/normalizers.py): data cleanup · Canonicalization and cleanup of raw market fields. · `lkfinance/parsers/normalizers.py`
- [CSE validators](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/validators/cse.py): validation · Checks completeness and validity of market payloads. · `lkfinance/validators/cse.py`
- [Market objects](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/market/__init__.py): domain objects · User-facing ticker, sector, index, summary, and lookup types. · `lkfinance/market/__init__.py`
- [Core models](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/models/__init__.py): immutable models · Dataclass-style data structures for tickers, sectors, summaries, and indices. · `lkfinance/models/__init__.py`

### Persistence

Storage, ORM models, repositories, sessions, and cache adapters.

- [Storage base](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/storage/base.py): storage interface · Abstract storage boundary for collector outputs. · `lkfinance/storage/base.py`
- [SQLite storage](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/storage/sqlite.py): storage impl · Default lightweight storage implementation. · `lkfinance/storage/sqlite.py`
- [Repositories](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/database/repositories.py): repository layer · Persistence-facing repositories for read and write access. · `lkfinance/database/repositories.py`
- [DB session](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/database/session.py): session mgmt · Database session creation and lifecycle management. · `lkfinance/database/session.py`
- [ORM models](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/database/models.py): ORM models · SQLAlchemy-backed persistence models. · `lkfinance/database/models.py`
- [Redis cache](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/database/redis_cache.py): cache adapter · Optional Redis-backed cache for repeated reads or fetches. · `lkfinance/database/redis_cache.py`

### Shared

Reusable cross-cutting client, cache, and configuration pieces.

- [CSE client](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/client.py): HTTP client · Outbound HTTP boundary to the Colombo Stock Exchange data source. · `lkfinance/client.py`
- [Cache](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/cache.py): cache facade · Shared cache layer used by higher-level components. · `lkfinance/cache.py`

### Runtime

Standalone entry points for API and batch execution.

- [Scheduler](https://github.com/dhivkrish7/lkfinance-engine/blob/main/lkfinance/scheduler.py): job scheduler · Runs collectors on intervals or cron schedules. · `lkfinance/scheduler.py`
- [Engine runner](https://github.com/dhivkrish7/lkfinance-engine/blob/main/run_engine.py): entry point · Standalone collector prototype for market summary ingestion. · `run_engine.py`

## Connections

- API app → Market route: mounts · Registers market endpoints on the application.
- Market route → Market service: calls · Route handlers delegate to the market service.
- Market service → API parser: parses · Service layer uses parser helpers for response shaping.
- Market service → CSE client: fetches · Service can reach live market data through the shared client boundary.
- Market service → Repositories: reads · Service can source persisted market data through repositories.
- API parser → Schemas: maps to · Parsed payloads are converted into API schemas.
- CSE client → CSE parser: feeds · Raw HTTP payloads are handed to CSE-specific parsing.
- CSE parser → Normalizers: cleans · Source fields are normalized into consistent values.
- CSE parser → CSE validators: checks · Parsed payloads are validated before becoming domain data.
- CSE validators → Core models: produces · Validated data becomes stable model objects.
- Core models → Market objects: wraps · Core models back the higher-level market objects.
- Collector engine → Collector base: orchestrates · Engine coordinates collector execution via shared base behavior.
- Collector engine → Summary pipeline: runs · Engine can invoke the summary pipeline as part of ingestion.
- Summary pipeline → CSE client: fetches · Pipeline uses the shared HTTP client to obtain live data.
- Summary pipeline → Core models: emits · Pipeline converts fetched data into durable model records.
- Summary pipeline → Storage base: writes · Pipeline hands results to the storage abstraction.
- SQLite storage → Storage base: implements · SQLite is the default storage implementation.
- Repositories → DB session: uses · Repositories operate through managed DB sessions.
- Repositories → ORM models: persists · Repositories translate domain state into ORM models.
- Repositories → Redis cache: accelerates · Repositories may consult Redis for cached values.
- Cache → Redis cache: adapts · Shared cache facade can route to Redis-backed storage.
- Scheduler → Collector engine: schedules · Scheduler triggers collector execution on a cadence.
- Engine runner → Summary pipeline: invokes · Standalone runner executes the market-summary pipeline.
- Market service → Cache: consults · API reads may use the shared cache layer for faster access.