# noviscient/novi-tally architecture

> Architecture diagram of the GitHub repository noviscient/novi-tally, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noviscient/novi-tally
- Repository: https://github.com/noviscient/novi-tally

## Overview

##### Project type and purpose
- This is a Python-based, open-source library (module) called “novi_tally” whose sole purpose is to reconcile position (and eventually trade) data from various brokers and sources. It provides:
- A user-facing API (the Position class)
- A plugin-style framework of dataloaders and connection wrappers
- Schemas to standardize data into pandas DataFrames
- Reconciliation logic to compare two positions by instrument

##### File structure
Top-level
- README.md, SETUP.md, config-example.toml, pyproject.toml
- examples/ – sample scripts that show how users invoke the library
Core package: novi_tally/
- api.py          – entrypoint functions, e.g. Position.from_config_file
- config.py       – TOML parsing and config validation
- connections/    – wrappers for external data sources (S3, SFTP, OpenFIGI, Formidium file)
- dataloaders/    – classes implementing the DataLoader protocol for each broker
- protocols.py    – definitions of the DataLoader protocol interfaces
- schemas.py      – definitions of standardized DataFrame schemas (positions, trades)
- errors.py       – custom exception types

##### README for architecture clues
- The library uses a layered approach:
1) Configuration →
2) Instantiate DataLoader →
3) DataLoader.extract() → pulls raw data via a Connection →
4) DataLoader.transform() → enforces a Schema →
5) .reconcile_with() → reconciliation logic

##### Elements
– Blue for core modules (api, config, protocols, schemas)
– Green for extension modules (dataloaders, connections)
– Orange for external systems (S3, APIs, file system)

##### Final advice
- Keep it readable: use consistent iconography for modules vs. external services, and annotate any complex flow with brief notes.
- The goal is to capture all essential layers—configuration, extraction, transformation, reconciliation—and show exactly where an engineer would add a new plugin.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Layer
    subgraph "User / Client Layer"
        USER["User / Client\n(samples, scripts)"]:::external
        EXAMPLE1["reconcile_positions_1.py"]:::external
    end

    %% API and Core Modules
    subgraph "Core Modules" 
        API["API Layer\nnovi_tally/api.py"]:::core
        CONFIG["Config Loader\nnovi_tally/config.py"]:::core
        PROTOCOL["DataLoader Protocol\nnovi_tally/protocols.py"]:::core
        SCHEMAS["Schema Definitions\nnovi_tally/schemas.py"]:::core
        ERRORS["Custom Exceptions\nnovi_tally/errors.py"]:::core
        RECON["Reconciliation Engine\n(Position.reconcile_with)"]:::core
    end

    %% Extension Modules
    subgraph "DataLoader Implementations" 
        ENF["Enfusion DataLoader\nnovi_tally/dataloaders/enfusion.py"]:::extension
        FORM_DL["Formidium DataLoader\nnovi_tally/dataloaders/formidium.py"]:::extension
        IB["IB DataLoader\nnovi_tally/dataloaders/ib.py"]:::extension
        RJO["RJO DataLoader\nnovi_tally/dataloaders/rjo/loaders.py"]:::extension
        RJO_HDR["RJO Headers\nnovi_tally/dataloaders/rjo/headers.py"]:::extension
    end

    subgraph "Connection Wrappers"
        FS_CONN["Local File System Connection\nnovi_tally/connections/file_systems.py"]:::extension
        FORM_CONN["Formidium API Connection\nnovi_tally/connections/formidium.py"]:::extension
        OPEN_CONN["OpenFIGI API Connection\nnovi_tally/connections/openfigi.py"]:::extension
    end

    %% External Systems
    subgraph "External Systems"
        S3["AWS S3"]:::external
        SFTP["SFTP"]:::external
        OFAPI["OpenFIGI API"]:::external
        LOCALFS["Local File System"]:::external
        BROKERAPI["Broker APIs"]:::external
    end

    %% Flows
    USER -->|"invoke Position.from_config_file()"| API
    EXAMPLE1 -->|imports library| API

    API -->|"load config"| CONFIG
    API -.->|"uses protocol"| PROTOCOL
    API -->|"instantiate DataLoader"| ENF & FORM_DL & IB & RJO

    ENF -->|"extract()"| FS_CONN
    FORM_DL -->|"extract()"| FORM_CONN
    IB -->|"extract()"| OPEN_CONN
    RJO -->|"extract()"| FS_CONN

    FS_CONN -->|"reads from"| LOCALFS
    FS_CONN -->|"reads from"| S3
    FS_CONN -->|"reads from"| SFTP
    FORM_CONN -->|"calls"| OFAPI
    OPEN_CONN -->|"calls"| OFAPI

    ENF -->|"transform(raw) applies"| SCHEMAS
    FORM_DL -->|"transform(raw) applies"| SCHEMAS
    IB -->|"transform(raw) applies"| SCHEMAS
    RJO -->|"transform(raw) applies"| SCHEMAS

    SCHEMAS -->|"pandas DataFrame"| RECON
    RECON -->|"reconcile_with(other)"| RECON

    %% Click Events
    click EXAMPLE1 "https://github.com/noviscient/novi-tally/blob/master/examples/reconcile_positions_1.py"
    click API "https://github.com/noviscient/novi-tally/blob/master/novi_tally/api.py"
    click CONFIG "https://github.com/noviscient/novi-tally/blob/master/novi_tally/config.py"
    click PROTOCOL "https://github.com/noviscient/novi-tally/blob/master/novi_tally/protocols.py"
    click ENF "https://github.com/noviscient/novi-tally/blob/master/novi_tally/dataloaders/enfusion.py"
    click FORM_DL "https://github.com/noviscient/novi-tally/blob/master/novi_tally/dataloaders/formidium.py"
    click IB "https://github.com/noviscient/novi-tally/blob/master/novi_tally/dataloaders/ib.py"
    click RJO "https://github.com/noviscient/novi-tally/blob/master/novi_tally/dataloaders/rjo/loaders.py"
    click RJO_HDR "https://github.com/noviscient/novi-tally/blob/master/novi_tally/dataloaders/rjo/headers.py"
    click FS_CONN "https://github.com/noviscient/novi-tally/blob/master/novi_tally/connections/file_systems.py"
    click FORM_CONN "https://github.com/noviscient/novi-tally/blob/master/novi_tally/connections/formidium.py"
    click OPEN_CONN "https://github.com/noviscient/novi-tally/blob/master/novi_tally/connections/openfigi.py"
    click SCHEMAS "https://github.com/noviscient/novi-tally/blob/master/novi_tally/schemas.py"
    click ERRORS "https://github.com/noviscient/novi-tally/blob/master/novi_tally/errors.py"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#0366d6,stroke-width:1px
    classDef extension fill:#E6FFEA,stroke:#28a745,stroke-width:1px
    classDef external fill:#FFF4D6,stroke:#d73a49,stroke-width:1px
```
