# dhananjay1405/tally-database-loader architecture

> Architecture diagram of the GitHub repository dhananjay1405/tally-database-loader, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dhananjay1405/tally-database-loader
- Repository: https://github.com/dhananjay1405/tally-database-loader

## Overview

##### Project Type and Purpose
- Project Type: A command-line ETL utility with an optional browser-based GUI, written in Node.js/TypeScript.
- Purpose: Export data from Tally Prime via its XML server, transform it according to YAML-defined schemas, and load it into a variety of database targets (MSSQL, MySQL, PostgreSQL, BigQuery, CSV/JSON dumps). Also bundles SQL report scripts for BI tools.

##### Top-Level Components
A. External Systems
1. Tally Prime (XML Server at port 9000)
2. Database Server (MSSQL, MySQL, PostgreSQL, Google BigQuery, etc.)
3. BI/Reporting Tools (Excel, Power BI, Tableau, Google Data Studio)

B. Node.js Application
Code under one “Data Loader Utility” container. Within it, identify these modules (corresponding to dist/*.mjs or src/*.mts):
1. Entry Points
- CLI Launcher (run.bat → invokes dist/index.mjs)
- GUI Launcher (run-gui.bat → starts dist/server.mjs)
2. Configuration Module
- config.json reader (holds DB + Tally settings)
3. Tally Communication Module (tally.mjs)
- Builds TDL/XML requests using definition.mjs (YAML => internal JSON)
- Sends HTTP requests to Tally XML Server
4. Definition/Mapping Module (definition.mjs)
- Parses tally-export-config*.yaml
- Maps Tally XML fields ⇄ database columns
5. Database Module (database.mjs)
- Chooses driver based on config.technology
- Loads data via INSERT or native bulk-load (file method)
6. Utility Helpers (utility.mjs)
- Core orchestration: full vs. incremental sync, date handling, frequency
7. Logger Module (logger.mjs)
- Writes import-log.txt and error-log.txt
8. HTTP Server / GUI Module (server.mjs)
- Serves gui.html for config editing
- Posts updated config to the backend (without writing to disk if desired)

C. Supporting Artifacts
- SQL Scripts for DB schema (database-structure.sql, incremental variants, platform/*)
- YAML config files (tally-export-config.yaml / incremental)
- PowerShell scripts (platform/powershell/*.ps1) for multi-company automation
- SQL report queries under reports/ for MSSQL, MySQL, BigQuery

##### Relationships and Data Flows
1. CLI/GUI → Application entry point
2. Application → Reads config.json
3. Definition Module → Loads YAML → passes schema to Tally and Database modules
4. Tally Module → HTTP requests → Tally XML Server
5. Tally XML Server → XML response → Tally Module
6. Tally Module → Converts XML to JSON → hands off to Utility Module
7. Utility → Invokes Database Module → executes SQL or file-based bulk load → Database Server
8. Database Module → Feedback to Logger → writes import/error logs
9. Post-load: BI Tools → Query Database → generate dashboards or reports

##### Architectural Patterns and Design Principles
- Modular monolith: clear separation of concerns (communication, mapping, storage, logging, UI)
- Config-driven: Behavior and data schema are defined in JSON + YAML, enabling full vs. incremental sync.
- Adapter pattern: Database module selects the proper driver/queries per technology.
- Single codebase serving both CLI and simple web UI.
- Incremental vs. full synchronization strategy encapsulated in Utility module.

##### Key Technologies, Frameworks, and Libraries
- Node.js + TypeScript (compiled to ESM in dist/)
- HTTP (built-in or Express-style server for GUI)
- YAML parser for tally-export-config.yaml
- Database drivers: mssql, mysql2, pg, @google-cloud/bigquery or CSV/JSON writers
- Tally XML over HTTP (custom TDL-based requests)
- PowerShell for scripting on Windows

##### Essential Elements Checklist
[ ] Callouts for configuration files and scripts

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External Systems"
        TALLY["Tally Prime\n(XML Server:9000)"]:::external
        DB["Database Server"]:::external
        BI["BI/Reporting Tools\n(Excel, Power BI, Tableau, Data Studio)"]:::external
    end

    %% Data Loader Utility Modules
    subgraph "Data Loader Utility"
        direction TB
        CLI["CLI Launcher"]:::service
        GUI["GUI Launcher"]:::service
        CONFIG["Configuration Module\n(config.json)"]:::service
        DEF["Definition/Mapping Module"]:::service
        TCOM["Tally Communication Module"]:::service
        UTIL["Core Utility Helpers"]:::service
        DBMOD["Database Module"]:::service
        LOG["Logger Module"]:::service
        HTTP["HTTP Server/GUI Module"]:::service
    end

    %% Supporting Artifacts
    subgraph "Supporting Artifacts"
        YAML["YAML Schema Definitions"]:::data
        SQLROOT["Root SQL Schema Scripts"]:::data
        SQLPLC["Platform-specific SQL Scripts"]:::data
        PS["PowerShell Automation Scripts"]:::data
        REPORTS["BI Report Queries"]:::data
    end

    %% Data Flows
    CLI -->|"invokes dist/index.mjs"| CONFIG
    CLI -->|starts| DEF
    GUI -->|"invokes dist/server.mjs"| HTTP
    CONFIG -->|"provides settings"| DEF
    DEF -->|"YAML schema"| TCOM
    TCOM -->|"HTTP/XML"| TALLY
    TALLY -->|"XML Response"| TCOM
    TCOM -->|"JSON payload"| UTIL
    UTIL -->|"invoke"| DBMOD
    DBMOD -->|"SQL INSERT / Bulk Load"| DB
    DBMOD -->|"logs status"| LOG
    LOG -->|"writes import-log.txt & error-log.txt"| LOG
    BI -->|"queries"| DB

    %% Artifacts Relations
    YAML --> DEF
    SQLROOT --> DBMOD
    SQLPLC --> DBMOD
    PS --> UTIL
    REPORTS --> BI

    %% Click Events
    click CLI "https://github.com/dhananjay1405/tally-database-loader/blob/main/run.bat"
    click GUI "https://github.com/dhananjay1405/tally-database-loader/blob/main/run-gui.bat"
    click CONFIG "https://github.com/dhananjay1405/tally-database-loader/blob/main/config.json"
    click DEF "https://github.com/dhananjay1405/tally-database-loader/blob/main/dist/definition.mjs"
    click TCOM "https://github.com/dhananjay1405/tally-database-loader/blob/main/dist/tally.mjs"
    click UTIL "https://github.com/dhananjay1405/tally-database-loader/blob/main/dist/utility.mjs"
    click DBMOD "https://github.com/dhananjay1405/tally-database-loader/blob/main/dist/database.mjs"
    click LOG "https://github.com/dhananjay1405/tally-database-loader/blob/main/dist/logger.mjs"
    click HTTP "https://github.com/dhananjay1405/tally-database-loader/blob/main/dist/server.mjs"
    click YAML "https://github.com/dhananjay1405/tally-database-loader/blob/main/tally-export-config.yaml"
    click SQLROOT "https://github.com/dhananjay1405/tally-database-loader/blob/main/database-structure.sql"
    click SQLPLC "https://github.com/dhananjay1405/tally-database-loader/blob/main/platform/mysql/database-structure.sql"
    click PS "https://github.com/dhananjay1405/tally-database-loader/tree/main/platform/powershell"
    click REPORTS "https://github.com/dhananjay1405/tally-database-loader/tree/main/reports/mssql"

    %% Styles
    classDef external fill:#cee5ff,stroke:#0366d6,color:#0366d6,stroke-width:1px;
    classDef service fill:#c6f6d5,stroke:#2f855a,color:#2f855a,stroke-width:1px;
    classDef data fill:#fff5b1,stroke:#d69e2e,color:#d69e2e,stroke-width:1px;
```
