How dhananjay1405/tally-database-loader works
Project Type and Purpose
• Project Type: A command-line ETL utility with an optional browser-based GUI, written in Node.js/TypeScript.
• Purpose: Export data from Tally Prime via its XML server, transform it according to YAML-defined schemas, and load it into a variety of database targets (MSSQL, MySQL, PostgreSQL, BigQuery, CSV/JSON dumps). Also bundles SQL report scripts for BI tools.
Top-Level Components
A. External Systems
1. Tally Prime (XML Server at port 9000)
2. Database Server (MSSQL, MySQL, PostgreSQL, Google BigQuery, etc.)
3. BI/Reporting Tools (Excel, Power BI, Tableau, Google Data Studio)
B. Node.js Application
Code under one “Data Loader Utility” container. Within it, identify these modules (corresponding to dist/*.mjs or src/*.mts):
1. Entry Points
• CLI Launcher (run.bat → invokes dist/index.mjs)
• GUI Launcher (run-gui.bat → starts dist/server.mjs)
2. Configuration Module
• config.json reader (holds DB + Tally settings)
3. Tally Communication Module (tally.mjs)
• Builds TDL/XML requests using definition.mjs (YAML => internal JSON)
• Sends HTTP requests to Tally XML Server
4. Definition/Mapping Module (definition.mjs)
• Parses tally-export-config*.yaml
• Maps Tally XML fields ⇄ database columns
5. Database Module (database.mjs)
• Chooses driver based on config.technology
• Loads data via INSERT or native bulk-load (file method)
6. Utility Helpers (utility.mjs)
• Core orchestration: full vs. incremental sync, date handling, frequency
7. Logger Module (logger.mjs)
• Writes import-log.txt and error-log.txt
8. HTTP Server / GUI Module (server.mjs)
• Serves gui.html for config editing
• Posts updated config to the backend (without writing to disk if desired)
C. Supporting Artifacts
• SQL Scripts for DB schema (database-structure.sql, incremental variants, platform/*)
• YAML config files (tally-export-config.yaml / incremental)
• PowerShell scripts (platform/powershell/*.ps1) for multi-company automation
• SQL report queries under reports/ for MSSQL, MySQL, BigQuery
Relationships and Data Flows
1. CLI/GUI → Application entry point
2. Application → Reads config.json
3. Definition Module → Loads YAML → passes schema to Tally and Database modules
4. Tally Module → HTTP requests → Tally XML Server
5. Tally XML Server → XML response → Tally Module
6. Tally Module → Converts XML to JSON → hands off to Utility Module
7. Utility → Invokes Database Module → executes SQL or file-based bulk load → Database Server
8. Database Module → Feedback to Logger → writes import/error logs
9. Post-load: BI Tools → Query Database → generate dashboards or reports
Architectural Patterns and Design Principles
• Modular monolith: clear separation of concerns (communication, mapping, storage, logging, UI)
• Config-driven: Behavior and data schema are defined in JSON + YAML, enabling full vs. incremental sync.
• Adapter pattern: Database module selects the proper driver/queries per technology.
• Single codebase serving both CLI and simple web UI.
• Incremental vs. full synchronization strategy encapsulated in Utility module.
Key Technologies, Frameworks, and Libraries
• Node.js + TypeScript (compiled to ESM in dist/)
• HTTP (built-in or Express-style server for GUI)
• YAML parser for tally-export-config.yaml
• Database drivers: mssql, mysql2, pg, @google-cloud/bigquery or CSV/JSON writers
• Tally XML over HTTP (custom TDL-based requests)
• PowerShell for scripting on Windows
Essential Elements Checklist
[ ] Callouts for configuration files and scripts