# abinavrameshs/sftp-transfer-automation architecture

> Architecture diagram of the GitHub repository abinavrameshs/sftp-transfer-automation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abinavrameshs/sftp-transfer-automation
- Repository: https://github.com/abinavrameshs/sftp-transfer-automation

## Overview

##### Project type and purpose
- This is a lightweight automation/ETL tool whose sole purpose is to extract data from an Impala database, transform it into CSV format, and load (transfer) it onto a target SFTP server. It wraps the Python script in a Windows batch file and triggers it on a schedule via Windows Task Scheduler.

##### File structure
- Top-level files:
- automation_script.py – the main Python ETL/transfer logic
- schedule_python_script.bat – a wrapper to invoke the Python script in a Windows shell
- README.md – project description and instructions
- .gitignore – exclusions for version control
- No dedicated folders; everything lives at the root.

##### README for additional insights
- Key steps in the workflow:
a. To Impala, run a query, fetch rows
b. Convert the resultset to CSV
c. To an SFTP server (over SSH), navigate remote directories, and upload the CSV
d. The .bat file simply calls the Python script with the correct interpreter
e. Windows Task Scheduler triggers the .bat at the configured time

##### Elements
- Scheduler & .bat: light gray rectangles (or rounded)
- Python script: blue rectangle

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Components
    Scheduler["Windows Task Scheduler"]:::scheduler
    subgraph "Compute Node"
        Batch["schedule_python_script.bat"]:::batch
        Python["automation_script.py"]:::python
    end
    Impala["Impala Database"]:::db
    FS["Local Filesystem / CSV Storage"]:::fs
    SFTP["SFTP Server"]:::sftp
    subgraph "Project Files"
        README["README.md"]:::doc
        GIT[".gitignore"]:::doc
    end

    %% Data Flow
    Scheduler -->|"invoke"| Batch
    Batch -->|"launch"| Python
    Python -->|"query via JDBC/ODBC"| Impala
    Python -->|"write CSV"| FS
    Python -->|"upload via SFTP"| SFTP

    %% Click Events
    click Batch "https://github.com/abinavrameshs/sftp-transfer-automation/blob/master/./schedule_python_script.bat"
    click Python "https://github.com/abinavrameshs/sftp-transfer-automation/blob/master/./automation_script.py"
    click README "https://github.com/abinavrameshs/sftp-transfer-automation/blob/master/./README.md"
    click GIT "https://github.com/abinavrameshs/sftp-transfer-automation/blob/master/./.gitignore"

    %% Styles
    classDef scheduler fill:#f0f0f0,stroke:#333,stroke-width:1px
    classDef batch fill:#e8e8e8,stroke:#333,stroke-width:1px
    classDef python fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef db fill:#d4edda,stroke:#333,stroke-width:1px
    classDef sftp fill:#ffe5cc,stroke:#333,stroke-width:1px
    classDef fs fill:#f8f9fa,stroke:#333,stroke-width:1px
    classDef doc fill:#fdfd96,stroke:#333,stroke-width:1px
```
