How abinavrameshs/sftp-transfer-automation works
Project type and purpose
• This is a lightweight automation/ETL tool whose sole purpose is to extract data from an Impala database, transform it into CSV format, and load (transfer) it onto a target SFTP server. It wraps the Python script in a Windows batch file and triggers it on a schedule via Windows Task Scheduler.
File structure
• Top-level files:
• automation_script.py – the main Python ETL/transfer logic
• schedule_python_script.bat – a wrapper to invoke the Python script in a Windows shell
• README.md – project description and instructions
• .gitignore – exclusions for version control
• No dedicated folders; everything lives at the root.
README for additional insights
• Key steps in the workflow:
a. To Impala, run a query, fetch rows
b. Convert the resultset to CSV
c. To an SFTP server (over SSH), navigate remote directories, and upload the CSV
d. The .bat file simply calls the Python script with the correct interpreter
e. Windows Task Scheduler triggers the .bat at the configured time
Elements
• Scheduler & .bat: light gray rectangles (or rounded)
• Python script: blue rectangle