How srajasimman/aws-cdk-sftp-to-s3-sync works
Project Type & Purpose
• This is an AWS infrastructure-as-code project built with AWS CDK (TypeScript) plus a small serverless application (Lambda) and EC2-based SFTP server.
• Its goal is to automate file ingestion from an SFTP endpoint (hosted on an EC2 instance) into an S3 bucket, with idempotency, error handling, scheduling, and secret management.
File Structure Analysis
• Top‐level CDK app:
• bin/app.ts – entry point for CDK.
• lib/sftp-ingest-stack.ts – defines the main CDK stack.
• lib/constructs/vpc.ts & s3-bucket.ts – reusable constructs for VPC and S3.
• cdk.json, tsconfig.json, package.json – CDK config & build.
• Lambda function code lives under src/:
• src/index.ts – handler.
• src/lib/{idempotency,logger,s3Writer,secrets}.ts – core logic (DynamoDB idempotency, S3 write, secret retrieval).
• Deployment toolchain files: .envrc, devbox.json, bun.lock, etc.
README Insights
• Lists all AWS components: VPC (public/private subnets), EC2 (SFTP), S3 bucket, Lambda, EventBridge (cron), optional DynamoDB, Secrets Manager, SQS DLQ.
• Highlights security best practices (private Lambda, IAM least privilege, encryption).
• Mentions outputs (public IP, ARNs) and post-deployment steps (upload credentials).
Diagramming the Architecture
a. Main Components:
• VPC: public & private subnets
• EC2 Instance (SFTP server) in public subnet
• EventBridge Rule (15-minute cron)
• Lambda Function in private subnet
• DynamoDB Table (idempotency) – optional
• S3 Bucket (file sink)
• Secrets Manager (SFTP credentials)
• SQS Dead‐Letter Queue
• IAM Roles & VPC Endpoints (for S3, Secrets Manager, DynamoDB)
b. Relationships & Interactions:
• EventBridge invokes Lambda on schedule.
• Lambda retrieves credentials from Secrets Manager.
• Lambda SSH-connects to EC2 SFTP server in public subnet.
• Lambda downloads new files, checks/updates DynamoDB for idempotency.
• Lambda writes processed files to S3.
• On Lambda failure, messages go to SQS DLQ.
• EC2 stores logs to CloudWatch via IAM/SSM.
c. Architectural Patterns & Principles:
• Infrastructure as Code (CDK) – reusable constructs, single Stack.
• Serverless pattern for ingestion logic (Lambda + EventBridge).
• Idempotent handling via DynamoDB.
• Secure secret storage (Secrets Manager).
• Private subnets for compute, public for remote access.
• Least-privilege IAM roles.
d. Key Technologies & Libraries:
• AWS CDK v2 (TypeScript)
• AWS Lambda, EC2, VPC, S3, DynamoDB, Secrets Manager, EventBridge, SQS.
• TypeScript helper modules (ssh via Node, S3 SDK, DynamoDB SDK).
Capturing Essential Elements
• Don’t omit IAM roles & VPC endpoints – they enforce least privilege and private networking.
• Stack’s outputs (public IP, ARNs) are annotated as CDK “Outputs.”