# raj264/aws-python architecture

> Architecture diagram of the GitHub repository raj264/aws-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raj264/aws-python
- Repository: https://github.com/raj264/aws-python

## Overview

##### Project type and purpose
- This is a cloud-native, serverless “Unstructured Data Pipeline” project written in Python.
- Its goal is to ingest files of any unstructured format (CSV, JSON, XML, TXT) into S3, stage them via Lambda, and catalog/transform them with AWS Glue.

##### File structure
- Top-level directory `unstructured-data-pipeline` contains:
- `data_pipeline.py` – orchestration entrypoint
- `requirements.txt` – Python dependencies
- `lambda_function/lambda_function.py` – the code triggered on S3 events
- `modules/` – “infrastructure as code” modules:
– `s3_setup.py` – creates and configures S3 buckets
– `lambda_deploy.py` – packages and deploys the Lambda function
– `glue_setup.py` – configures Glue database, crawler, ETL job
- No traditional “frontend” or “backend” folders, but a clear separation of infra-setup code vs. runtime functions vs. orchestration script.

##### README for additional insights
- Project uses AWS services: S3, Lambda, Glue, IAM.
- Orchestration is fully automated: running `data_pipeline.py` will spin up all resources and start the pipeline.
- Emphasis on modularity and reusability in real-world data lake ingestion.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Provisioning Layer
    subgraph "Provisioning"
        direction TB
        OScript["Orchestration Script"]:::custom
        Req["requirements.txt"]:::custom
        Readme["README.md"]:::custom
        subgraph "Modules (IaC)"
            direction TB
            ModInit["__init__.py"]:::iac
            S3Mod["s3_setup.py"]:::iac
            LamMod["lambda_deploy.py"]:::iac
            GlueMod["glue_setup.py"]:::iac
        end
        CodeLambda["lambda_function.py"]:::custom
        S3RawBucket["S3 RAW Bucket"]:::aws
        S3StagingBucket["S3 STAGING Bucket"]:::aws
        S3ProcessedBucket["S3 PROCESSED Bucket"]:::aws
        LambdaFunc["AWS Lambda Function"]:::aws
        GlueCatalog["Glue Data Catalog"]:::aws
        GlueCrawler["Glue Crawler"]:::aws
        GlueETL["Glue ETL Job"]:::aws

        OScript --> Req
        OScript --> Readme
        OScript --> ModInit
        OScript --> S3Mod
        OScript --> LamMod
        OScript --> GlueMod

        S3Mod --> S3RawBucket
        S3Mod --> S3StagingBucket
        S3Mod --> S3ProcessedBucket

        LamMod --> CodeLambda
        CodeLambda --> LambdaFunc

        GlueMod --> GlueCatalog
        GlueMod --> GlueCrawler
        GlueMod --> GlueETL
    end

    %% Runtime Data Flow Layer
    subgraph "Runtime Data Flow"
        direction TB
        Producer["Data Producer / Client"]:::external
        Producer -->|"upload file"| S3RawBucket
        S3RawBucket -->|"S3:ObjectCreated triggers"| LambdaFunc
        LambdaFunc -->|"stage data"| S3StagingBucket
        LambdaFunc -->|"invoke crawler"| GlueCrawler
        GlueCrawler -->|"populate catalog"| GlueCatalog
        GlueCatalog -->|"metadata for ETL"| GlueETL
        GlueETL -->|"transform data"| S3ProcessedBucket
    end

    %% Click Events
    click OScript "https://github.com/raj264/aws-python/blob/main/unstructured-data-pipeline/data_pipeline.py"
    click S3Mod "https://github.com/raj264/aws-python/blob/main/unstructured-data-pipeline/modules/s3_setup.py"
    click LamMod "https://github.com/raj264/aws-python/blob/main/unstructured-data-pipeline/modules/lambda_deploy.py"
    click GlueMod "https://github.com/raj264/aws-python/blob/main/unstructured-data-pipeline/modules/glue_setup.py"
    click CodeLambda "https://github.com/raj264/aws-python/blob/main/unstructured-data-pipeline/lambda_function/lambda_function.py"
    click Req "https://github.com/raj264/aws-python/blob/main/unstructured-data-pipeline/requirements.txt"
    click Readme "https://github.com/raj264/aws-python/blob/main/unstructured-data-pipeline/README.md"
    click ModInit "https://github.com/raj264/aws-python/blob/main/unstructured-data-pipeline/modules/__init__.py"

    %% Styles
    classDef custom fill:#88CCFF,stroke:#333,stroke-width:1px;
    classDef iac fill:#BBEFAA,stroke:#333,stroke-width:1px;
    classDef aws fill:#FFCC66,stroke:#333,stroke-width:1px;
    classDef external fill:#CCCCCC,stroke:#333,stroke-width:1px;
```
