# lucjankonopka/portfolio architecture

> Architecture diagram of the GitHub repository lucjankonopka/portfolio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lucjankonopka/portfolio
- Repository: https://github.com/lucjankonopka/portfolio

## Overview

##### Project type and purpose
- This repository is a Data Engineer Portfolio. It is essentially a static documentation site (a single README.md) that links out to several hands-on data engineering projects. Its purpose is to showcase Lucjan’s skills across web scraping, ETL pipelines, big-data processing, streaming, analytics, and visualization.

##### File structure
- Top-level: only `README.md`.
- No application code or traditional directories—this is purely a content/marketing site in markdown form.
- All “real work” lives in external GitHub repos (the linked subprojects) or third-party platforms (Tableau Public, LinkedIn).

##### README for additional insights
- It lists six major projects (Beautiful Soup web scraper, Apache Beam/Cloud Dataflow, Spark on EMR, Kafka streaming, Google Data Analytics capstone, plus various SQL/Tableau/Python mini-projects).
- Each project uses distinct cloud services and frameworks: GCP Dataflow & BigQuery, AWS EMR/S3, Apache Kafka, Python/pandas, Tableau, etc.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Portfolio Portal
    Portal["GitHub README Portal"]:::doc
    click Portal "https://github.com/lucjankonopka/portfolio/blob/main/README.md"

    %% Projects
    Portal --> BSProj["Beautiful Soup Scraper"]:::doc
    Portal --> BeamProj["Apache Beam ETL"]:::doc
    Portal --> SparkProj["Spark on AWS EMR"]:::doc
    Portal --> KafkaProj["Kafka Streaming"]:::doc
    Portal --> GDCProj["Google Data Analytics Capstone"]:::doc
    Portal --> MiniProj["SQL/Python Mini-Projects"]:::doc
    Portal --> Profiles["External Profiles"]:::external

    %% Beautiful Soup Scraper Subgraph
    subgraph "Beautiful Soup Scraper"
        BSWeb["Web Scraper (Python/Beautiful Soup)"]:::comp
        BSData[(Raw HTML Files)]:::storage
        BSWeb --> BSData
    end

    %% Apache Beam ETL Subgraph
    subgraph "Apache Beam ETL"
        BeamJob["Apache Beam Job"]:::comp
        Dataflow["GCP Dataflow"]:::comp
        BigQuery1[(BigQuery)]:::storage
        BeamJob --> Dataflow
        Dataflow --> BigQuery1
    end

    %% Spark on EMR Subgraph
    subgraph "Spark on AWS EMR"
        SparkJob["Spark Job"]:::comp
        EMR["AWS EMR Cluster"]:::comp
        S3_1[(AWS S3)]:::storage
        SparkJob --> EMR
        EMR --> S3_1
    end

    %% Kafka Streaming Subgraph
    subgraph "Kafka Streaming"
        Producer["Kafka Producer"]:::comp
        Kafka["Kafka Cluster"]:::comp
        S3_2[(AWS S3)]:::storage
        Athena["AWS Athena"]:::comp
        Producer --> Kafka
        Kafka --> S3_2
        S3_2 --> Athena
    end

    %% Google Data Analytics Capstone Subgraph
    subgraph "Google Data Analytics Capstone"
        Sheets["Google Sheets Extract"]:::comp
        BigQuery2[(BigQuery)]:::storage
        Viz["Tableau Public"]:::external
        Sheets --> BigQuery2
        BigQuery2 --> Viz
    end

    %% Mini Projects Subgraph
    subgraph "SQL/Python Mini-Projects"
        SQL["SQL Queries"]:::comp
        Pandas["Python/pandas"]:::comp
        Pics["Matplotlib/Seaborn"]:::comp
    end

    %% External Profiles Subgraph
    subgraph "External Profiles"
        LinkedIn["LinkedIn"]:::external
        TableauPub["Tableau Public"]:::external
    end

    %% Styles
    classDef doc fill:#D6E9FF,stroke:#333,stroke-width:1px;
    classDef comp fill:#FFD8B1,stroke:#333,stroke-width:1px;
    classDef storage fill:#D4EFDF,stroke:#333,stroke-width:1px;
    classDef external fill:#E0E0E0,stroke:#333,stroke-width:1px;
```
