# andrymaster/bigdata_yandex architecture

> Architecture diagram of the GitHub repository andrymaster/bigdata_yandex, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/andrymaster/bigdata_yandex
- Repository: https://github.com/andrymaster/bigdata_yandex

## Overview

##### Project Type and Purpose
- This is a Big Data educational project from the Yandex Лицей specialization. It contains a series of lessons (Lesson_1 through Lesson_8) demonstrating various techniques in data transformation, statistical analysis, and ETL processes.
- The final deliverable is a comprehensive dashboard built in Yandex DataLens with a backend powered by ClickHouse. The project’s purpose is to transform raw user log data from a large CSV file into interactive visualizations with derived tables and additional analytics.

##### Analyzing the File Structure
- Top-Level Organization:
– Multiple Lesson_* folders (Lesson_1, Lesson_2 … Lesson_8) each subdivided into two parts:
▸ DeepDive – containing exploratory notebooks, sample scripts (Python, SQL) and raw data files (CSV, TSV).
▸ HomeWork – containing practical assignments, additional notebooks, scripts, and data files used in exercises.
– A separate ‘Project’ folder, which though mostly light on code, contains configuration files (config.py), data (CityPosition.csv, users.csv) and documents that support the final dashboard project.
– Other files include the README, certificate, and license which emphasize the educational aspect.
- Architectural Implications:
– The many notebooks, scripts (like .py and .sql files) indicate a layered developmental process where initial exploration (analysis) feeds into transformation (ETL), and finally into visualization using DataLens and a ClickHouse database.
– The naming and structure point toward a “pipeline” or data flow approach – raw files are ingested, processed through various stages, and then visualized.

##### Insights from the README
- The README confirms that the specialization focused on processing large datasets (a 700 MB log file), transforming data using libraries like Pandas, and ultimately delivering interactive dashboards in DataLens with ClickHouse as the data source.
- It provides context on the ETL architecture, the transformation steps used to create derivative tables, and points to external references (screenshots, certificates) that further document the system.

a. Main Components:
– Data Sources: This includes the raw CSV/TSV log files and additional data (e.g., CityPosition.csv, customer_transactions_log.csv, etc.) provided in the lesson folders.
– Ingestion/ETL Layer: Represent the series of Python scripts (.py files), Jupyter Notebooks (.ipynb), and SQL files spread across the lesson folders that transform, clean, and prepare the data.
– Data Storage: The ClickHouse database (as mentioned in the README) that holds both raw and derived data tables.
– Visualization Layer: Yandex DataLens, where the final dashboard has been built and interacts with the ClickHouse backend.
– Additional Supporting Elements: A configuration component (e.g., config.py), and possibly a “project orchestrator” represented by the start.ipynb and generation scripts in the Project folder.

b. Relationships and Interactions:
– Data flows from the raw log files (input) into the ETL layer where data processing scripts and notebooks run transformations.
– Processed data is then loaded into the ClickHouse database.
– DataLens connects to ClickHouse to retrieve the data, enabling interactive visualizations and dashboards.
– Supporting configuration files (config.py, config.json) control how the ETL and deployment process operates.

c. Architectural Patterns and Principles:
– The system is built following a typical Data Pipeline architecture with clearly separated ingestion, processing, storage, and visualization stages.
– The use of interactive notebooks in lessons mirrors an iterative development process often seen in prototyping ETL pipelines.
– Emphasis on modularity: each lesson represents a discrete component (or stage) of the overall data processing flow, which ultimately converges into the final dashboard.

d. Relevant Technologies:
– Pandas, Matplotlib for data manipulation and visualization (in the notebooks).
– Python and SQL for data processing.
– ClickHouse as the database engine.
– Yandex DataLens for visualization and dashboarding.
– Supplementary tools (e.g., Jupyter Notebook for prototyping and analysis).

##### Instructing What
– Clear labels for each major component (e.g., “Raw Data Files”, “ETL Pipeline”, “ClickHouse Database”, “DataLens Dashboard”).
- Processing components (e.g., rectangle for notebooks or script environments).
– Optionally, include subcomponents or annotations to indicate:
- Where configuration files or scripts like config.py and start.ipynb are placed.
- How lesson components (DeepDive and HomeWork) contribute to incremental building of the ETL process.

##### Final Recommendations
– Be very detailed: capture every essential stage of the pipeline from data ingestion through processing to final presentation.
– Don’t hesitate to break down the architecture into as many components (nodes) as makes sense, especially if each lesson represents a key transformation.

Following this roadmap will provide the principal software engineer with a comprehensive overview and a detailed visual representation of this Big Data project’s architecture.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources subgraph
    subgraph "Data Sources"
        DDF["DeepDive Data Files"]:::data
        HDDF["HomeWork Data Files"]:::data
    end

    %% ETL & Processing subgraph
    subgraph "ETL Layer"
        ETL_N["Notebooks & Scripts"]:::process
        ETL_SQL["SQL Transformations"]:::process
        ORCH["Project Orchestration"]:::config
    end

    %% Data Storage & Visualization subgraph
    subgraph "Data Storage & Visualization"
        DB["ClickHouse Database"]:::database
        DASH["DataLens Dashboard"]:::dashboard
    end

    %% Data flow connections
    DDF -->|"ingests"| ETL_N
    HDDF -->|"ingests"| ETL_N
    ORCH -->|"orchestrates"| ETL_N
    ETL_N -->|"loads data into"| DB
    ETL_SQL -->|"processes and loads"| DB
    DB -->|"feeds data to"| DASH

    %% Click Events for Component Mappings
    click DDF "https://github.com/andrymaster/bigdata_yandex/tree/master/Lesson_1/DeepDive"
    click HDDF "https://github.com/andrymaster/bigdata_yandex/tree/master/Lesson_1/HomeWork"
    click ETL_N "https://github.com/andrymaster/bigdata_yandex/blob/master/Lesson_1/DeepDive/les_1_p1.ipynb"
    click ETL_SQL "https://github.com/andrymaster/bigdata_yandex/blob/master/Lesson_3/DeepDive/res_1.sql"
    click ORCH "https://github.com/andrymaster/bigdata_yandex/blob/master/Project/config.py"
    click ORCH "https://github.com/andrymaster/bigdata_yandex/blob/master/Project/start.ipynb"
    click ORCH "https://github.com/andrymaster/bigdata_yandex/blob/master/Project/generate.ipynb"

    %% Styles
    classDef data fill:#A1D490,stroke:#333,stroke-width:2px;
    classDef process fill:#F9D56E,stroke:#333,stroke-width:2px;
    classDef config fill:#FFD3E0,stroke:#333,stroke-width:2px;
    classDef database fill:#B0E0E6,stroke:#333,stroke-width:2px;
    classDef dashboard fill:#ADD8E6,stroke:#333,stroke-width:2px;
```
