# pthie/rwi-geo-redc architecture

> Architecture diagram of the GitHub repository pthie/rwi-geo-redc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pthie/rwi-geo-redc
- Repository: https://github.com/pthie/rwi-geo-redc

## Overview

##### Main Components
- Data Input & Reading: Represented by the scripts under the “src/read” folder. This includes reading from various sources (e.g., housing data, geographical grids, organizational data).
- Data Cleaning: Scripts in “src/clean” handle cleaning of different data sources (e.g., SUF data, append data, microm and org data). Use a block labeled “Data Cleaning” with subcomponents that show cleaning for SUF, microm, org, etc.
- Data Creation: The “src/create” scripts (e.g., creating folder structure, value and variable labels) are part of the transformation process. Represent these as a separate module labeled “Data Creation” where the necessary metadata and structure is built.
- Orchestration and Naming Consistency: The “src/make” subdirectory shows preparation steps for ensuring consistent file naming. This can be a small module that links into both reading and exporting processes.
- Testing: Finally, the “src/test” folder contains various testing scripts. Represent a “Testing & Validation” block that interacts with every stage of the pipeline to confirm compliance (e.g., anonymization, testing spatial and temporal trends).

##### Relationships and Interactions
- Similarly, ensure that Testing & Validation has interactions with every main process, indicating regular checkpoint validations in the pipeline.

##### Architectural Patterns
- The project uses a procedural, script-driven pipeline (typical in data processing work), so it is beneficial to show this as a sequential data processing flow.
- That the project is modular: each task (clean, create, export) is separated into a distinct module or folder. This modular approach enhances flexibility and maintainability.

##### Relevant Technologies and Data Flow Details
- Since the project is entirely R-based, note R as the main language/environment and mention any R-specific practices (for example, source() commands to load scripts).
- External entities if necessary (for example, reference the online ImmoScout24 source in a “Data Origin” component, even if not directly part of the script structure).

##### Final Advice for the Principal Software Engineer
- Breaking down the project into as granular components as possible following the folders in the file tree.
- Sequential data flow (the ETL style process) and highlight the modular nature with separate components for reading, cleaning, creating, and exporting.
- To include “Testing & Validation” as a comprehensive aspect, showing that each stage of the pipeline is verified.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Core Pipeline Subgraphs
    subgraph "Core Pipeline"
        %% Data Input & Reading Subgraph
        subgraph "Data Input & Reading"
            READ["Data Input & Reading"]:::main
            read1["reading_exported_data.R"]:::file
            read2["reading_geo_data.R"]:::file
            read3["reading_grids.R"]:::file
            read4["reading_microm_data.R"]:::file
            read5["reading_org_data.R"]:::file
            READ --> read1
            READ --> read2
            READ --> read3
            READ --> read4
            READ --> read5
        end

        %% Data Cleaning Subgraph
        subgraph "Data Cleaning"
            CLEAN["Data Cleaning"]:::main
            clean1["cleaning_SUF_data.R"]:::file
            clean2["cleaning_append_data.R"]:::file
            clean3["cleaning_microm_data.R"]:::file
            clean4["cleaning_org_data.R"]:::file
            CLEAN --> clean1
            CLEAN --> clean2
            CLEAN --> clean3
            CLEAN --> clean4
        end

        %% Data Creation Subgraph
        subgraph "Data Creation"
            CREATE["Data Creation"]:::main
            create1["creating_folder_structure.R"]:::file
            create2["creating_value_labels.R"]:::file
            create3["creating_variable_labels.R"]:::file
            CREATE --> create1
            CREATE --> create2
            CREATE --> create3
        end

        %% Data Export Subgraph
        subgraph "Data Export"
            EXPORT["Data Export"]:::main
            export1["exporting_column_infos.R"]:::file
            export2["exporting_housing_data.R"]:::file
            export3["exporting_spatial_unit_names.R"]:::file
            EXPORT --> export1
            EXPORT --> export2
            EXPORT --> export3
        end
    end

    %% Supporting Modules Subgraph
    subgraph "Supporting Modules"
        HELPERS["Supporting Helpers"]:::support
        AUX["Auxiliary Processing"]:::support
        MAKE["Orchestration and Naming Consistency"]:::support
        TEST["Testing & Validation"]:::validation
    end

    %% Data Flow Connections within Core Pipeline
    READ -->|"flow"| CLEAN
    CLEAN -->|"flow"| CREATE
    CREATE -->|"flow"| EXPORT

    %% Auxiliary Processing flow connection
    READ -->|"aux_process"| AUX
    AUX -->|"aux_process"| EXPORT

    %% Orchestration linking to Input and Export stages
    MAKE -->|"orchestrates"| READ
    MAKE -->|"orchestrates"| EXPORT

    %% Supporting Helpers connection to Cleaning and Creation stages
    HELPERS -->|"supports"| CLEAN
    HELPERS -->|"supports"| CREATE

    %% Testing & Validation checkpoints for all main stages
    TEST -->|"validates"| READ
    TEST -->|"validates"| CLEAN
    TEST -->|"validates"| CREATE
    TEST -->|"validates"| EXPORT

    %% Click Events for Data Input & Reading
    click READ "https://github.com/pthie/rwi-geo-redc/tree/main/src/read"
    click read1 "https://github.com/pthie/rwi-geo-redc/blob/main/src/read/reading_exported_data.R"
    click read2 "https://github.com/pthie/rwi-geo-redc/blob/main/src/read/reading_geo_data.R"
    click read3 "https://github.com/pthie/rwi-geo-redc/blob/main/src/read/reading_grids.R"
    click read4 "https://github.com/pthie/rwi-geo-redc/blob/main/src/read/reading_microm_data.R"
    click read5 "https://github.com/pthie/rwi-geo-redc/blob/main/src/read/reading_org_data.R"

    %% Click Events for Data Cleaning
    click CLEAN "https://github.com/pthie/rwi-geo-redc/tree/main/src/clean"
    click clean1 "https://github.com/pthie/rwi-geo-redc/blob/main/src/clean/cleaning_SUF_data.R"
    click clean2 "https://github.com/pthie/rwi-geo-redc/blob/main/src/clean/cleaning_append_data.R"
    click clean3 "https://github.com/pthie/rwi-geo-redc/blob/main/src/clean/cleaning_microm_data.R"
    click clean4 "https://github.com/pthie/rwi-geo-redc/blob/main/src/clean/cleaning_org_data.R"

    %% Click Events for Data Creation
    click CREATE "https://github.com/pthie/rwi-geo-redc/tree/main/src/create"
    click create1 "https://github.com/pthie/rwi-geo-redc/blob/main/src/create/creating_folder_structure.R"
    click create2 "https://github.com/pthie/rwi-geo-redc/blob/main/src/create/creating_value_labels.R"
    click create3 "https://github.com/pthie/rwi-geo-redc/blob/main/src/create/creating_variable_labels.R"

    %% Click Events for Data Export
    click EXPORT "https://github.com/pthie/rwi-geo-redc/tree/main/src/export"
    click export1 "https://github.com/pthie/rwi-geo-redc/blob/main/src/export/exporting_column_infos.R"
    click export2 "https://github.com/pthie/rwi-geo-redc/blob/main/src/export/exporting_housing_data.R"
    click export3 "https://github.com/pthie/rwi-geo-redc/blob/main/src/export/exporting_spatial_unit_names.R"

    %% Click Events for Supporting Modules
    click HELPERS "https://github.com/pthie/rwi-geo-redc/tree/main/src/helpers"
    click AUX "https://github.com/pthie/rwi-geo-redc/tree/main/src/others"
    click MAKE "https://github.com/pthie/rwi-geo-redc/tree/main/src/make"
    click TEST "https://github.com/pthie/rwi-geo-redc/tree/main/src/test"

    %% Styles
    classDef main fill:#AED6F1,stroke:#2471A3,stroke-width:2px;
    classDef file fill:#D5F5E3,stroke:#27AE60,stroke-width:2px;
    classDef support fill:#F9E79F,stroke:#F1C40F,stroke-width:2px;
    classDef validation fill:#F5B7B1,stroke:#E74C3C,stroke-width:2px;
```
