# floschnell/properwatcher architecture

> Architecture diagram of the GitHub repository floschnell/properwatcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/floschnell/properwatcher
- Repository: https://github.com/floschnell/properwatcher

## Overview

##### Main System Components

a) Core Components:
- Crawler System (src/crawlers/)
- Filter System (src/filters/)
- Enricher System (src/enrichers/)
- Observer System (src/observers/)
- Configuration Manager (src/configuration.rs)
- Data Models (src/models/)

##### External Integration Points
- Property Websites (ImmobilienScout24, Wohnungsboerse, etc.)
- AWS DynamoDB
- Telegram API
- SMTP Email Services
- OpenStreetMap Nominatim API

##### Specific Implementation Details

Technical details in subcomponents:
- Parallel processing in the crawler section
- Async operations using Rust's async/await
- Firefox Servo technology integration
- AWS Lambda integration points

##### Key Elements to Highlight

- Parallel processing capability
- Modular architecture
- AWS Lambda support
- Data transformation pipeline
- External service integration

**[Property Websites]**
- ↓
**[Crawler System (Parallel)]**
- ↓
**[Filter System]**
- ↓
**[Enricher System]**
- ↓
**[Observer System]**
- ↓
**[Storage & Notifications]**

Important Considerations:

##### Technical details
- Thread pools for parallel processing
- Async runtime indicators
- AWS Lambda configuration
- Database schemas

##### Configuration flow
- TOML/JSON config input
- Configuration distribution to components
- Runtime configuration management

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Data Sources
    subgraph DataSources
        IS24["ImmobilienScout24"]:::external
        WB["Wohnungsboerse"]:::external
        WG["WG-Gesucht"]:::external
        IW["Immowelt"]:::external
        SD["Sueddeutsche"]:::external
    end

    %% Core Processing Pipeline
    subgraph CorePipeline
        %% Crawler System
        subgraph CrawlerSystem
            CS["Crawler System"]:::core
            IS24C["ImmobilienScout24 Crawler"]:::crawler
            WBC["Wohnungsboerse Crawler"]:::crawler
            WGC["WG-Gesucht Crawler"]:::crawler
            IWC["Immowelt Crawler"]:::crawler
            SDC["Sueddeutsche Crawler"]:::crawler
        end

        %% Filter System
        subgraph FilterSystem
            FS["Filter System"]:::core
            CF["Core Filter"]:::filter
            DF["DynamoDB Filter"]:::filter
            CSVF["CSV Filter"]:::filter
            FC["Filter Criteria"]:::filter
        end

        %% Enricher System
        subgraph EnricherSystem
            ES["Enricher System"]:::core
            CE["Core Enricher"]:::enricher
            NE["Nominatim Enricher"]:::enricher
        end

        %% Observer System
        subgraph ObserverSystem
            OS["Observer System"]:::core
            CO["Core Observer"]:::observer
            DO["DynamoDB Observer"]:::observer
            TO["Telegram Observer"]:::observer
            EO["Email Observer"]:::observer
            CSVO["CSV Observer"]:::observer
            DBO["Debug Observer"]:::observer
        end
    end

    %% External Services
    subgraph ExternalServices
        DB[(AWS DynamoDB)]:::storage
        TA["Telegram API"]:::notification
        EA["SMTP Email"]:::notification
        OSM["OpenStreetMap Nominatim"]:::external
    end

    %% Configuration
    CM["Configuration Manager"]:::config

    %% Data Models
    subgraph Models
        FM["Flat Model"]:::model
        EN["Encodings"]:::model
    end

    %% Relationships
    IS24 --> IS24C
    WB --> WBC
    WG --> WGC
    IW --> IWC
    SD --> SDC

    IS24C & WBC & WGC & IWC & SDC --> FS
    FS --> ES
    ES --> OS
    
    NE --> OSM
    DO --> DB
    TO --> TA
    EO --> EA

    CM --> CrawlerSystem & FilterSystem & EnricherSystem & ObserverSystem

    %% Click Events
    click CS "https://github.com/floschnell/properwatcher/tree/master/src/crawlers/"
    click IS24C "https://github.com/floschnell/properwatcher/blob/master/src/crawlers/immoscout.rs"
    click WBC "https://github.com/floschnell/properwatcher/blob/master/src/crawlers/wohnungsboerse.rs"
    click WGC "https://github.com/floschnell/properwatcher/blob/master/src/crawlers/wggesucht.rs"
    click IWC "https://github.com/floschnell/properwatcher/blob/master/src/crawlers/immowelt.rs"
    click SDC "https://github.com/floschnell/properwatcher/blob/master/src/crawlers/sueddeutsche.rs"
    
    click FS "https://github.com/floschnell/properwatcher/tree/master/src/filters/"
    click CF "https://github.com/floschnell/properwatcher/blob/master/src/filters/filter.rs"
    click DF "https://github.com/floschnell/properwatcher/blob/master/src/filters/dynamodb.rs"
    click CSVF "https://github.com/floschnell/properwatcher/blob/master/src/filters/csv.rs"
    click FC "https://github.com/floschnell/properwatcher/blob/master/src/filters/criteria.rs"
    
    click ES "https://github.com/floschnell/properwatcher/tree/master/src/enrichers/"
    click CE "https://github.com/floschnell/properwatcher/blob/master/src/enrichers/enricher.rs"
    click NE "https://github.com/floschnell/properwatcher/blob/master/src/enrichers/nominatim.rs"
    
    click OS "https://github.com/floschnell/properwatcher/tree/master/src/observers/"
    click CO "https://github.com/floschnell/properwatcher/blob/master/src/observers/observer.rs"
    click DO "https://github.com/floschnell/properwatcher/blob/master/src/observers/dynamodb.rs"
    click TO "https://github.com/floschnell/properwatcher/blob/master/src/observers/telegram.rs"
    click EO "https://github.com/floschnell/properwatcher/blob/master/src/observers/mail.rs"
    click CSVO "https://github.com/floschnell/properwatcher/blob/master/src/observers/csv.rs"
    click DBO "https://github.com/floschnell/properwatcher/blob/master/src/observers/debug.rs"
    
    click CM "https://github.com/floschnell/properwatcher/blob/master/src/configuration.rs"
    click FM "https://github.com/floschnell/properwatcher/blob/master/src/models/flat.rs"
    click EN "https://github.com/floschnell/properwatcher/blob/master/src/models/encodings.rs"

    %% Styles
    classDef external fill:#C2E0FF,stroke:#333
    classDef core fill:#B8E0D2,stroke:#333
    classDef crawler fill:#D4A5A5,stroke:#333
    classDef filter fill:#FFE5A5,stroke:#333
    classDef enricher fill:#A5D4AD,stroke:#333
    classDef observer fill:#D4A5D4,stroke:#333
    classDef storage fill:#FFB366,stroke:#333
    classDef notification fill:#DDA0DD,stroke:#333
    classDef config fill:#90EE90,stroke:#333
    classDef model fill:#87CEEB,stroke:#333
```
