# orangecoding/fredy architecture

> Architecture diagram of the GitHub repository orangecoding/fredy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/orangecoding/fredy
- Repository: https://github.com/orangecoding/fredy

## Overview

1. Fredy is a self-hosted real-estate aggregation system split into two major runtimes: a Node.js backend and a separate React/Vite frontend. The backend exposes the application API and runs the scraping, job execution, storage, notification, and cron logic; the frontend is only a UI client over those backend endpoints.

2. The central backend orchestrator is the job pipeline, centered around `FredyPipelineExecutioner`. A job combines one or more providers with one or more notification adapters, runs on a configured schedule, deduplicates results, and persists new listings so they are not re-sent.

3. Providers are the inbound integration layer for real-estate sources. Each provider module under `lib/provider` knows how to fetch and normalize one marketplace, including browser-based scraping for sites like Immowelt, Immonet, Kleinanzeigen, WG-Gesucht, etc., plus a special reverse-engineered mobile API path for ImmoScout. This is the main boundary where Fredy interacts with external sites and anti-bot measures.

4. Extraction is split into a browser-based scraper stack and parser/normalization utilities. The extraction services use a hardened headless browser setup (`CloakBrowser`) for sites that require a real browser fingerprint, while ImmoScout uses its own API translator. Proxy support matters here because it changes how the scraper reaches those external systems.

5. Notification adapters form the outbound delivery layer. The adapter modules under `lib/notification/adapter` cover channels such as Slack, Telegram, email providers, ntfy, Discord, Mattermost, and SQLite/console variants. The job pipeline fans out new listings to all enabled adapters for that job.

6. SQLite is the primary persistence layer and an important architectural boundary. Storage modules manage listings, jobs, users, settings, watchlists, backup/restore, and migrations. The database lives separately from the app process and is intended to be mounted as a durable volume in Docker deployments.

7. The backend API is organized by domain-specific routers under `lib/api/routes`: login/security, dashboard, jobs, listings, providers, users, settings, backup, debugging, tracking, version, and notification adapters. These routes are the contract consumed by the web UI and by any other clients such as the MCP interface.

8. Real-time or push-style updates are handled through an SSE broker. That gives the UI a way to reflect backend events like job progress, logs, or status changes without polling everything continuously.

9. There are several background services that keep the system current even when no user is interacting with it. Cron jobs handle geocoding, checking whether listings are still alive, and tracking-related tasks; these operate against stored data and external services, then update the SQLite state.

10. Geocoding and distance calculation are a separate service area. Listing coordinates can be enriched via a geocoding client built around Nominatim, and distance calculations are used to support spatial filtering and map-based views in the UI.

11. Duplicate detection and listing quality are handled in backend services before data reaches users. The similarity cache, active-listing checks, and listing-format utilities help prevent repeated notifications and support cleanup of stale results.

12. The UI is a full admin console, not just a static viewer. It covers dashboard analytics, jobs, listings, maps, users, settings, debug tooling, and provider/adapter configuration. It depends on backend APIs for all state and uses shared client utilities, i18n, and local transformers to present backend models cleanly.

13. The repo includes a built-in MCP server as a separate integration surface. It exposes Fredy’s real-estate data to LLM clients via HTTP and stdio adapters, with authentication and normalization layers, so AI tools can query or enrich listings without going through the web UI.

14. Debugging and observability are first-class features. Fredy has structured logging, optional database-backed debug log capture, downloadable debug bundles, and test coverage around these paths. That makes diagnostics a distinct subsystem rather than just console output.

15. Operationally, the project is designed for self-hosting. Docker, docker-compose, config files under `conf/`, and volume-mounted SQLite storage define the deployment boundary, while the test suite covers provider scraping, storage migrations, notification adapters, API routes, and offline fixture-based parsing.

Main entry points: [`lib/api/routes`](https://github.com/orangecoding/fredy/tree/master/lib/api/routes) (API routes), [`lib/FredyPipelineExecutioner.js`](https://github.com/orangecoding/fredy/blob/master/lib/FredyPipelineExecutioner.js) (Job pipeline), [`lib/services/storage`](https://github.com/orangecoding/fredy/tree/master/lib/services/storage) (SQLite storage), [`lib/services/crons`](https://github.com/orangecoding/fredy/tree/master/lib/services/crons) (Cron services)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_backend["Backend runtime"]
  node_entry["Server entry<br/>bootstrap<br/>[index.js]"]
  node_pipeline["Job pipeline<br/>orchestrator"]
  node_providers["Providers<br/>source integrations"]
  node_extractor["Extraction stack<br/>scraping services"]
  node_immoscout["ImmoScout path<br/>mobile-api translator"]
  node_notifications["Notifications<br/>delivery adapters"]
  node_storage[("SQLite storage<br/>persistence layer")]
  node_api["API routes<br/>http contract"]
  node_security["Security<br/>authz/authn<br/>[security.js]"]
  node_sse["SSE broker<br/>push channel<br/>[sse-broker.js]"]
  node_crons["Cron services<br/>background tasks"]
  node_geocoding["Geocoding<br/>location service"]
  node_listingqc["Listing QC<br/>dedupe/quality"]
end

subgraph group_frontend["Frontend app"]
  node_uientry["UI entry<br/>app bootstrap<br/>[Index.jsx]"]
  node_dashboard["Dashboard<br/>admin view<br/>[Dashboard.jsx]"]
  node_jobsui["Jobs UI<br/>admin view"]
  node_listingsui["Listings UI<br/>admin view"]
  node_adminui["Admin config<br/>admin view"]
end

subgraph group_ops["Operational surface"]
  node_debug["Debugging<br/>observability"]
  node_mcp["MCP server<br/>llm integration"]
end

node_entry -->|"serves"| node_api
node_entry -->|"starts"| node_pipeline
node_pipeline -->|"pulls listings"| node_providers
node_pipeline -->|"scrapes"| node_extractor
node_pipeline -->|"special case"| node_immoscout
node_pipeline -->|"dedupes"| node_listingqc
node_pipeline -->|"persists"| node_storage
node_pipeline -->|"fans out"| node_notifications
node_providers -->|"uses"| node_extractor
node_providers -->|"reads config"| node_storage
node_notifications -->|"reads state"| node_storage
node_api -->|"protects"| node_security
node_api -->|"reads/writes"| node_storage
node_api -->|"publishes"| node_sse
node_crons -->|"enriches"| node_geocoding
node_crons -->|"checks"| node_listingqc
node_crons -->|"updates"| node_storage
node_geocoding -->|"stores coords"| node_storage
node_debug -->|"captures logs"| node_storage
node_mcp -->|"queries"| node_storage
node_uientry -->|"consumes"| node_api
node_uientry -->|"subscribes"| node_sse
node_dashboard -->|"loads"| node_api
node_jobsui -->|"manages"| node_api
node_listingsui -->|"queries"| node_api
node_adminui -->|"configures"| node_api
node_adminui -->|"shows"| node_debug

click node_entry "https://github.com/orangecoding/fredy/blob/master/index.js"
click node_pipeline "https://github.com/orangecoding/fredy/blob/master/lib/FredyPipelineExecutioner.js"
click node_providers "https://github.com/orangecoding/fredy/tree/master/lib/provider"
click node_extractor "https://github.com/orangecoding/fredy/tree/master/lib/services/extractor"
click node_immoscout "https://github.com/orangecoding/fredy/tree/master/lib/services/immoscout"
click node_notifications "https://github.com/orangecoding/fredy/tree/master/lib/notification/adapter"
click node_storage "https://github.com/orangecoding/fredy/tree/master/lib/services/storage"
click node_api "https://github.com/orangecoding/fredy/tree/master/lib/api/routes"
click node_security "https://github.com/orangecoding/fredy/blob/master/lib/api/security.js"
click node_sse "https://github.com/orangecoding/fredy/blob/master/lib/services/sse/sse-broker.js"
click node_crons "https://github.com/orangecoding/fredy/tree/master/lib/services/crons"
click node_geocoding "https://github.com/orangecoding/fredy/tree/master/lib/services/geocoding"
click node_listingqc "https://github.com/orangecoding/fredy/tree/master/lib/services/listings"
click node_debug "https://github.com/orangecoding/fredy/tree/master/lib/services/debug"
click node_mcp "https://github.com/orangecoding/fredy/tree/master/lib/mcp"
click node_uientry "https://github.com/orangecoding/fredy/blob/master/ui/src/Index.jsx"
click node_dashboard "https://github.com/orangecoding/fredy/blob/master/ui/src/views/dashboard/Dashboard.jsx"
click node_jobsui "https://github.com/orangecoding/fredy/tree/master/ui/src/views/jobs"
click node_listingsui "https://github.com/orangecoding/fredy/tree/master/ui/src/views/listings"
click node_adminui "https://github.com/orangecoding/fredy/tree/master/ui/src/views"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_entry,node_pipeline,node_providers,node_extractor,node_immoscout,node_notifications,node_storage,node_api,node_security,node_sse,node_crons,node_geocoding,node_listingqc toneBlue
class node_uientry,node_dashboard,node_jobsui,node_listingsui,node_adminui toneAmber
class node_debug,node_mcp toneMint
```

## Components

### Backend runtime

Node.js server, API, jobs, storage, and integrations.

- [Server entry](https://github.com/orangecoding/fredy/blob/master/index.js): bootstrap · Starts the backend process and wires runtime configuration. · `index.js`
- [Job pipeline](https://github.com/orangecoding/fredy/blob/master/lib/FredyPipelineExecutioner.js): orchestrator · Coordinates provider runs, deduplication, storage, and notification fan-out. · `lib/FredyPipelineExecutioner.js`
- [Providers](https://github.com/orangecoding/fredy/tree/master/lib/provider): source integrations · Marketplace-specific fetchers and normalizers for real-estate sources. · `lib/provider`
- [Extraction stack](https://github.com/orangecoding/fredy/tree/master/lib/services/extractor): scraping services · Browser-based scraping, parsing, and bot-prevention support. · `lib/services/extractor`
- [ImmoScout path](https://github.com/orangecoding/fredy/tree/master/lib/services/immoscout): mobile-api translator · Special reverse-engineered path for ImmoScout data retrieval and normalization. · `lib/services/immoscout`
- [Notifications](https://github.com/orangecoding/fredy/tree/master/lib/notification/adapter): delivery adapters · Outbound adapters that send new listings to enabled channels. · `lib/notification/adapter`
- [SQLite storage](https://github.com/orangecoding/fredy/tree/master/lib/services/storage): persistence layer · Jobs, listings, users, settings, watchlists, backups, and migrations. · `lib/services/storage`
- [API routes](https://github.com/orangecoding/fredy/tree/master/lib/api/routes): http contract · Domain routers for auth, dashboard, jobs, listings, users, settings, backup, debug, tracking, version, and adapters. · `lib/api/routes`
- [Security](https://github.com/orangecoding/fredy/blob/master/lib/api/security.js): authz/authn · Login, session, and route protection for backend access. · `lib/api/security.js`
- [SSE broker](https://github.com/orangecoding/fredy/blob/master/lib/services/sse/sse-broker.js): push channel · Broadcasts backend events to the UI without polling. · `lib/services/sse/sse-broker.js`
- [Cron services](https://github.com/orangecoding/fredy/tree/master/lib/services/crons): background tasks · Geocoding, listing liveness, and tracker maintenance jobs. · `lib/services/crons`
- [Geocoding](https://github.com/orangecoding/fredy/tree/master/lib/services/geocoding): location service · Nominatim-backed geocoding and distance calculations. · `lib/services/geocoding`
- [Listing QC](https://github.com/orangecoding/fredy/tree/master/lib/services/listings): dedupe/quality · Similarity cache, active-listing checks, and listing formatting helpers. · `lib/services/listings`

### Frontend app

React/Vite admin console consuming backend endpoints.

- [UI entry](https://github.com/orangecoding/fredy/blob/master/ui/src/Index.jsx): app bootstrap · Vite/React entry points for the admin console. · `ui/src/Index.jsx`
- [Dashboard](https://github.com/orangecoding/fredy/blob/master/ui/src/views/dashboard/Dashboard.jsx): admin view · Overview and KPI surfaces for jobs and listings. · `ui/src/views/dashboard/Dashboard.jsx`
- [Jobs UI](https://github.com/orangecoding/fredy/tree/master/ui/src/views/jobs): admin view · Job creation, mutation, and execution monitoring. · `ui/src/views/jobs`
- [Listings UI](https://github.com/orangecoding/fredy/tree/master/ui/src/views/listings): admin view · Listing search, detail, map, and watchlist management. · `ui/src/views/listings`
- [Admin config](https://github.com/orangecoding/fredy/tree/master/ui/src/views): admin view · Settings, users, login, provider and adapter configuration screens. · `ui/src/views`

### Operational surface

Deployment, debugging, and external integration surfaces.

- [Debugging](https://github.com/orangecoding/fredy/tree/master/lib/services/debug): observability · Structured logs, debug log storage, and downloadable debug bundles. · `lib/services/debug`
- [MCP server](https://github.com/orangecoding/fredy/tree/master/lib/mcp): llm integration · HTTP and stdio access to Fredy data for LLM clients with auth and normalization. · `lib/mcp`

## Connections

- Server entry → API routes: serves · Backend process exposes the API layer.
- Server entry → Job pipeline: starts · Server bootstraps job execution infrastructure.
- Job pipeline → Providers: pulls listings · Pipeline runs one or more source integrations per job.
- Job pipeline → Extraction stack: scrapes · Browser-based extraction and parsing feed the pipeline.
- Job pipeline → ImmoScout path: special case · ImmoScout uses a dedicated translator path.
- Job pipeline → Listing QC: dedupes · New listings are filtered before persistence and notification.
- Job pipeline → SQLite storage: persists · Listings and job state are written to SQLite.
- Job pipeline → Notifications: fans out · Accepted listings are delivered through enabled adapters.
- Providers → Extraction stack: uses · Most providers rely on browser-based extraction helpers.
- Providers → SQLite storage: reads config · Providers are configured and tracked through persisted state.
- Notifications → SQLite storage: reads state · Adapter settings and job wiring are stored in SQLite.
- API routes → Security: protects · Routes are guarded by backend security logic.
- API routes → SQLite storage: reads/writes · Routes expose persisted jobs, listings, users, settings, and backups.
- API routes → SSE broker: publishes · API events are forwarded to the push broker.
- Cron services → Geocoding: enriches · Background tasks update coordinates and distances.
- Cron services → Listing QC: checks · Cron jobs maintain listing liveness and tracking state.
- Cron services → SQLite storage: updates · Background maintenance mutates persisted records.
- Geocoding → SQLite storage: stores coords · Geocoding results are saved with listings.
- Debugging → SQLite storage: captures logs · Optional DB-backed debug logs are persisted.
- MCP server → SQLite storage: queries · MCP surfaces normalized Fredy data from stored records.
- UI entry → API routes: consumes · Frontend depends on backend routes for application state.
- UI entry → SSE broker: subscribes · UI listens for live backend updates.
- Dashboard → API routes: loads · Dashboard reads aggregated backend data.
- Jobs UI → API routes: manages · Jobs screens create and inspect job definitions.
- Listings UI → API routes: queries · Listings views fetch listing data and actions.
- Admin config → API routes: configures · Login, users, settings, providers, and adapters are driven by API calls.
- Admin config → Debugging: shows · UI can surface debug tooling and diagnostics.
