How orangecoding/fredy works
1. Fredy is a self-hosted real-estate aggregation system split into two major runtimes: a Node.js backend and a separate React/Vite frontend. The backend exposes the application API and runs the scraping, job execution, storage, notification, and cron logic; the frontend is only a UI client over those backend endpoints.
2. The central backend orchestrator is the job pipeline, centered around FredyPipelineExecutioner. A job combines one or more providers with one or more notification adapters, runs on a configured schedule, deduplicates results, and persists new listings so they are not re-sent.
3. Providers are the inbound integration layer for real-estate sources. Each provider module under lib/provider knows how to fetch and normalize one marketplace, including browser-based scraping for sites like Immowelt, Immonet, Kleinanzeigen, WG-Gesucht, etc., plus a special reverse-engineered mobile API path for ImmoScout. This is the main boundary where Fredy interacts with external sites and anti-bot measures.
4. Extraction is split into a browser-based scraper stack and parser/normalization utilities. The extraction services use a hardened headless browser setup (CloakBrowser) for sites that require a real browser fingerprint, while ImmoScout uses its own API translator. Proxy support matters here because it changes how the scraper reaches those external systems.
5. Notification adapters form the outbound delivery layer. The adapter modules under lib/notification/adapter cover channels such as Slack, Telegram, email providers, ntfy, Discord, Mattermost, and SQLite/console variants. The job pipeline fans out new listings to all enabled adapters for that job.
6. SQLite is the primary persistence layer and an important architectural boundary. Storage modules manage listings, jobs, users, settings, watchlists, backup/restore, and migrations. The database lives separately from the app process and is intended to be mounted as a durable volume in Docker deployments.
7. The backend API is organized by domain-specific routers under lib/api/routes: login/security, dashboard, jobs, listings, providers, users, settings, backup, debugging, tracking, version, and notification adapters. These routes are the contract consumed by the web UI and by any other clients such as the MCP interface.
8. Real-time or push-style updates are handled through an SSE broker. That gives the UI a way to reflect backend events like job progress, logs, or status changes without polling everything continuously.
9. There are several background services that keep the system current even when no user is interacting with it. Cron jobs handle geocoding, checking whether listings are still alive, and tracking-related tasks; these operate against stored data and external services, then update the SQLite state.
10. Geocoding and distance calculation are a separate service area. Listing coordinates can be enriched via a geocoding client built around Nominatim, and distance calculations are used to support spatial filtering and map-based views in the UI.
11. Duplicate detection and listing quality are handled in backend services before data reaches users. The similarity cache, active-listing checks, and listing-format utilities help prevent repeated notifications and support cleanup of stale results.
12. The UI is a full admin console, not just a static viewer. It covers dashboard analytics, jobs, listings, maps, users, settings, debug tooling, and provider/adapter configuration. It depends on backend APIs for all state and uses shared client utilities, i18n, and local transformers to present backend models cleanly.
13. The repo includes a built-in MCP server as a separate integration surface. It exposes Fredy’s real-estate data to LLM clients via HTTP and stdio adapters, with authentication and normalization layers, so AI tools can query or enrich listings without going through the web UI.
14. Debugging and observability are first-class features. Fredy has structured logging, optional database-backed debug log capture, downloadable debug bundles, and test coverage around these paths. That makes diagnostics a distinct subsystem rather than just console output.
15. Operationally, the project is designed for self-hosting. Docker, docker-compose, config files under conf/, and volume-mounted SQLite storage define the deployment boundary, while the test suite covers provider scraping, storage migrations, notification adapters, API routes, and offline fixture-based parsing.
Main entry points: routes API routes · FredyPipelineExecutioner.js Job pipeline · storage SQLite storage · crons Cron services
Components 20
Backend runtime
Node.js server, API, jobs, storage, and integrations.
- Server entry · Starts the backend process and wires runtime configuration.
index.js - Job pipeline · Coordinates provider runs, deduplication, storage, and notification fan-out.
lib/FredyPipelineExecutioner.js - Providers · Marketplace-specific fetchers and normalizers for real-estate sources.
lib/provider - Extraction stack · Browser-based scraping, parsing, and bot-prevention support.
lib/services/extractor - ImmoScout path · Special reverse-engineered path for ImmoScout data retrieval and normalization.
lib/services/immoscout - Notifications · Outbound adapters that send new listings to enabled channels.
lib/notification/adapter - SQLite storage · Jobs, listings, users, settings, watchlists, backups, and migrations.
lib/services/storage - API routes · Domain routers for auth, dashboard, jobs, listings, users, settings, backup, debug, tracking, version, and adapters.
lib/api/routes - Security · Login, session, and route protection for backend access.
lib/api/security.js - SSE broker · Broadcasts backend events to the UI without polling.
lib/services/sse/sse-broker.js - Cron services · Geocoding, listing liveness, and tracker maintenance jobs.
lib/services/crons - Geocoding · Nominatim-backed geocoding and distance calculations.
lib/services/geocoding - Listing QC · Similarity cache, active-listing checks, and listing formatting helpers.
lib/services/listings
Frontend app
React/Vite admin console consuming backend endpoints.
- UI entry · Vite/React entry points for the admin console.
ui/src/Index.jsx - Dashboard · Overview and KPI surfaces for jobs and listings.
ui/src/views/dashboard/Dashboard.jsx - Jobs UI · Job creation, mutation, and execution monitoring.
ui/src/views/jobs - Listings UI · Listing search, detail, map, and watchlist management.
ui/src/views/listings - Admin config · Settings, users, login, provider and adapter configuration screens.
ui/src/views
Operational surface
Deployment, debugging, and external integration surfaces.
- Debugging · Structured logs, debug log storage, and downloadable debug bundles.
lib/services/debug - MCP server · HTTP and stdio access to Fredy data for LLM clients with auth and normalization.
lib/mcp
Connections 27
- Server entry → API routes: serves · Backend process exposes the API layer.
- Server entry → Job pipeline: starts · Server bootstraps job execution infrastructure.
- Job pipeline → Providers: pulls listings · Pipeline runs one or more source integrations per job.
- Job pipeline → Extraction stack: scrapes · Browser-based extraction and parsing feed the pipeline.
- Job pipeline → ImmoScout path: special case · ImmoScout uses a dedicated translator path.
- Job pipeline → Listing QC: dedupes · New listings are filtered before persistence and notification.
- Job pipeline → SQLite storage: persists · Listings and job state are written to SQLite.
- Job pipeline → Notifications: fans out · Accepted listings are delivered through enabled adapters.
- Providers → Extraction stack: uses · Most providers rely on browser-based extraction helpers.
- Providers → SQLite storage: reads config · Providers are configured and tracked through persisted state.
- Notifications → SQLite storage: reads state · Adapter settings and job wiring are stored in SQLite.
- API routes → Security: protects · Routes are guarded by backend security logic.
- API routes → SQLite storage: reads/writes · Routes expose persisted jobs, listings, users, settings, and backups.
- API routes → SSE broker: publishes · API events are forwarded to the push broker.
- Cron services → Geocoding: enriches · Background tasks update coordinates and distances.
- Cron services → Listing QC: checks · Cron jobs maintain listing liveness and tracking state.
- Cron services → SQLite storage: updates · Background maintenance mutates persisted records.
- Geocoding → SQLite storage: stores coords · Geocoding results are saved with listings.
- Debugging → SQLite storage: captures logs · Optional DB-backed debug logs are persisted.
- MCP server → SQLite storage: queries · MCP surfaces normalized Fredy data from stored records.
- UI entry → API routes: consumes · Frontend depends on backend routes for application state.
- UI entry → SSE broker: subscribes · UI listens for live backend updates.
- Dashboard → API routes: loads · Dashboard reads aggregated backend data.
- Jobs UI → API routes: manages · Jobs screens create and inspect job definitions.
- Listings UI → API routes: queries · Listings views fetch listing data and actions.
- Admin config → API routes: configures · Login, users, settings, providers, and adapters are driven by API calls.
- Admin config → Debugging: shows · UI can surface debug tooling and diagnostics.