How denisisaev-create/wop works
1. Overall shape
- This is a Python backend that replaces a Google Sheets-based workflow for the “WOP” feed.
- The system is centered on a FastAPI service, a relational database layer, an ingestion/sync pipeline, and a small browser UI served as static assets.
2. Primary runtime entrypoint
- app/main.py is the application entrypoint and likely wires together the API app, routes, startup hooks, and scheduler initialization.
- The repo is intended to run under uvicorn app.main:app, so the API server is the main runtime boundary.
3. HTTP/API boundary
- app/views.py defines the HTTP endpoints, while app/schemas.py defines the request/response contracts.
- The documented endpoints show three major concerns: health checks, load querying, and sync operations (/sync, /sync/upload, /sync/status, /loads, /schedule).
- This is the boundary where external clients interact with the backend.
4. Data persistence layer
- app/db.py and app/models.py form the persistence layer.
- app/config.py centralizes environment-driven settings such as database connection details and IMAP/scheduler flags.
- The code likely uses an ORM-backed relational store; the important architectural point is that all ingestion and API reads/writes go through this database layer, not directly through the source feed.
5. Ingestion pipeline
- The app/ingest/ package is the core transformation pipeline.
- feed_source.py abstracts where feed data comes from, parser.py reads raw feed content, normalize.py standardizes it, mappers.py converts source fields into internal models, and upsert.py writes the resulting records into the database.
- This package is the main boundary between messy external feed formats and the internal canonical load representation.
6. Sync orchestration
- app/sync.py appears to coordinate end-to-end synchronization: fetch feed data, run parsing/normalization/mapping, then apply database updates.
- app/clock.py and app/scheduler.py suggest time-based orchestration, with the scheduler triggering periodic syncs after startup.
- The readme indicates the scheduler runs one sync on startup and then every 15 minutes when enabled.
7. Feed sources and manual fallback
- The system supports two feed acquisition modes:
- Live email sync via Gmail IMAP using an app password and the WOP FEED label.
- Manual CSV upload via /sync/upload for fallback or early setup.
- This means the ingestion pipeline is source-agnostic at the top, but converges into a single normalization/upsert path downstream.
8. Business rules and safety controls
- app/rules.py and app/enums.py likely encode domain-specific load status, archival, and filtering logic.
- The readme describes a key safety boundary: if a sync would archive too large a fraction of active loads, the archive step is skipped and recorded in sync history.
- This is important for architecture because it prevents bad upstream data from deleting too much state during a single run.
9. UI and presentation layer
- app/static/index.html, app/static/app.js, and app/static/styles.css implement a lightweight frontend.
- This is not a separate SPA backend; it is a thin presentation layer served by the same Python application, likely used to inspect loads and sync state.
10. Operational/observability surface
- /health provides basic liveness checking.
- /sync/status and probably a sync_log-style persistence path provide operational visibility into recent runs and whether the archive guard tripped.
- This makes sync execution a first-class operational workflow, not just a background job.
11. Utility and backfill tooling
- scripts/load_csv.py is a utility for importing CSV data outside the live HTTP path.
- docs/HANDOFF.md, PROJECT_STATUS.md, and the WOP spec/prompts files document the domain and transition from the old Sheets workflow.
12. Tests as architectural guardrails
- tests/test_api.py, tests/test_ingest.py, and tests/test_mappers.py show the three main boundaries being protected: HTTP behavior, ingestion behavior, and source-to-domain mapping.
- That test split matches the architecture: API layer, transformation pipeline, and domain mapping/persistence.
Main entry points: sync.py sync · views.py views · upsert.py upsert · db.py db
Components 18
API layer
HTTP boundary and app bootstrap
Core services
Sync orchestration, business rules, and scheduling
- sync · Coordinates end-to-end feed synchronization and persistence.
app/sync.py - scheduler · Triggers periodic sync runs when enabled.
app/scheduler.py - clock · Provides scheduling time boundaries and run timing support.
app/clock.py - rules · Encodes archive and load-state safety rules.
app/rules.py - enums · Defines load and sync enumerations used across the app.
app/enums.py
Ingestion pipeline
Source acquisition and feed transformation
- feed source · Abstracts where raw feed data comes from.
app/ingest/feed_source.py - parser · Reads raw feed content into structured records.
app/ingest/parser.py - normalize · Cleans and standardizes feed records.
app/ingest/normalize.py - mappers · Converts source fields into canonical internal models.
app/ingest/mappers.py - upsert · Persists normalized records into the database.
app/ingest/upsert.py
Persistence
Database access and domain models
Connections 21
- main → views: mounts · Registers the HTTP routes with the application.
- main → scheduler: starts · Initializes periodic sync scheduling on startup.
- main → db: initializes · Brings up database access for the app lifecycle.
- views → schemas: uses · Validates and serializes request/response payloads.
- views → sync: triggers · Starts on-demand sync operations and status queries.
- views → db: reads/writes · Exposes load and sync data through the persistence layer.
- scheduler → sync: runs · Invokes scheduled synchronization runs.
- scheduler → clock: times · Uses clock boundaries for recurring execution.
- sync → feed source: fetches · Obtains feed data from live or manual sources.
- sync → rules: applies · Checks safety and archive policy before committing changes.
- sync → parser: parses · Converts raw feed content into structured items.
- sync → normalize: normalizes · Standardizes feed records before mapping.
- sync → mappers: maps · Transforms source fields into canonical load records.
- sync → upsert: persists · Writes transformed records to the database.
- upsert → db: uses · Stores records through the database layer.
- upsert → models: writes · Operates on the ORM/domain model layer.
- models → enums: depends on · Uses shared domain state and status values.
- db → config: configured by · Reads connection and runtime settings from config.
- static UI → views: served by · Browser UI is delivered from the same application.
- load_csv → mappers: reuses · Shares the mapping path for offline CSV imports.
- load_csv → upsert: loads · Writes imported rows through the normal persistence path.