How cmharlow/lc-reconcile works
Project Type & Purpose
• This is an open-source, Python-based HTTP reconciliation service (a microservice) that speaks the OpenRefine Reconciliation API.
• It sits between the OpenRefine UI (the client) and the Library of Congress APIs (id.loc.gov), enriching OpenRefine with fuzzy and alternate‐label matching for LCNAF and LCSH.
File Structure & Architectural Hints
• Top-level files:
– reconcile.py: main Flask (or similar) HTTP server handling API requests.
– text.py: normalization library for cleaning and canonicalizing query strings.
– requirements.txt: indicates Flask, requests, fuzzywuzzy, etc.
– README.md: details the reconciliation logic (Suggest + DidYouMean + ranking via fuzzywuzzy).
• No database, no frontend code—this is a thin, stateless HTTP service.
Key Components
a. OpenRefine Client
– Represents the user’s browser instance running the OpenRefine UI.
– Initiates reconciliation via HTTP calls.
b. Reconciliation Service (Your Code)
– HTTP Server Layer (Flask app in reconcile.py)
• Exposes endpoints: root for metadata, POST /reconcile or GET /reconcile queries (per the API spec).
– Business Logic Layer
• Query Normalizer (text.py)
• Suggest & DidYouMean orchestrator (invokes id.loc.gov endpoints)
• Ranking Module (fuzzywuzzy scoring and top-3 selection)
– Response Formatter
• Builds JSON result conforming to OpenRefine spec.
c. External Services
– id.loc.gov Suggest API
– id.loc.gov DidYouMean API
d. (Optional) Monitoring/Debug
– Debug flag in reconcile.py; logs printed to console.
Architectural Patterns & Design Principles
• Layered decomposition:
– API Layer (Flask)
– Service Layer (normalization + orchestration)
– Integration Layer (external HTTP clients)
• Single Responsibility: text.py only normalizes text; reconcile.py orchestrates heavy lifting.
• Statelessness: no persisted state or DB; each request is independent.
Technologies & Libraries
• Python 3.7+
• Flask (or Bottle) for HTTP server
• requests for external HTTP calls
• fuzzywuzzy for string matching/ranking
• text.py: custom normalizer for LC peculiarities
What to Include on the Canvas
• Title: “Library of Congress Reconciliation Service – Architecture”
– Client Layer
– Service Layer
– External API Layer
• All endpoints and modules with clear names (e.g., ReconService, TextNormalizer)