How us-check/server-firestore works
Project Type and Purpose
• This is a backend service built with Django (Python) whose primary role is to expose RESTful endpoints (in uscheck_firestore/api/) and to synchronize or migrate data into Google Firestore (via code in uscheck_firestore/firestore/ and the script upload_json_to_firestore.py).
• It also integrates with Google Cloud Platform (GCP) components—Cloud Storage (for file uploads/exports), Pub/Sub (for asynchronous messaging), and Cloud Functions (for event-driven processing).
Key Elements from the File Structure
• Top-level Django project (uscheck_firestore/uscheck_firestore/): settings, URLs, WSGI/ASGI entry points.
• Two Django apps:
• api/ – Models, views, serializers (if any), URLs that define the HTTP API.
• firestore/ – Code for interacting with Firestore: models, services, and views likely used internally or administratively.
• Management script upload_json_to_firestore.py for bulk data ingestion.
• requirements.txt defines Python dependencies (e.g., Django, google-cloud-firestore, google-cloud-storage, etc.).
Architectural Patterns and Design Principles
• MVC (Django’s MVT) pattern:
• Models in each app (models.py)
• Views/Controllers (views.py, api/services.py)
• URL routing layer (urls.py)
• Service layer abstraction (api/services.py) to encapsulate business logic.
• Event-driven integration with GCP: Pub/Sub topics and Cloud Functions subscribe to storage or messaging events.
Main Components
a. Django Backend (Compute Engine / App Engine / GKE)
• HTTP(S) Load Balancer → Django application servers
• Internal modules: API app, Firestore app, Management scripts
b. Firestore Database
• Two Firestore collections (e.g., tour data, user data)
c. Google Cloud Storage (GCS)
• Bucket(s) used for uploading source JSON files (us_tourdata_final.txt), storing exports/backups
d. Pub/Sub
• Topics for “new file uploaded” and “data ingest complete” events
e. Cloud Functions
• Triggered by GCS “OBJECT_FINALIZE” events and Pub/Sub messages
• Perform parsing, call Django’s ingestion endpoints or call Firestore SDK directly
f. External Clients
• CLI or web clients consuming the REST API
Relationships and Data Flows
• External client → (1) HTTP POST/GET → Load Balancer → Django API → (2) Firestore SDK → Firestore
• Management script (upload_json_to_firestore.py) → (3) directly writes to Firestore via SDK
• Client uploads JSON to GCS → (4) GCS “OBJECT_FINALIZE” event → Pub/Sub → (5) Cloud Function → (6) HTTP request to Django ingestion endpoint / direct Firestore write
• Django publishes a status message → (7) Pub/Sub topic “ingest-complete” → downstream subscribers or Cloud Function for notifications