How usgpo/bill-status works
Project Type and Purpose
• This is neither a traditional “frontend-backend” app nor a compiler; it’s a data-pipeline + static-content repository that harvests, organizes, and publishes U.S. Congress Bill Status XML in bulk.
• It runs scheduled jobs to pull XML files from the GPO/GovInfo Bulk Data API, organizes them (by Congress, by sample category), generates ancillary artifacts (RSS feed, sitemap), and publishes everything via static hosting (GovInfo’s site + GitHub).
a. Scheduler/Orchestrator
– Cron or workflow engine triggering two pipelines:
• “Current Congress” job (every 4 hrs)
• “Prior Congress” job (daily)
b. Ingestion Service
– Scripts (Python/Perl/Node/etc.) that call the GPO Bulk Data REST endpoints, fetch XML bundles for specified ranges.
– Handles retries, error logging.
c. Processing & Organization
– Post-fetch “ETL”: unzip/process XML, move into directory structure (samples/actions-committees, amendment-votes, formatChange/current|future, legacy, textVersions).
– Any filtering or enrichment (e.g., extracting sample files).
d. Storage Layer
– File system or cloud object store (S3/GCS) holding:
• All raw and organized XML files
• Generated RSS feed XML
• Sitemap XML
e. Publication Layer
– Static hosting (GitHub Pages + GovInfo Bulk Data site) serving:
• /bulkdata/BILLSTATUS/… directories
• RSS feed endpoint (/rss/billstatus-batch.xml)
– Automatic sitemap index (/sitemap/bulkdata/BILLSTATUS/…)
f. Documentation Artifacts
– User guides (PDF/MD), CHANGELOG, meeting slides—published alongside or linked from the site.
• Scheduler → Ingestion Service
• Ingestion Service → Processing/Organization
• Processing/Organization → Storage Layer
• Storage Layer → Publication Layer (trigger static site rebuild or sync)
• Storage Layer → RSS Generator → Storage Layer (write updated feed)
• Publication Layer → End users (via GovInfo website, GitHub, RSS clients)
Architectural Patterns & Principles
• Batch-Oriented ETL Pipeline (discrete runs on schedule)
• Static-Site Publication (no dynamic backend)
• Clear separation between data ingestion, processing, storage, and publication
• Use of existing GovInfo Bulk Data APIs and static-hosting infrastructure
Relevant Technologies & Frameworks
• Cron or workflow scheduler (e.g., Jenkins, Airflow, plain cron)
• HTTP client libraries in Python/Node/etc. for REST calls
• File I/O and directory management
• Static site tooling (Jekyll, Hugo, or GovInfo’s own pipeline)
• XML generators for RSS and sitemap (simple templating)
• Git + GitHub Pages for hosting / versioning
• GovInfo CDN / GovInfo.gov bulk-data infrastructure
Essential Elements
• Clear component labels and boundaries
• Compute/Jobs
• Data Storage
• Publication/Consumer Endpoints
• Notes on timing (4 hrs vs. daily) and external dependencies (GPO Bulk Data API)