How patlittle/gc-socials works
Purpose and Published Surface
GC-Socials is a Git-backed historical tracker of official Government of Canada social-media accounts. Its primary consumer-facing artifacts are versioned CSV datasets and a generated README.md containing current summaries, account-change tables, and Mermaid charts. FlatGitHub is the interactive data viewer linked from README.md.
Paths: README.md, sm.csv, platform_counts.csv, department_counts.csv
Source Acquisition and Canonical Dataset
The pipeline retrieves the official English and French Canada.ca social-account listings, converts fetched JSON into CSV, and maintains sm.csv as the canonical account-level dataset. Each repository update represents a snapshot used for longitudinal comparisons.
Paths: fetch_json_to_csv.py, sm.csv, README.md
Change Detection and History
A Git-diff stage compares the current account snapshot with its prior repository version. It emits deleted-account records and supplies recent additions/deletions for the README’s “last 14 days” reporting. Git history is therefore the persistence mechanism for account evolution rather than a database.
Paths: create_git_diff_csv.py, deleted_rows.csv, recent_changes_snippet.md
Aggregation Layer
Derived tabular summaries count accounts over time by platform/language and department. These CSVs are committed outputs, enabling FlatGitHub users to inspect time-series data directly without executing scripts or accessing an API.
Paths: platform_counts.csv, department_counts.csv, sm.csv
Visualization and README Assembly
Specialized generators produce Mermaid source for the platform-language Sankey and pie-chart sections. A static README template/content is combined with generated snippets to publish the current analytical dashboard in README.md.
Paths: generate_sankey.py, sankey_diagram.md, generate_pie_chart_readme.py, readme_static.md, README.md
Organization Enrichment Boundary
The project also maintains Government of Canada organization reference data and Wikidata identifiers. This is a separate enrichment/update flow from social-account scraping, likely supporting organization normalization or future linkage rather than the core public snapshot itself.
Paths: wikidataorgsupdate.py, gc_orgs_current.csv, gc_orgs_with_wikidata_ids.csv
Runtime, Automation, and External Dependencies
The implementation is Python-script based and repository-state driven. GitHub Actions orchestrates independent scheduled or manually triggered stages for the main data pipeline, organization updates, Sankey generation, and pie-chart/README generation. External service boundaries are Canada.ca’s English/French social listing pages, Wikidata for organization IDs, GitHub Actions/Git version history for automation and state, and FlatGitHub for CSV presentation.
Paths: .github/workflows/main.yml, .github/workflows/update-orgs.yml, .github/workflows/sankey.yml, .github/workflows/generate_pie_chart_workflow.yml
Main entry points: sm.csv Account snapshot · create_git_diff_csv.py Change detection · README.md Published dashboard · wikidataorgsupdate.py Organization ID updater
Components 23
GitHub Actions
Scheduled or manual workflows run and commit repository updates.
- Main pipeline workflow
.github/workflows/main.yml - Sankey workflow
.github/workflows/sankey.yml - Pie and README workflow
.github/workflows/generate_pie_chart_workflow.yml - Organization update workflow
.github/workflows/update-orgs.yml
Social account pipeline
- Canada.ca social listings · Official English and French account listings.
- Fetch JSON to CSV · Retrieves listings and converts them to account records.
fetch_json_to_csv.py - Account snapshot · Canonical account-level dataset for each repository revision.
sm.csv - Git history · Persists prior snapshots for longitudinal comparison.
- Change detection · Compares current and prior account snapshots.
create_git_diff_csv.py - Deleted accounts
deleted_rows.csv
Published analytics
- Recent changes snippet · Additions and deletions reported for the last 14 days.
recent_changes_snippet.md - Time-series summaries · Counts by platform/language and department.
platform_counts.csv - Department counts
department_counts.csv - Sankey generator
generate_sankey.py - Sankey Mermaid
sankey_diagram.md - Pie chart generator
generate_pie_chart_readme.py - README static content
readme_static.md - Published dashboard · Current summaries, changes, and Mermaid charts.
README.md - FlatGitHub · Interactive viewer for committed CSV datasets.
Organization enrichment
Separate reference-data update flow.
- Wikidata
- Organization ID updater
wikidataorgsupdate.py - Current organizations
gc_orgs_current.csv - Organizations with Wikidata IDs
gc_orgs_with_wikidata_ids.csv
Connections 25
- Main pipeline workflow → Fetch JSON to CSV: runs
- Canada.ca social listings → Fetch JSON to CSV: listing data
- Fetch JSON to CSV → Account snapshot: writes
- Account snapshot → Git history: committed snapshot
- Main pipeline workflow → Change detection: runs
- Git history → Change detection: prior version
- Account snapshot → Change detection: current version
- Change detection → Deleted accounts: emits
- Change detection → Recent changes snippet: produces
- Account snapshot → Time-series summaries: aggregates to
- Account snapshot → Department counts: aggregates to
- Sankey workflow → Sankey generator: runs
- Account snapshot → Sankey generator: source data
- Sankey generator → Sankey Mermaid: generates
- Pie and README workflow → Pie chart generator: runs
- Time-series summaries → Pie chart generator: summary data
- Pie chart generator → Published dashboard: updates charts
- README static content → Published dashboard: static content
- Recent changes snippet → Published dashboard: change section
- Sankey Mermaid → Published dashboard: diagram section
- Published dashboard → FlatGitHub: links to
- Organization update workflow → Organization ID updater: runs
- Current organizations → Organization ID updater: organization data
- Wikidata → Organization ID updater: identifiers
- Organization ID updater → Organizations with Wikidata IDs: writes