# patlittle/gc-socials architecture

> Architecture diagram of the GitHub repository patlittle/gc-socials, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/patlittle/gc-socials
- Repository: https://github.com/patlittle/gc-socials

## Overview

#### Purpose and Published Surface
GC-Socials is a Git-backed historical tracker of official Government of Canada social-media accounts. Its primary consumer-facing artifacts are versioned CSV datasets and a generated `README.md` containing current summaries, account-change tables, and Mermaid charts. FlatGitHub is the interactive data viewer linked from `README.md`.

Paths: `README.md`, `sm.csv`, `platform_counts.csv`, `department_counts.csv`

#### Source Acquisition and Canonical Dataset
The pipeline retrieves the official English and French Canada.ca social-account listings, converts fetched JSON into CSV, and maintains `sm.csv` as the canonical account-level dataset. Each repository update represents a snapshot used for longitudinal comparisons.

Paths: `fetch_json_to_csv.py`, `sm.csv`, `README.md`

#### Change Detection and History
A Git-diff stage compares the current account snapshot with its prior repository version. It emits deleted-account records and supplies recent additions/deletions for the README’s “last 14 days” reporting. Git history is therefore the persistence mechanism for account evolution rather than a database.

Paths: `create_git_diff_csv.py`, `deleted_rows.csv`, `recent_changes_snippet.md`

#### Aggregation Layer
Derived tabular summaries count accounts over time by platform/language and department. These CSVs are committed outputs, enabling FlatGitHub users to inspect time-series data directly without executing scripts or accessing an API.

Paths: `platform_counts.csv`, `department_counts.csv`, `sm.csv`

#### Visualization and README Assembly
Specialized generators produce Mermaid source for the platform-language Sankey and pie-chart sections. A static README template/content is combined with generated snippets to publish the current analytical dashboard in `README.md`.

Paths: `generate_sankey.py`, `sankey_diagram.md`, `generate_pie_chart_readme.py`, `readme_static.md`, `README.md`

#### Organization Enrichment Boundary
The project also maintains Government of Canada organization reference data and Wikidata identifiers. This is a separate enrichment/update flow from social-account scraping, likely supporting organization normalization or future linkage rather than the core public snapshot itself.

Paths: `wikidataorgsupdate.py`, `gc_orgs_current.csv`, `gc_orgs_with_wikidata_ids.csv`

#### Runtime, Automation, and External Dependencies
The implementation is Python-script based and repository-state driven. GitHub Actions orchestrates independent scheduled or manually triggered stages for the main data pipeline, organization updates, Sankey generation, and pie-chart/README generation. External service boundaries are Canada.ca’s English/French social listing pages, Wikidata for organization IDs, GitHub Actions/Git version history for automation and state, and FlatGitHub for CSV presentation.

Paths: `.github/workflows/main.yml`, `.github/workflows/update-orgs.yml`, `.github/workflows/sankey.yml`, `.github/workflows/generate_pie_chart_workflow.yml`

Main entry points: [`sm.csv`](https://github.com/patlittle/gc-socials/blob/main/sm.csv) (Account snapshot), [`create_git_diff_csv.py`](https://github.com/patlittle/gc-socials/blob/main/create_git_diff_csv.py) (Change detection), [`README.md`](https://github.com/patlittle/gc-socials/blob/main/README.md) (Published dashboard), [`wikidataorgsupdate.py`](https://github.com/patlittle/gc-socials/blob/main/wikidataorgsupdate.py) (Organization ID updater)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_automation["GitHub Actions"]
  node_main_workflow["Main pipeline workflow<br/>[main.yml]"]
  node_sankey_workflow["Sankey workflow<br/>[sankey.yml]"]
  node_pie_workflow["Pie and README workflow"]
  node_orgs_workflow["Organization update workflow<br/>[update-orgs.yml]"]
end

subgraph group_social["Social account pipeline"]
  node_canada_social_listings{{"Canada.ca social listings<br/>external source"}}
  node_fetch_converter["Fetch JSON to CSV<br/>Python script"]
  node_social_snapshot[("Account snapshot<br/>canonical CSV<br/>[sm.csv]")]
  node_git_history[("Git history<br/>versioned state")]
  node_diff_generator["Change detection<br/>Python script"]
  node_deleted_accounts["Deleted accounts<br/>derived CSV<br/>[deleted_rows.csv]"]
end

subgraph group_publishing["Published analytics"]
  node_recent_changes["Recent changes snippet<br/>generated Markdown"]
  node_summary_tables[("Time-series summaries<br/>derived CSVs")]
  node_department_counts["Department counts<br/>derived CSV"]
  node_sankey_generator["Sankey generator<br/>Python script<br/>[generate_sankey.py]"]
  node_sankey_snippet["Sankey Mermaid<br/>generated Markdown<br/>[sankey_diagram.md]"]
  node_pie_generator["Pie chart generator<br/>Python script"]
  node_readme_template["README static content<br/>Markdown template<br/>[readme_static.md]"]
  node_readme["Published dashboard<br/>README<br/>[README.md]"]
  node_flatgithub{{"FlatGitHub<br/>external viewer"}}
end

subgraph group_orgs["Organization enrichment"]
  node_wikidata{{"Wikidata<br/>external source"}}
  node_org_updater["Organization ID updater<br/>Python script"]
  node_current_orgs[("Current organizations<br/>reference CSV")]
  node_orgs_with_ids[("Organizations with Wikidata IDs<br/>enriched CSV")]
end

node_main_workflow -->|"runs"| node_fetch_converter
node_canada_social_listings -->|"listing data"| node_fetch_converter
node_fetch_converter -->|"writes"| node_social_snapshot
node_social_snapshot -->|"committed snapshot"| node_git_history
node_main_workflow -->|"runs"| node_diff_generator
node_git_history -->|"prior version"| node_diff_generator
node_social_snapshot -->|"current version"| node_diff_generator
node_diff_generator -->|"emits"| node_deleted_accounts
node_diff_generator -->|"produces"| node_recent_changes
node_social_snapshot -->|"aggregates to"| node_summary_tables
node_social_snapshot -->|"aggregates to"| node_department_counts
node_sankey_workflow -->|"runs"| node_sankey_generator
node_social_snapshot -->|"source data"| node_sankey_generator
node_sankey_generator -->|"generates"| node_sankey_snippet
node_pie_workflow -->|"runs"| node_pie_generator
node_summary_tables -->|"summary data"| node_pie_generator
node_pie_generator -->|"updates charts"| node_readme
node_readme_template -->|"static content"| node_readme
node_recent_changes -->|"change section"| node_readme
node_sankey_snippet -->|"diagram section"| node_readme
node_readme -->|"links to"| node_flatgithub
node_orgs_workflow -->|"runs"| node_org_updater
node_current_orgs -->|"organization data"| node_org_updater
node_wikidata -->|"identifiers"| node_org_updater
node_org_updater -->|"writes"| node_orgs_with_ids

click node_main_workflow "https://github.com/patlittle/gc-socials/blob/main/.github/workflows/main.yml"
click node_fetch_converter "https://github.com/patlittle/gc-socials/blob/main/fetch_json_to_csv.py"
click node_social_snapshot "https://github.com/patlittle/gc-socials/blob/main/sm.csv"
click node_diff_generator "https://github.com/patlittle/gc-socials/blob/main/create_git_diff_csv.py"
click node_deleted_accounts "https://github.com/patlittle/gc-socials/blob/main/deleted_rows.csv"
click node_recent_changes "https://github.com/patlittle/gc-socials/blob/main/recent_changes_snippet.md"
click node_summary_tables "https://github.com/patlittle/gc-socials/blob/main/platform_counts.csv"
click node_department_counts "https://github.com/patlittle/gc-socials/blob/main/department_counts.csv"
click node_sankey_workflow "https://github.com/patlittle/gc-socials/blob/main/.github/workflows/sankey.yml"
click node_sankey_generator "https://github.com/patlittle/gc-socials/blob/main/generate_sankey.py"
click node_sankey_snippet "https://github.com/patlittle/gc-socials/blob/main/sankey_diagram.md"
click node_pie_workflow "https://github.com/patlittle/gc-socials/blob/main/.github/workflows/generate_pie_chart_workflow.yml"
click node_pie_generator "https://github.com/patlittle/gc-socials/blob/main/generate_pie_chart_readme.py"
click node_readme_template "https://github.com/patlittle/gc-socials/blob/main/readme_static.md"
click node_readme "https://github.com/patlittle/gc-socials/blob/main/README.md"
click node_orgs_workflow "https://github.com/patlittle/gc-socials/blob/main/.github/workflows/update-orgs.yml"
click node_org_updater "https://github.com/patlittle/gc-socials/blob/main/wikidataorgsupdate.py"
click node_current_orgs "https://github.com/patlittle/gc-socials/blob/main/gc_orgs_current.csv"
click node_orgs_with_ids "https://github.com/patlittle/gc-socials/blob/main/gc_orgs_with_wikidata_ids.csv"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_workflow,node_sankey_workflow,node_pie_workflow,node_orgs_workflow toneBlue
class node_canada_social_listings,node_fetch_converter,node_social_snapshot,node_git_history,node_diff_generator,node_deleted_accounts toneAmber
class node_recent_changes,node_summary_tables,node_department_counts,node_sankey_generator,node_sankey_snippet,node_pie_generator,node_readme_template,node_readme,node_flatgithub toneMint
class node_wikidata,node_org_updater,node_current_orgs,node_orgs_with_ids toneRose
```

## Components

### GitHub Actions

Scheduled or manual workflows run and commit repository updates.

- [Main pipeline workflow](https://github.com/patlittle/gc-socials/blob/main/.github/workflows/main.yml): workflow · `.github/workflows/main.yml`
- [Sankey workflow](https://github.com/patlittle/gc-socials/blob/main/.github/workflows/sankey.yml): workflow · `.github/workflows/sankey.yml`
- [Pie and README workflow](https://github.com/patlittle/gc-socials/blob/main/.github/workflows/generate_pie_chart_workflow.yml): workflow · `.github/workflows/generate_pie_chart_workflow.yml`
- [Organization update workflow](https://github.com/patlittle/gc-socials/blob/main/.github/workflows/update-orgs.yml): workflow · `.github/workflows/update-orgs.yml`

### Social account pipeline

- **Canada.ca social listings**: external source · Official English and French account listings.
- [Fetch JSON to CSV](https://github.com/patlittle/gc-socials/blob/main/fetch_json_to_csv.py): Python script · Retrieves listings and converts them to account records. · `fetch_json_to_csv.py`
- [Account snapshot](https://github.com/patlittle/gc-socials/blob/main/sm.csv): canonical CSV · Canonical account-level dataset for each repository revision. · `sm.csv`
- **Git history**: versioned state · Persists prior snapshots for longitudinal comparison.
- [Change detection](https://github.com/patlittle/gc-socials/blob/main/create_git_diff_csv.py): Python script · Compares current and prior account snapshots. · `create_git_diff_csv.py`
- [Deleted accounts](https://github.com/patlittle/gc-socials/blob/main/deleted_rows.csv): derived CSV · `deleted_rows.csv`

### Published analytics

- [Recent changes snippet](https://github.com/patlittle/gc-socials/blob/main/recent_changes_snippet.md): generated Markdown · Additions and deletions reported for the last 14 days. · `recent_changes_snippet.md`
- [Time-series summaries](https://github.com/patlittle/gc-socials/blob/main/platform_counts.csv): derived CSVs · Counts by platform/language and department. · `platform_counts.csv`
- [Department counts](https://github.com/patlittle/gc-socials/blob/main/department_counts.csv): derived CSV · `department_counts.csv`
- [Sankey generator](https://github.com/patlittle/gc-socials/blob/main/generate_sankey.py): Python script · `generate_sankey.py`
- [Sankey Mermaid](https://github.com/patlittle/gc-socials/blob/main/sankey_diagram.md): generated Markdown · `sankey_diagram.md`
- [Pie chart generator](https://github.com/patlittle/gc-socials/blob/main/generate_pie_chart_readme.py): Python script · `generate_pie_chart_readme.py`
- [README static content](https://github.com/patlittle/gc-socials/blob/main/readme_static.md): Markdown template · `readme_static.md`
- [Published dashboard](https://github.com/patlittle/gc-socials/blob/main/README.md): README · Current summaries, changes, and Mermaid charts. · `README.md`
- **FlatGitHub**: external viewer · Interactive viewer for committed CSV datasets.

### Organization enrichment

Separate reference-data update flow.

- **Wikidata**: external source
- [Organization ID updater](https://github.com/patlittle/gc-socials/blob/main/wikidataorgsupdate.py): Python script · `wikidataorgsupdate.py`
- [Current organizations](https://github.com/patlittle/gc-socials/blob/main/gc_orgs_current.csv): reference CSV · `gc_orgs_current.csv`
- [Organizations with Wikidata IDs](https://github.com/patlittle/gc-socials/blob/main/gc_orgs_with_wikidata_ids.csv): enriched CSV · `gc_orgs_with_wikidata_ids.csv`

## Connections

- Main pipeline workflow → Fetch JSON to CSV: runs
- Canada.ca social listings → Fetch JSON to CSV: listing data
- Fetch JSON to CSV → Account snapshot: writes
- Account snapshot → Git history: committed snapshot
- Main pipeline workflow → Change detection: runs
- Git history → Change detection: prior version
- Account snapshot → Change detection: current version
- Change detection → Deleted accounts: emits
- Change detection → Recent changes snippet: produces
- Account snapshot → Time-series summaries: aggregates to
- Account snapshot → Department counts: aggregates to
- Sankey workflow → Sankey generator: runs
- Account snapshot → Sankey generator: source data
- Sankey generator → Sankey Mermaid: generates
- Pie and README workflow → Pie chart generator: runs
- Time-series summaries → Pie chart generator: summary data
- Pie chart generator → Published dashboard: updates charts
- README static content → Published dashboard: static content
- Recent changes snippet → Published dashboard: change section
- Sankey Mermaid → Published dashboard: diagram section
- Published dashboard → FlatGitHub: links to
- Organization update workflow → Organization ID updater: runs
- Current organizations → Organization ID updater: organization data
- Wikidata → Organization ID updater: identifiers
- Organization ID updater → Organizations with Wikidata IDs: writes