# danielmiessler/substrate architecture

> Architecture diagram of the GitHub repository danielmiessler/substrate, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/danielmiessler/substrate
- Repository: https://github.com/danielmiessler/substrate

## Overview

1. **Repository role**
   - This is not an application server or product backend; it is a **knowledge infrastructure repository**.
   - The main artifact types are **Markdown documents, CSV datasets, and TypeScript update scripts**.
   - The architecture is centered on **human-curated knowledge + automated data ingestion**.

2. **Top-level knowledge model**
   - The root directories like `Problems/`, `Solutions/`, `Claims/`, `Arguments/`, `Questions/`, `Values/`, `Ideas/`, `Plans/`, `People/`, `Organizations/`, `Projects/`, `Frames/`, `Models/`, `Outcomes/`, `Risks/`, `Threats/`, and `Funding-Sources/` are the conceptual entities of the system.
   - These folders are the repository’s **semantic layer**: each one holds structured, likely markdown-based entries representing a distinct knowledge type.
   - The architecture depends on explicit cross-linking between these entity types rather than a database schema.

3. **Data subsystem**
   - `Data/` is the primary evidence store.
   - It contains curated datasets such as US GDP, inflation, Pulitzer winners, Bay Area wastewater, US approval polling, common metrics, and knowledge-worker salary data.
   - The design pattern is consistent: each dataset folder contains the raw/derived data files plus supporting docs like `README.md`, `SUMMARY.md`, `source.md`, `RESOURCES.md`, `UPDATES.md`, and sometimes `update.log`.

4. **Source catalog / ingestion layer**
   - `Data/sources/` is the connector layer for external authoritative sources.
   - Each `DS-xxxxx—SourceName/` directory maps to one upstream provider such as WHO, UN SDGs, World Bank, FRED, CDC WONDER, Census ACS, BLS JOLTS, EPA AQS, EIA, and Treasury FiscalData.
   - These source folders pair documentation with `update.ts`, making them the repo’s **ETL boundary**: fetch from outside systems, normalize, and write repo-local artifacts.

5. **Automation flow**
   - The repo uses **TypeScript with Bun** for update automation.
   - The typical flow is: **external API or public dataset → `update.ts` script → normalized CSV/JSON/Markdown outputs → logs/documentation**.
   - Some datasets also include helper executables or scripts, such as wastewater scraping/update commands, showing that ingestion can combine API pulls, scraping, and local transformation.

6. **Documentation-first boundary**
   - Almost every content area has its own `README.md` or summary document, so documentation is not incidental; it is part of the system boundary.
   - Dataset documentation captures provenance, methodology, limitations, and update cadence.
   - The repo’s “library science” framing means each dataset is intended to be both **human readable and machine parseable**.

7. **Research / analysis subsystem**
   - `research/` contains deeper analytical work products rather than canonical datasets.
   - These folders hold methodology, source lists, findings, syntheses, executive summaries, and topical analyses.
   - This is a downstream consumer of the data layer: the flow is **data collection → research analysis → written conclusions / findings**.

8. **Project-specific data pipelines**
   - Some datasets are self-contained mini-pipelines, especially in `Data/Bay-Area-COVID-Wastewater/` and several `Data/sources/DS-.../` directories.
   - These often include generated dashboards, update logs, and “latest” outputs, indicating a pattern where raw source data is transformed into a stable published dataset inside the repo.
   - The `dashboard.html` file in the wastewater dataset suggests some outputs are intended for direct viewing, not just storage.

9. **Governance and contribution boundary**
   - The repository is structured for community contribution through GitHub, with GitHub Actions in `.github/workflows/` providing automated checks or AI-assisted review.
   - `CLAUDE.md` and the Claude workflow files indicate AI is part of the maintenance and review process.
   - This means the architecture includes a **human + AI editorial workflow**, not just code automation.

10. **Dependency and integration model**
   - There is little evidence of a traditional runtime backend, database service, or frontend app.
   - Instead, the repo integrates with external public-data services and persists results as versioned files in Git.
   - Git itself is the durability layer: version history, diffs, and logs are the primary system of record.

11. **Data publication pattern**
   - Each major dataset/source follows the same publication contract: source description, methodology, extracted data, update history, and sometimes validation notes.
   - This creates a repeatable architecture for adding new datasets without changing the overall repo structure.
   - The resulting outputs are designed to be browsed directly from the file tree or consumed by scripts/AI tools.

12. **How to draw the architecture**
   - A good diagram would show:
     - **Top-level knowledge categories** as the core domain model.
     - **Data/sources** as ingestion connectors to external agencies/APIs.
     - **Dataset folders** as curated evidence repositories.
     - **Research/** as analytical outputs built from the datasets.
     - **GitHub Actions / Bun TypeScript scripts** as the automation and validation layer.
   - The central idea is a **knowledge graph-like content system** backed by source-backed datasets and lightweight automation, all stored in Git.

Main entry points: [`Data/README.md`](https://github.com/danielmiessler/substrate/blob/main/Data/README.md) (Data), [`research/README.md`](https://github.com/danielmiessler/substrate/blob/main/research/README.md) (Research), [`README.md`](https://github.com/danielmiessler/substrate/blob/main/README.md) (Substrate), [`.gitignore`](https://github.com/danielmiessler/substrate/blob/main/.gitignore) (Git history)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_knowledge["Knowledge layer"]
  node_node_entity_types["Entity folders<br/>semantic model"]
  node_node_crosslinking(("Cross-linking<br/>knowledge graph"))
  node_node_knowledge_docs["Entity docs<br/>markdown entries"]
end

subgraph group_group_data["Data layer"]
  node_node_data_root[("Data<br/>evidence store<br/>[README.md]")]
  node_node_dataset_pattern["Dataset pattern<br/>published dataset"]
  node_node_gdp["US GDP<br/>economic dataset"]
  node_node_inflation["US Inflation<br/>inflation dataset"]
  node_node_approval["Approval polls<br/>political polling dataset"]
  node_node_wastewater["Wastewater<br/>public health pipeline"]
end

subgraph group_group_sources["Source connectors"]
  node_node_sources_root[("Sources<br/>ingestion catalog")]
  node_node_source_scripts["update.ts scripts<br/>ETL automation"]
  node_node_external_apis(("External sources<br/>public APIs"))
end

subgraph group_group_research["Research layer"]
  node_node_research_root["Research<br/>analysis layer<br/>[README.md]"]
  node_node_research_methods["Methods<br/>research methodology"]
  node_node_research_findings["Findings<br/>syntheses"]
end

subgraph group_group_ops["Automation & governance"]
  node_node_validation["Validation<br/>checks and logs"]
  node_node_github_actions["GitHub Actions<br/>workflow automation<br/>[claude.yml]"]
  node_node_claude_md["CLAUDE.md<br/>editorial policy<br/>[CLAUDE.md]"]
  node_node_git(("Git history<br/>versioned storage<br/>[.gitignore]"))
end

node_node_root["Substrate<br/>knowledge repo<br/>[README.md]"]

node_node_root -->|"organizes"| node_node_entity_types
node_node_root -->|"contains"| node_node_data_root
node_node_root -->|"contains"| node_node_sources_root
node_node_root -->|"contains"| node_node_research_root
node_node_entity_types -->|"uses"| node_node_crosslinking
node_node_entity_types -->|"documented by"| node_node_knowledge_docs
node_node_data_root -->|"standardized by"| node_node_dataset_pattern
node_node_data_root -->|"includes"| node_node_gdp
node_node_data_root -->|"includes"| node_node_inflation
node_node_data_root -->|"includes"| node_node_approval
node_node_data_root -->|"includes"| node_node_wastewater
node_node_sources_root -->|"contains"| node_node_source_scripts
node_node_source_scripts -->|"pulls from"| node_node_external_apis
node_node_source_scripts -->|"writes to"| node_node_data_root
node_node_source_scripts -->|"emits"| node_node_validation
node_node_github_actions -->|"runs"| node_node_source_scripts
node_node_claude_md -->|"guides"| node_node_github_actions
node_node_git -->|"stores"| node_node_data_root
node_node_git -->|"stores"| node_node_research_root
node_node_research_root -->|"contains"| node_node_research_methods
node_node_research_root -->|"contains"| node_node_research_findings
node_node_research_root -->|"analyzes"| node_node_data_root

click node_node_root "https://github.com/danielmiessler/substrate/blob/main/README.md"
click node_node_data_root "https://github.com/danielmiessler/substrate/blob/main/Data/README.md"
click node_node_dataset_pattern "https://github.com/danielmiessler/substrate/blob/main/Data/DATASET-TEMPLATE.md"
click node_node_gdp "https://github.com/danielmiessler/substrate/tree/main/Data/US-GDP"
click node_node_inflation "https://github.com/danielmiessler/substrate/tree/main/Data/US-Inflation"
click node_node_approval "https://github.com/danielmiessler/substrate/tree/main/Data/US-Presidential-Approval"
click node_node_wastewater "https://github.com/danielmiessler/substrate/tree/main/Data/Bay-Area-COVID-Wastewater"
click node_node_sources_root "https://github.com/danielmiessler/substrate/tree/main/Data/sources"
click node_node_github_actions "https://github.com/danielmiessler/substrate/blob/main/.github/workflows/claude.yml"
click node_node_claude_md "https://github.com/danielmiessler/substrate/blob/main/CLAUDE.md"
click node_node_git "https://github.com/danielmiessler/substrate/blob/main/.gitignore"
click node_node_research_root "https://github.com/danielmiessler/substrate/blob/main/research/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_entity_types,node_node_crosslinking,node_node_knowledge_docs toneBlue
class node_node_data_root,node_node_dataset_pattern,node_node_gdp,node_node_inflation,node_node_approval,node_node_wastewater toneAmber
class node_node_sources_root,node_node_source_scripts,node_node_external_apis toneMint
class node_node_research_root,node_node_research_methods,node_node_research_findings toneRose
class node_node_validation,node_node_github_actions,node_node_claude_md,node_node_git toneIndigo
class node_node_root toneNeutral
```

## Components

### Knowledge layer

Core semantic entities organized as linked markdown knowledge types.

- **Entity folders**: semantic model · Top-level knowledge categories such as Problems, Solutions, Claims, Arguments, Questions, Values, Ideas, Plans, People, Organizations, Projects, Frames, Models, Outcomes, Risks, Threats, and Funding-Sources.
- **Cross-linking**: knowledge graph · Explicit relationships between markdown entries across entity folders.
- **Entity docs**: markdown entries · Folder-level readmes and entry files that make each knowledge type both human readable and machine parseable.

### Data layer

Curated datasets and published evidence repositories.

- [Data](https://github.com/danielmiessler/substrate/blob/main/Data/README.md): evidence store · Primary curated data area containing published datasets, summaries, resources, and update history. · `Data/README.md`
- [Dataset pattern](https://github.com/danielmiessler/substrate/blob/main/Data/DATASET-TEMPLATE.md): published dataset · Common per-dataset structure: source, summary, README, resources, updates, outputs, and logs. · `Data/DATASET-TEMPLATE.md`
- [US GDP](https://github.com/danielmiessler/substrate/tree/main/Data/US-GDP): economic dataset · Historical and quarterly GDP publications with markdown analysis and CSV outputs. · `Data/US-GDP`
- [US Inflation](https://github.com/danielmiessler/substrate/tree/main/Data/US-Inflation): inflation dataset · CPI time series with supporting documentation and update history. · `Data/US-Inflation`
- [Approval polls](https://github.com/danielmiessler/substrate/tree/main/Data/US-Presidential-Approval): political polling dataset · Historical and current presidential approval data with analysis documents and derived CSVs. · `Data/US-Presidential-Approval`
- [Wastewater](https://github.com/danielmiessler/substrate/tree/main/Data/Bay-Area-COVID-Wastewater): public health pipeline · Bay Area COVID wastewater dataset with dashboard, scrape/update commands, and published outputs. · `Data/Bay-Area-COVID-Wastewater`

### Source connectors

External source-specific ingestion and normalization boundaries.

- [Sources](https://github.com/danielmiessler/substrate/tree/main/Data/sources): ingestion catalog · Connector catalog for external authoritative data providers and dataset-specific ETL scripts. · `Data/sources`
- **update.ts scripts**: ETL automation · TypeScript/Bun update scripts that fetch, normalize, and write repo-local artifacts.
- **External sources**: public APIs · Upstream providers such as WHO, UN, World Bank, FRED, CDC WONDER, Census ACS, BLS JOLTS, EPA AQS, EIA, and Treasury FiscalData.

### Research layer

Analytical work products derived from the data layer.

- [Research](https://github.com/danielmiessler/substrate/blob/main/research/README.md): analysis layer · Downstream analytical work products that synthesize datasets into findings and conclusions. · `research/README.md`
- **Methods**: research methodology · Research project methodology, sources, and analytical framing.
- **Findings**: syntheses · Executive summaries, findings, and argument documents derived from the data layer.

### Automation & governance

Scripts, workflows, and contribution controls.

- **Validation**: checks and logs · Validation notes, update logs, and dataset-specific QA artifacts.
- [GitHub Actions](https://github.com/danielmiessler/substrate/blob/main/.github/workflows/claude.yml): workflow automation · Repository workflows for review, checks, and AI-assisted maintenance. · `.github/workflows/claude.yml`
- [CLAUDE.md](https://github.com/danielmiessler/substrate/blob/main/CLAUDE.md): editorial policy · Maintenance and review guidance for human plus AI contribution workflows. · `CLAUDE.md`
- [Git history](https://github.com/danielmiessler/substrate/blob/main/.gitignore): versioned storage · Git is the durability and publication layer for all repository artifacts. · `.gitignore`

### Other components

- [Substrate](https://github.com/danielmiessler/substrate/blob/main/README.md): knowledge repo · Repository-wide knowledge infrastructure centered on linked content, datasets, and lightweight automation. · `README.md`

## Connections

- Substrate → Entity folders: organizes · Repository root is structured around semantic knowledge folders.
- Substrate → Data: contains · Root includes the curated evidence store.
- Substrate → Sources: contains · Root includes the source connector catalog.
- Substrate → Research: contains · Root includes downstream analytical work.
- Entity folders → Cross-linking: uses · Knowledge categories are connected by explicit links rather than database relations.
- Entity folders → Entity docs: documented by · Each conceptual entity is documented through markdown files.
- Data → Dataset pattern: standardized by · Datasets follow a common publication contract.
- Data → US GDP: includes · Major curated dataset example.
- Data → US Inflation: includes · Major curated dataset example.
- Data → Approval polls: includes · Major curated dataset example.
- Data → Wastewater: includes · Pipeline-style dataset with dashboard and derived outputs.
- Sources → update.ts scripts: contains · Each source directory pairs documentation with update automation.
- update.ts scripts → External sources: pulls from · Scripts fetch from upstream public data providers.
- update.ts scripts → Data: writes to · Normalization outputs land in repo-local datasets.
- update.ts scripts → Validation: emits · Update scripts produce logs and validation artifacts.
- GitHub Actions → update.ts scripts: runs · Workflows automate review or execution of repo maintenance tasks.
- CLAUDE.md → GitHub Actions: guides · AI-assisted maintenance is governed by repo guidance and workflows.
- Git history → Data: stores · Version control is the system of record for dataset artifacts.
- Git history → Research: stores · Research outputs are also versioned in Git.
- Research → Methods: contains · Research projects include methodology and source framing.
- Research → Findings: contains · Research projects culminate in findings and synthesized conclusions.
- Research → Data: analyzes · Research consumes curated datasets as evidence.