How danielmiessler/substrate works
1. Repository role
- This is not an application server or product backend; it is a knowledge infrastructure repository.
- The main artifact types are Markdown documents, CSV datasets, and TypeScript update scripts.
- The architecture is centered on human-curated knowledge + automated data ingestion.
2. Top-level knowledge model
- The root directories like Problems/, Solutions/, Claims/, Arguments/, Questions/, Values/, Ideas/, Plans/, People/, Organizations/, Projects/, Frames/, Models/, Outcomes/, Risks/, Threats/, and Funding-Sources/ are the conceptual entities of the system.
- These folders are the repository’s semantic layer: each one holds structured, likely markdown-based entries representing a distinct knowledge type.
- The architecture depends on explicit cross-linking between these entity types rather than a database schema.
3. Data subsystem
- Data/ is the primary evidence store.
- It contains curated datasets such as US GDP, inflation, Pulitzer winners, Bay Area wastewater, US approval polling, common metrics, and knowledge-worker salary data.
- The design pattern is consistent: each dataset folder contains the raw/derived data files plus supporting docs like README.md, SUMMARY.md, source.md, RESOURCES.md, UPDATES.md, and sometimes update.log.
4. Source catalog / ingestion layer
- Data/sources/ is the connector layer for external authoritative sources.
- Each DS-xxxxx—SourceName/ directory maps to one upstream provider such as WHO, UN SDGs, World Bank, FRED, CDC WONDER, Census ACS, BLS JOLTS, EPA AQS, EIA, and Treasury FiscalData.
- These source folders pair documentation with update.ts, making them the repo’s ETL boundary: fetch from outside systems, normalize, and write repo-local artifacts.
5. Automation flow
- The repo uses TypeScript with Bun for update automation.
- The typical flow is: external API or public dataset → update.ts script → normalized CSV/JSON/Markdown outputs → logs/documentation.
- Some datasets also include helper executables or scripts, such as wastewater scraping/update commands, showing that ingestion can combine API pulls, scraping, and local transformation.
6. Documentation-first boundary
- Almost every content area has its own README.md or summary document, so documentation is not incidental; it is part of the system boundary.
- Dataset documentation captures provenance, methodology, limitations, and update cadence.
- The repo’s “library science” framing means each dataset is intended to be both human readable and machine parseable.
7. Research / analysis subsystem
- research/ contains deeper analytical work products rather than canonical datasets.
- These folders hold methodology, source lists, findings, syntheses, executive summaries, and topical analyses.
- This is a downstream consumer of the data layer: the flow is data collection → research analysis → written conclusions / findings.
8. Project-specific data pipelines
- Some datasets are self-contained mini-pipelines, especially in Data/Bay-Area-COVID-Wastewater/ and several Data/sources/DS-.../ directories.
- These often include generated dashboards, update logs, and “latest” outputs, indicating a pattern where raw source data is transformed into a stable published dataset inside the repo.
- The dashboard.html file in the wastewater dataset suggests some outputs are intended for direct viewing, not just storage.
9. Governance and contribution boundary
- The repository is structured for community contribution through GitHub, with GitHub Actions in .github/workflows/ providing automated checks or AI-assisted review.
- CLAUDE.md and the Claude workflow files indicate AI is part of the maintenance and review process.
- This means the architecture includes a human + AI editorial workflow, not just code automation.
10. Dependency and integration model
- There is little evidence of a traditional runtime backend, database service, or frontend app.
- Instead, the repo integrates with external public-data services and persists results as versioned files in Git.
- Git itself is the durability layer: version history, diffs, and logs are the primary system of record.
11. Data publication pattern
- Each major dataset/source follows the same publication contract: source description, methodology, extracted data, update history, and sometimes validation notes.
- This creates a repeatable architecture for adding new datasets without changing the overall repo structure.
- The resulting outputs are designed to be browsed directly from the file tree or consumed by scripts/AI tools.
12. How to draw the architecture
- A good diagram would show:
- Top-level knowledge categories as the core domain model.
- Data/sources as ingestion connectors to external agencies/APIs.
- Dataset folders as curated evidence repositories.
- Research/ as analytical outputs built from the datasets.
- GitHub Actions / Bun TypeScript scripts as the automation and validation layer.
- The central idea is a knowledge graph-like content system backed by source-backed datasets and lightweight automation, all stored in Git.
Main entry points: Data/README.md Data · research/README.md Research · README.md Substrate · .gitignore Git history
Components 20
Knowledge layer
Core semantic entities organized as linked markdown knowledge types.
- Entity folders · Top-level knowledge categories such as Problems, Solutions, Claims, Arguments, Questions, Values, Ideas, Plans, People, Organizations, Projects, Frames, Models, Outcomes, Risks, Threats, and Funding-Sources.
- Cross-linking · Explicit relationships between markdown entries across entity folders.
- Entity docs · Folder-level readmes and entry files that make each knowledge type both human readable and machine parseable.
Data layer
Curated datasets and published evidence repositories.
- Data · Primary curated data area containing published datasets, summaries, resources, and update history.
Data/README.md - Dataset pattern · Common per-dataset structure: source, summary, README, resources, updates, outputs, and logs.
Data/DATASET-TEMPLATE.md - US GDP · Historical and quarterly GDP publications with markdown analysis and CSV outputs.
Data/US-GDP - US Inflation · CPI time series with supporting documentation and update history.
Data/US-Inflation - Approval polls · Historical and current presidential approval data with analysis documents and derived CSVs.
Data/US-Presidential-Approval - Wastewater · Bay Area COVID wastewater dataset with dashboard, scrape/update commands, and published outputs.
Data/Bay-Area-COVID-Wastewater
Source connectors
External source-specific ingestion and normalization boundaries.
- Sources · Connector catalog for external authoritative data providers and dataset-specific ETL scripts.
Data/sources - update.ts scripts · TypeScript/Bun update scripts that fetch, normalize, and write repo-local artifacts.
- External sources · Upstream providers such as WHO, UN, World Bank, FRED, CDC WONDER, Census ACS, BLS JOLTS, EPA AQS, EIA, and Treasury FiscalData.
Research layer
Analytical work products derived from the data layer.
- Research · Downstream analytical work products that synthesize datasets into findings and conclusions.
research/README.md - Methods · Research project methodology, sources, and analytical framing.
- Findings · Executive summaries, findings, and argument documents derived from the data layer.
Automation & governance
Scripts, workflows, and contribution controls.
- Validation · Validation notes, update logs, and dataset-specific QA artifacts.
- GitHub Actions · Repository workflows for review, checks, and AI-assisted maintenance.
.github/workflows/claude.yml - CLAUDE.md · Maintenance and review guidance for human plus AI contribution workflows.
CLAUDE.md - Git history · Git is the durability and publication layer for all repository artifacts.
.gitignore
Other components
- Substrate · Repository-wide knowledge infrastructure centered on linked content, datasets, and lightweight automation.
README.md
Connections 22
- Substrate → Entity folders: organizes · Repository root is structured around semantic knowledge folders.
- Substrate → Data: contains · Root includes the curated evidence store.
- Substrate → Sources: contains · Root includes the source connector catalog.
- Substrate → Research: contains · Root includes downstream analytical work.
- Entity folders → Cross-linking: uses · Knowledge categories are connected by explicit links rather than database relations.
- Entity folders → Entity docs: documented by · Each conceptual entity is documented through markdown files.
- Data → Dataset pattern: standardized by · Datasets follow a common publication contract.
- Data → US GDP: includes · Major curated dataset example.
- Data → US Inflation: includes · Major curated dataset example.
- Data → Approval polls: includes · Major curated dataset example.
- Data → Wastewater: includes · Pipeline-style dataset with dashboard and derived outputs.
- Sources → update.ts scripts: contains · Each source directory pairs documentation with update automation.
- update.ts scripts → External sources: pulls from · Scripts fetch from upstream public data providers.
- update.ts scripts → Data: writes to · Normalization outputs land in repo-local datasets.
- update.ts scripts → Validation: emits · Update scripts produce logs and validation artifacts.
- GitHub Actions → update.ts scripts: runs · Workflows automate review or execution of repo maintenance tasks.
- CLAUDE.md → GitHub Actions: guides · AI-assisted maintenance is governed by repo guidance and workflows.
- Git history → Data: stores · Version control is the system of record for dataset artifacts.
- Git history → Research: stores · Research outputs are also versioned in Git.
- Research → Methods: contains · Research projects include methodology and source framing.
- Research → Findings: contains · Research projects culminate in findings and synthesized conclusions.
- Research → Data: analyzes · Research consumes curated datasets as evidence.