# danielmiessler/robotsdisallowed architecture

> Architecture diagram of the GitHub repository danielmiessler/robotsdisallowed, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/danielmiessler/robotsdisallowed
- Repository: https://github.com/danielmiessler/robotsdisallowed

## Overview

1. **What this repository is**
   - This is not an application or service; it is a **static dataset plus generation pipeline** for collecting and refining `robots.txt` disallowed paths from high-traffic websites.
   - The main architectural idea is: **crawl public robots.txt files → extract `Disallow` entries → clean and normalize them → publish usable wordlists** for recon/security work.

2. **Primary data sources**
   - The source population is a **top-website ranking list**. The repo history says it started with Alexa Top 100K and later switched to **Majestic Top 100K**.
   - The archived `top-1m.csv` and related files in `archive/code` are part of that input layer, used to drive which sites get queried.

3. **Collection stage**
   - `archive/code/pullrobots.sh` is the collection step: it fetches robots.txt files from the ranked site list.
   - The README notes a shift to **Chromium as the user agent** in the 2019 refresh, so the crawler behavior changed over time even though the repository is now mostly the resulting artifacts.

4. **Extraction stage**
   - `archive/code/extractonly.sh` represents the parsing boundary: it takes downloaded robots.txt content and pulls out the `Disallow` paths.
   - This is the point where raw robot files become a candidate directory/path corpus.

5. **Cleanup and normalization stage**
   - `archive/code/cleanuponly.sh` is the curation boundary: the extracted paths are “a mess” and are cleaned up to be more useful.
   - This likely includes normalization, deduplication, removal of malformed entries, and other text-level filtering. The important architectural point is that **the published lists are curated artifacts, not raw scrape output**.

6. **Shell-based processing flow**
   - The helper scripts in `archive/code` show a simple Unix pipeline architecture rather than a compiled app:
     - `xargsonly.sh` suggests batch processing over many sites/files.
     - `tocomma.csv` suggests conversion into a comma-separated form.
     - `top-1m.csv` is the ranked input set.
   - The repo’s processing model is therefore **batch, offline, and text-centric**.

7. **Published outputs**
   - The root-level files `top10.txt`, `top100.txt`, `top1000.txt`, `top10000.txt`, `curated.txt`, and the archived `Top*-RobotsDisallowed.txt` files are the deliverables.
   - These are tiered by size/scope, so the consumer chooses a list based on assessment depth and time available.

8. **Curated subset**
   - `curated.txt` is the highest-signal output: the README says it contains entries matching terms like `admin`, `user`, `account`, `password`, `forgot`, `login`, `backup`, plus the top 25 entries from the full list.
   - This is an intentional **high-value subset** optimized for discovery work, not a comprehensive dump.

9. **Archive boundary and versioning**
   - The `archive/` directory preserves older snapshots and the older code path, including the 2019-era update.
   - This repository therefore has a **current canonical output set** at the root and a **historical record** in `archive/`, which matters if you’re diagramming lineage or reproducibility.

10. **What is intentionally not included**
   - The repo does **not** keep the full robots.txt corpus anymore; the README explicitly says those files were removed because they were large and low-value.
   - That means the architecture is centered on **derived intelligence products**, not long-term storage of source crawls.

11. **Operational context**
   - The intended consumer is a security assessor or bug bounty hunter using the lists offline as reconnaissance input.
   - There are no runtime services, databases, or APIs here; the whole system is a **data harvesting and curation pipeline** that ends in downloadable text lists.

Main entry points: [`archive`](https://github.com/danielmiessler/robotsdisallowed/tree/master/archive) (Archive set), [`archive/code/cleanuponly.sh`](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/cleanuponly.sh) (Clean paths), [`archive/code/pullrobots.sh`](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/pullrobots.sh) (Fetch robots), [`README.md`](https://github.com/danielmiessler/robotsdisallowed/blob/master/README.md) (README)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_root_outputs["Current outputs"]
  node_top10["Top 10<br/>wordlist<br/>[top10.txt]"]
  node_top100["Top 100<br/>wordlist<br/>[top100.txt]"]
  node_top1000["Top 1K<br/>wordlist<br/>[top1000.txt]"]
  node_top10000["Top 10K<br/>wordlist<br/>[top10000.txt]"]
  node_curated["Curated<br/>high-signal wordlist<br/>[curated.txt]"]
end

subgraph group_pipeline["Pipeline"]
  node_pullrobots["Fetch robots<br/>collector script<br/>[pullrobots.sh]"]
  node_extractonly["Extract paths<br/>parser script<br/>[extractonly.sh]"]
  node_cleanuponly["Clean paths<br/>curation script<br/>[cleanuponly.sh]"]
  node_xargsonly["Batch runner<br/>batch helper<br/>[xargsonly.sh]"]
  node_tocomma["CSV convert<br/>transform helper<br/>[tocomma.csv]"]
  node_top1m["Rank list<br/>site ranking input<br/>[top-1m.csv]"]
end

subgraph group_archive["Archive"]
  node_archive_bundle["Archive set<br/>historical corpus"]
  node_arch10["Top 10 old<br/>legacy wordlist"]
  node_arch100["Top 100 old<br/>legacy wordlist"]
  node_arch1k["Top 1K old<br/>legacy wordlist"]
  node_arch10k["Top 10K old<br/>legacy wordlist"]
  node_arch100k["Top 100K old<br/>legacy wordlist"]
  node_arch500["Top 500 old<br/>legacy wordlist"]
  node_interesting["Interesting dirs<br/>legacy subset"]
  node_raw2019["2019 raw<br/>legacy source snapshot<br/>[raw_2019.txt]"]
end

node_readme["README<br/>docs<br/>[README.md]"]

node_readme -->|"documents"| node_top10
node_readme -->|"describes"| node_curated
node_top1m -->|"drives"| node_pullrobots
node_tocomma -.->|"formats"| node_top1m
node_xargsonly -.->|"batches"| node_pullrobots
node_pullrobots -->|"feeds"| node_extractonly
node_extractonly -->|"cleans"| node_cleanuponly
node_cleanuponly -->|"publishes"| node_curated
node_cleanuponly -->|"publishes"| node_top10
node_cleanuponly -->|"publishes"| node_top100
node_cleanuponly -->|"publishes"| node_top1000
node_cleanuponly -->|"publishes"| node_top10000
node_pullrobots -.->|"historical source"| node_raw2019
node_archive_bundle -->|"contains"| node_arch10
node_archive_bundle -->|"contains"| node_arch100
node_archive_bundle -->|"contains"| node_arch1k
node_archive_bundle -->|"contains"| node_arch10k
node_archive_bundle -->|"contains"| node_arch100k
node_archive_bundle -->|"contains"| node_arch500
node_archive_bundle -->|"contains"| node_interesting
node_archive_bundle -->|"contains"| node_raw2019

click node_readme "https://github.com/danielmiessler/robotsdisallowed/blob/master/README.md"
click node_top10 "https://github.com/danielmiessler/robotsdisallowed/blob/master/top10.txt"
click node_top100 "https://github.com/danielmiessler/robotsdisallowed/blob/master/top100.txt"
click node_top1000 "https://github.com/danielmiessler/robotsdisallowed/blob/master/top1000.txt"
click node_top10000 "https://github.com/danielmiessler/robotsdisallowed/blob/master/top10000.txt"
click node_curated "https://github.com/danielmiessler/robotsdisallowed/blob/master/curated.txt"
click node_pullrobots "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/pullrobots.sh"
click node_extractonly "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/extractonly.sh"
click node_cleanuponly "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/cleanuponly.sh"
click node_xargsonly "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/xargsonly.sh"
click node_tocomma "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/tocomma.csv"
click node_top1m "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/top-1m.csv"
click node_archive_bundle "https://github.com/danielmiessler/robotsdisallowed/tree/master/archive"
click node_arch10 "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top10-RobotsDisallowed.txt"
click node_arch100 "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top100-RobotsDisallowed.txt"
click node_arch1k "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top1000-RobotsDisallowed.txt"
click node_arch10k "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top10000-RobotsDisallowed.txt"
click node_arch100k "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top100000-RobotsDisallowed.txt"
click node_arch500 "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top500-RobotsDisallowed.txt"
click node_interesting "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/InterestingDirectories.txt"
click node_raw2019 "https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/raw_2019.txt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_top10,node_top100,node_top1000,node_top10000,node_curated toneBlue
class node_pullrobots,node_extractonly,node_cleanuponly,node_xargsonly,node_tocomma,node_top1m toneAmber
class node_archive_bundle,node_arch10,node_arch100,node_arch1k,node_arch10k,node_arch100k,node_arch500,node_interesting,node_raw2019 toneMint
class node_readme toneNeutral
```

## Components

### Current outputs

Canonical downloadable wordlists at the repository root.

- [Top 10](https://github.com/danielmiessler/robotsdisallowed/blob/master/top10.txt): wordlist · Smallest published list. · `top10.txt`
- [Top 100](https://github.com/danielmiessler/robotsdisallowed/blob/master/top100.txt): wordlist · Tiered output for quick use. · `top100.txt`
- [Top 1K](https://github.com/danielmiessler/robotsdisallowed/blob/master/top1000.txt): wordlist · Mid-size published list. · `top1000.txt`
- [Top 10K](https://github.com/danielmiessler/robotsdisallowed/blob/master/top10000.txt): wordlist · Large published list. · `top10000.txt`
- [Curated](https://github.com/danielmiessler/robotsdisallowed/blob/master/curated.txt): high-signal wordlist · Filtered set of high-value paths. · `curated.txt`

### Pipeline

Shell-based collection, extraction, and cleanup flow.

- [Fetch robots](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/pullrobots.sh): collector script · Downloads robots.txt files from ranked sites. · `archive/code/pullrobots.sh`
- [Extract paths](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/extractonly.sh): parser script · Pulls Disallow entries from robots content. · `archive/code/extractonly.sh`
- [Clean paths](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/cleanuponly.sh): curation script · Normalizes, deduplicates, and filters extracted paths. · `archive/code/cleanuponly.sh`
- [Batch runner](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/xargsonly.sh): batch helper · Fan-out shell processing over many files or sites. · `archive/code/xargsonly.sh`
- [CSV convert](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/tocomma.csv): transform helper · Converts intermediate output into comma-separated form. · `archive/code/tocomma.csv`
- [Rank list](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/code/top-1m.csv): site ranking input · Top-1M site list used to choose collection targets. · `archive/code/top-1m.csv`

### Archive

Historical snapshots and older generated lists.

- [Archive set](https://github.com/danielmiessler/robotsdisallowed/tree/master/archive): historical corpus · Legacy outputs and older source snapshots kept under archive/. · `archive`
- [Top 10 old](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top10-RobotsDisallowed.txt): legacy wordlist · Archived generated list. · `archive/Top10-RobotsDisallowed.txt`
- [Top 100 old](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top100-RobotsDisallowed.txt): legacy wordlist · Archived generated list. · `archive/Top100-RobotsDisallowed.txt`
- [Top 1K old](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top1000-RobotsDisallowed.txt): legacy wordlist · Archived generated list. · `archive/Top1000-RobotsDisallowed.txt`
- [Top 10K old](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top10000-RobotsDisallowed.txt): legacy wordlist · Archived generated list. · `archive/Top10000-RobotsDisallowed.txt`
- [Top 100K old](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top100000-RobotsDisallowed.txt): legacy wordlist · Archived generated list. · `archive/Top100000-RobotsDisallowed.txt`
- [Top 500 old](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/Top500-RobotsDisallowed.txt): legacy wordlist · Archived generated list. · `archive/Top500-RobotsDisallowed.txt`
- [Interesting dirs](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/InterestingDirectories.txt): legacy subset · Older curated directory list. · `archive/InterestingDirectories.txt`
- [2019 raw](https://github.com/danielmiessler/robotsdisallowed/blob/master/archive/raw_2019.txt): legacy source snapshot · Archived raw collection snapshot from the 2019 refresh. · `archive/raw_2019.txt`

### Other components

- [README](https://github.com/danielmiessler/robotsdisallowed/blob/master/README.md): docs · Project overview, history, and usage notes. · `README.md`

## Connections

- README → Top 10: documents · Explains the published tiers.
- README → Curated: describes · Documents the high-signal subset.
- Rank list → Fetch robots: drives · Ranking input selects targets for collection.
- CSV convert → Rank list: formats · Helper conversion around the ranked input list.
- Batch runner → Fetch robots: batches · Fan-out execution for collection.
- Fetch robots → Extract paths: feeds · Collected robots.txt content is parsed next.
- Extract paths → Clean paths: cleans · Raw Disallow paths become curated candidates.
- Clean paths → Curated: publishes · Curated output is derived from the cleaned corpus.
- Clean paths → Top 10: publishes · One of the tiered release lists.
- Clean paths → Top 100: publishes · One of the tiered release lists.
- Clean paths → Top 1K: publishes · One of the tiered release lists.
- Clean paths → Top 10K: publishes · One of the tiered release lists.
- Fetch robots → 2019 raw: historical source · Archived raw collection snapshot from the 2019 refresh.
- Archive set → Top 10 old: contains · Historical generated outputs.
- Archive set → Top 100 old: contains · Historical generated outputs.
- Archive set → Top 1K old: contains · Historical generated outputs.
- Archive set → Top 10K old: contains · Historical generated outputs.
- Archive set → Top 100K old: contains · Historical generated outputs.
- Archive set → Top 500 old: contains · Historical generated outputs.
- Archive set → Interesting dirs: contains · Older curated directory list.
- Archive set → 2019 raw: contains · Archived raw collection snapshot.
