How danielmiessler/robotsdisallowed works
1. What this repository is
- This is not an application or service; it is a static dataset plus generation pipeline for collecting and refining robots.txt disallowed paths from high-traffic websites.
- The main architectural idea is: crawl public robots.txt files → extract Disallow entries → clean and normalize them → publish usable wordlists for recon/security work.
2. Primary data sources
- The source population is a top-website ranking list. The repo history says it started with Alexa Top 100K and later switched to Majestic Top 100K.
- The archived top-1m.csv and related files in archive/code are part of that input layer, used to drive which sites get queried.
3. Collection stage
- archive/code/pullrobots.sh is the collection step: it fetches robots.txt files from the ranked site list.
- The README notes a shift to Chromium as the user agent in the 2019 refresh, so the crawler behavior changed over time even though the repository is now mostly the resulting artifacts.
4. Extraction stage
- archive/code/extractonly.sh represents the parsing boundary: it takes downloaded robots.txt content and pulls out the Disallow paths.
- This is the point where raw robot files become a candidate directory/path corpus.
5. Cleanup and normalization stage
- archive/code/cleanuponly.sh is the curation boundary: the extracted paths are “a mess” and are cleaned up to be more useful.
- This likely includes normalization, deduplication, removal of malformed entries, and other text-level filtering. The important architectural point is that the published lists are curated artifacts, not raw scrape output.
6. Shell-based processing flow
- The helper scripts in archive/code show a simple Unix pipeline architecture rather than a compiled app:
- xargsonly.sh suggests batch processing over many sites/files.
- tocomma.csv suggests conversion into a comma-separated form.
- top-1m.csv is the ranked input set.
- The repo’s processing model is therefore batch, offline, and text-centric.
7. Published outputs
- The root-level files top10.txt, top100.txt, top1000.txt, top10000.txt, curated.txt, and the archived Top*-RobotsDisallowed.txt files are the deliverables.
- These are tiered by size/scope, so the consumer chooses a list based on assessment depth and time available.
8. Curated subset
- curated.txt is the highest-signal output: the README says it contains entries matching terms like admin, user, account, password, forgot, login, backup, plus the top 25 entries from the full list.
- This is an intentional high-value subset optimized for discovery work, not a comprehensive dump.
9. Archive boundary and versioning
- The archive/ directory preserves older snapshots and the older code path, including the 2019-era update.
- This repository therefore has a current canonical output set at the root and a historical record in archive/, which matters if you’re diagramming lineage or reproducibility.
10. What is intentionally not included
- The repo does not keep the full robots.txt corpus anymore; the README explicitly says those files were removed because they were large and low-value.
- That means the architecture is centered on derived intelligence products, not long-term storage of source crawls.
11. Operational context
- The intended consumer is a security assessor or bug bounty hunter using the lists offline as reconnaissance input.
- There are no runtime services, databases, or APIs here; the whole system is a data harvesting and curation pipeline that ends in downloadable text lists.
Main entry points: archive Archive set · cleanuponly.sh Clean paths · pullrobots.sh Fetch robots · README.md README
Components 21
Current outputs
Canonical downloadable wordlists at the repository root.
Pipeline
Shell-based collection, extraction, and cleanup flow.
- Fetch robots · Downloads robots.txt files from ranked sites.
archive/code/pullrobots.sh - Extract paths · Pulls Disallow entries from robots content.
archive/code/extractonly.sh - Clean paths · Normalizes, deduplicates, and filters extracted paths.
archive/code/cleanuponly.sh - Batch runner · Fan-out shell processing over many files or sites.
archive/code/xargsonly.sh - CSV convert · Converts intermediate output into comma-separated form.
archive/code/tocomma.csv - Rank list · Top-1M site list used to choose collection targets.
archive/code/top-1m.csv
Archive
Historical snapshots and older generated lists.
- Archive set · Legacy outputs and older source snapshots kept under archive/.
archive - Top 10 old · Archived generated list.
archive/Top10-RobotsDisallowed.txt - Top 100 old · Archived generated list.
archive/Top100-RobotsDisallowed.txt - Top 1K old · Archived generated list.
archive/Top1000-RobotsDisallowed.txt - Top 10K old · Archived generated list.
archive/Top10000-RobotsDisallowed.txt - Top 100K old · Archived generated list.
archive/Top100000-RobotsDisallowed.txt - Top 500 old · Archived generated list.
archive/Top500-RobotsDisallowed.txt - Interesting dirs · Older curated directory list.
archive/InterestingDirectories.txt - 2019 raw · Archived raw collection snapshot from the 2019 refresh.
archive/raw_2019.txt
Other components
- README · Project overview, history, and usage notes.
README.md
Connections 21
- README → Top 10: documents · Explains the published tiers.
- README → Curated: describes · Documents the high-signal subset.
- Rank list → Fetch robots: drives · Ranking input selects targets for collection.
- CSV convert → Rank list: formats · Helper conversion around the ranked input list.
- Batch runner → Fetch robots: batches · Fan-out execution for collection.
- Fetch robots → Extract paths: feeds · Collected robots.txt content is parsed next.
- Extract paths → Clean paths: cleans · Raw Disallow paths become curated candidates.
- Clean paths → Curated: publishes · Curated output is derived from the cleaned corpus.
- Clean paths → Top 10: publishes · One of the tiered release lists.
- Clean paths → Top 100: publishes · One of the tiered release lists.
- Clean paths → Top 1K: publishes · One of the tiered release lists.
- Clean paths → Top 10K: publishes · One of the tiered release lists.
- Fetch robots → 2019 raw: historical source · Archived raw collection snapshot from the 2019 refresh.
- Archive set → Top 10 old: contains · Historical generated outputs.
- Archive set → Top 100 old: contains · Historical generated outputs.
- Archive set → Top 1K old: contains · Historical generated outputs.
- Archive set → Top 10K old: contains · Historical generated outputs.
- Archive set → Top 100K old: contains · Historical generated outputs.
- Archive set → Top 500 old: contains · Historical generated outputs.
- Archive set → Interesting dirs: contains · Older curated directory list.
- Archive set → 2019 raw: contains · Archived raw collection snapshot.