# disabandhu/github-developer-discovery architecture

> Architecture diagram of the GitHub repository disabandhu/github-developer-discovery, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/disabandhu/github-developer-discovery
- Repository: https://github.com/disabandhu/github-developer-discovery

## Overview

1. **Repository purpose**
   - This is a Python data pipeline centered on GitHub developer discovery: crawl GitHub user/profile data, enrich it with email discovery, then export the results for analysis.

2. **Primary execution model**
   - The core runtime is a **Scrapy** project under `github_scraper/github_scraper/`.
   - The flow is crawler-first: a spider collects raw profile records, Scrapy item/pipeline components normalize them, and helper scripts perform enrichment/export steps.

3. **Crawler subsystem**
   - `github_scraper/github_scraper/spiders/github_users.py` is the main crawl entry point.
   - Based on the README, it likely queries GitHub user search results and expands across multiple tech-related queries to build a broader developer dataset.
   - This subsystem is responsible for extracting profile-level fields such as name, username, bio, company, and location.

4. **Scrapy project boundaries**
   - `items.py` defines the structured record passed through the pipeline.
   - `settings.py` controls crawl behavior and project-level Scrapy configuration.
   - `middlewares.py` provides request/response interception points, typically for header, throttle, or spider behavior customization.
   - `pipelines.py` is the handoff boundary where scraped items are cleaned, validated, deduplicated, or prepared for persistence.

5. **Data enrichment subsystem**
   - `find_email.py` is a separate enrichment script outside the Scrapy package.
   - It likely uses **requests** and **regex** to inspect profile pages and README content for email addresses.
   - This is an important architectural boundary: scraping collects profile metadata, while this script performs secondary discovery on already-collected users.

6. **Export / transformation subsystem**
   - `jsontocsv.py` converts JSON output into CSV for downstream consumption.
   - The README indicates JSON is the intermediate system-of-record, with CSV as a presentation/export format.
   - This suggests the pipeline is file-based rather than database-backed.

7. **Data flow**
   - Raw GitHub search/profile pages → spider extraction → Scrapy item/pipeline processing → JSON output.
   - JSON records → email discovery/enrichment → `users_with_emails.json`.
   - Enriched JSON → CSV conversion → `users_with_emails.csv`.

8. **Output artifacts**
   - The repository explicitly produces `users.json`, `users_with_emails.json`, and `users_with_emails.csv`.
   - These files represent three stages of fidelity: raw scraped data, enriched data, and tabular export.

9. **External dependencies and their architectural role**
   - **Scrapy** is the main crawling framework and event-driven processing engine.
   - **Requests** appears to support direct HTTP retrieval in the email discovery step.
   - **Regex** is used for pattern-based extraction, especially for emails.
   - GitHub is the external data source and operational dependency.

10. **What is not present**
   - There is no evidence of a database, message queue, or web service layer.
   - The architecture is a local/offline ETL-style pipeline with file-based interchange between stages.
   - `venv/` is just the Python environment and not part of the application design.

11. **How to draw the architecture**
   - Represent three main boxes: **Scrapy crawler**, **email enrichment script**, and **JSON/CSV export utilities**.
   - Show GitHub as the external source feeding the crawler, then a file-based handoff to the enrichment and conversion steps.
   - The clean boundary is between crawl-time collection and post-processing enrichment/export.

Main entry points: [`github_scraper/find_email.py`](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/find_email.py) (Email finder), [`github_scraper/github_scraper/spiders/github_users.py`](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/spiders/github_users.py) (User spider), [`github_scraper/github_scraper/items.py`](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/items.py) (Items), [`github_scraper/github_scraper/middlewares.py`](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/middlewares.py) (Middleware)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_external["External source"]
  node_github(("GitHub<br/>data source"))
end

subgraph group_crawler["Scrapy crawler"]
  node_spider["User spider<br/>crawler<br/>[github_users.py]"]
  node_items["Items<br/>schema<br/>[items.py]"]
  node_middleware["Middleware<br/>interception<br/>[middlewares.py]"]
  node_pipelines["Pipeline<br/>processing<br/>[pipelines.py]"]
  node_settings["Settings<br/>config<br/>[settings.py]"]
  node_scrapycfg["Scrapy cfg<br/>launcher config<br/>[scrapy.cfg]"]
  node_usersjson[("users.json<br/>raw output")]
end

subgraph group_post["Post-processing"]
  node_enrich["Email finder<br/>enrichment script<br/>[find_email.py]"]
  node_requests(("Requests<br/>http client"))
  node_regex(("Regex<br/>pattern extraction"))
  node_emailsjson[("Emails JSON<br/>enriched output")]
  node_csvtool["JSON to CSV<br/>transform script<br/>[jsontocsv.py]"]
  node_emailscsv[("users_with_emails.csv<br/>tabular export")]
end

node_github -->|"search/profile pages"| node_spider
node_spider -->|"extracts"| node_items
node_items -->|"passes through"| node_middleware
node_middleware -->|"handoff"| node_pipelines
node_settings -.->|"controls"| node_spider
node_scrapycfg -.->|"loads"| node_settings
node_pipelines -->|"writes"| node_usersjson
node_usersjson -->|"input"| node_enrich
node_enrich -->|"fetches pages"| node_requests
node_enrich -->|"matches emails"| node_regex
node_enrich -->|"writes"| node_emailsjson
node_emailsjson -->|"input"| node_csvtool
node_csvtool -->|"writes"| node_emailscsv

click node_spider "https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/spiders/github_users.py"
click node_items "https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/items.py"
click node_middleware "https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/middlewares.py"
click node_pipelines "https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/pipelines.py"
click node_settings "https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/settings.py"
click node_scrapycfg "https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/scrapy.cfg"
click node_enrich "https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/find_email.py"
click node_csvtool "https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/jsontocsv.py"
click node_emailscsv "https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/users_with_emails.csv"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_github toneBlue
class node_spider,node_items,node_middleware,node_pipelines,node_settings,node_scrapycfg,node_usersjson toneAmber
class node_enrich,node_requests,node_regex,node_emailsjson,node_csvtool,node_emailscsv toneMint
```

## Components

### External source

Upstream data source used by the pipeline.

- **GitHub**: data source · External profile and search data source.

### Scrapy crawler

Core crawling project that discovers and normalizes GitHub developer profiles.

- [User spider](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/spiders/github_users.py): crawler · Main Scrapy spider that searches GitHub users and extracts profile fields. · `github_scraper/github_scraper/spiders/github_users.py`
- [Items](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/items.py): schema · Structured record definition passed through Scrapy processing. · `github_scraper/github_scraper/items.py`
- [Middleware](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/middlewares.py): interception · Request and response hooks for crawler behavior customization. · `github_scraper/github_scraper/middlewares.py`
- [Pipeline](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/pipelines.py): processing · Item cleaning, validation, deduplication, and persistence boundary. · `github_scraper/github_scraper/pipelines.py`
- [Settings](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/github_scraper/settings.py): config · Scrapy project configuration and crawl behavior controls. · `github_scraper/github_scraper/settings.py`
- [Scrapy cfg](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/scrapy.cfg): launcher config · Project-level Scrapy configuration entry point. · `github_scraper/scrapy.cfg`
- **users.json**: raw output · Primary JSON output from the crawl stage.

### Post-processing

Local enrichment and export steps that operate on harvested records.

- [Email finder](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/find_email.py): enrichment script · Secondary script that discovers email addresses from profile pages and related content. · `github_scraper/find_email.py`
- **Requests**: http client · Direct HTTP retrieval used during email discovery.
- **Regex**: pattern extraction · Pattern matching used to identify email addresses.
- **Emails JSON**: enriched output · Enriched JSON containing discovered email addresses.
- [JSON to CSV](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/jsontocsv.py): transform script · Converts enriched JSON into tabular CSV. · `github_scraper/jsontocsv.py`
- [users_with_emails.csv](https://github.com/disabandhu/github-developer-discovery/blob/main/github_scraper/users_with_emails.csv): tabular export · Final CSV export for downstream analysis. · `github_scraper/users_with_emails.csv`

## Connections

- GitHub → User spider: search/profile pages · GitHub content is crawled by the spider.
- User spider → Items: extracts · Spider populates structured profile records.
- Items → Middleware: passes through · Scrapy request and item flow hooks may adjust processing.
- Middleware → Pipeline: handoff · Processed items continue into the pipeline layer.
- Settings → User spider: controls · Project settings configure crawl behavior.
- Scrapy cfg → Settings: loads · Scrapy CLI bootstrap uses project configuration.
- Pipeline → users.json: writes · Cleaned crawl output is persisted as JSON.
- users.json → Email finder: input · Raw JSON feeds the email discovery step.
- Email finder → Requests: fetches pages · The enrichment script retrieves profile content over HTTP.
- Email finder → Regex: matches emails · Regex is used to detect email patterns.
- Email finder → Emails JSON: writes · Enriched user records are stored as JSON.
- Emails JSON → JSON to CSV: input · Enriched JSON is the source for CSV conversion.
- JSON to CSV → users_with_emails.csv: writes · Final tabular export is produced.