How disabandhu/github-developer-discovery works
1. Repository purpose
- This is a Python data pipeline centered on GitHub developer discovery: crawl GitHub user/profile data, enrich it with email discovery, then export the results for analysis.
2. Primary execution model
- The core runtime is a Scrapy project under github_scraper/github_scraper/.
- The flow is crawler-first: a spider collects raw profile records, Scrapy item/pipeline components normalize them, and helper scripts perform enrichment/export steps.
3. Crawler subsystem
- github_scraper/github_scraper/spiders/github_users.py is the main crawl entry point.
- Based on the README, it likely queries GitHub user search results and expands across multiple tech-related queries to build a broader developer dataset.
- This subsystem is responsible for extracting profile-level fields such as name, username, bio, company, and location.
4. Scrapy project boundaries
- items.py defines the structured record passed through the pipeline.
- settings.py controls crawl behavior and project-level Scrapy configuration.
- middlewares.py provides request/response interception points, typically for header, throttle, or spider behavior customization.
- pipelines.py is the handoff boundary where scraped items are cleaned, validated, deduplicated, or prepared for persistence.
5. Data enrichment subsystem
- find_email.py is a separate enrichment script outside the Scrapy package.
- It likely uses requests and regex to inspect profile pages and README content for email addresses.
- This is an important architectural boundary: scraping collects profile metadata, while this script performs secondary discovery on already-collected users.
6. Export / transformation subsystem
- jsontocsv.py converts JSON output into CSV for downstream consumption.
- The README indicates JSON is the intermediate system-of-record, with CSV as a presentation/export format.
- This suggests the pipeline is file-based rather than database-backed.
7. Data flow
- Raw GitHub search/profile pages → spider extraction → Scrapy item/pipeline processing → JSON output.
- JSON records → email discovery/enrichment → users_with_emails.json.
- Enriched JSON → CSV conversion → users_with_emails.csv.
8. Output artifacts
- The repository explicitly produces users.json, users_with_emails.json, and users_with_emails.csv.
- These files represent three stages of fidelity: raw scraped data, enriched data, and tabular export.
9. External dependencies and their architectural role
- Scrapy is the main crawling framework and event-driven processing engine.
- Requests appears to support direct HTTP retrieval in the email discovery step.
- Regex is used for pattern-based extraction, especially for emails.
- GitHub is the external data source and operational dependency.
10. What is not present
- There is no evidence of a database, message queue, or web service layer.
- The architecture is a local/offline ETL-style pipeline with file-based interchange between stages.
- venv/ is just the Python environment and not part of the application design.
11. How to draw the architecture
- Represent three main boxes: Scrapy crawler, email enrichment script, and JSON/CSV export utilities.
- Show GitHub as the external source feeding the crawler, then a file-based handoff to the enrichment and conversion steps.
- The clean boundary is between crawl-time collection and post-processing enrichment/export.
Main entry points: find_email.py Email finder · github_users.py User spider · items.py Items · middlewares.py Middleware
Components 14
External source
Upstream data source used by the pipeline.
- GitHub · External profile and search data source.
Scrapy crawler
Core crawling project that discovers and normalizes GitHub developer profiles.
- User spider · Main Scrapy spider that searches GitHub users and extracts profile fields.
github_scraper/github_scraper/spiders/github_users.py - Items · Structured record definition passed through Scrapy processing.
github_scraper/github_scraper/items.py - Middleware · Request and response hooks for crawler behavior customization.
github_scraper/github_scraper/middlewares.py - Pipeline · Item cleaning, validation, deduplication, and persistence boundary.
github_scraper/github_scraper/pipelines.py - Settings · Scrapy project configuration and crawl behavior controls.
github_scraper/github_scraper/settings.py - Scrapy cfg · Project-level Scrapy configuration entry point.
github_scraper/scrapy.cfg - users.json · Primary JSON output from the crawl stage.
Post-processing
Local enrichment and export steps that operate on harvested records.
- Email finder · Secondary script that discovers email addresses from profile pages and related content.
github_scraper/find_email.py - Requests · Direct HTTP retrieval used during email discovery.
- Regex · Pattern matching used to identify email addresses.
- Emails JSON · Enriched JSON containing discovered email addresses.
- JSON to CSV · Converts enriched JSON into tabular CSV.
github_scraper/jsontocsv.py - users_with_emails.csv · Final CSV export for downstream analysis.
github_scraper/users_with_emails.csv
Connections 13
- GitHub → User spider: search/profile pages · GitHub content is crawled by the spider.
- User spider → Items: extracts · Spider populates structured profile records.
- Items → Middleware: passes through · Scrapy request and item flow hooks may adjust processing.
- Middleware → Pipeline: handoff · Processed items continue into the pipeline layer.
- Settings → User spider: controls · Project settings configure crawl behavior.
- Scrapy cfg → Settings: loads · Scrapy CLI bootstrap uses project configuration.
- Pipeline → users.json: writes · Cleaned crawl output is persisted as JSON.
- users.json → Email finder: input · Raw JSON feeds the email discovery step.
- Email finder → Requests: fetches pages · The enrichment script retrieves profile content over HTTP.
- Email finder → Regex: matches emails · Regex is used to detect email patterns.
- Email finder → Emails JSON: writes · Enriched user records are stored as JSON.
- Emails JSON → JSON to CSV: input · Enriched JSON is the source for CSV conversion.
- JSON to CSV → users_with_emails.csv: writes · Final tabular export is produced.