disabandhu/developer-lead-pipeline

1. Repository shape: This is a Python batch ETL pipeline, not a web app. The core job is to discover developers, enrich their public profiles from multiple sources, score them, and export a structured dataset.

2. Primary orchestrator: main.py is the likely top-level runner for the pipeline. bulk_enrichment.py suggests a batch mode for processing many developers, while developers.csv, clean_developers.csv, and clean_developers.json are the main dataset artifacts.

3. Discovery layer: dev_api.py connects to the DEV.to API, which is the starting source of developer identities. username_collector.py appears to normalize or derive usernames so downstream enrichment can join records across platforms.

4. Profile collection layer: profile_collector.py is the first transformation stage after discovery. Based on the README, it gathers identity fields such as name, summary, location, website, and social links into a structured internal record.

5. Website enrichment boundary: website_enrichment.py drives analysis of personal websites, and website_parser.py extracts signals from page content. This stage is responsible for finding emails, LinkedIn/GitHub URLs, and other social links from portfolio sites.

6. Website classification: website_classifier.py likely separates personal/portfolio sites from other kinds of websites so the parser only spends effort on relevant pages. This is important for precision because the pipeline is trying to enrich developer profiles, not crawl the whole web.

7. External enrichment sources: The README says GitHub API data is part of the pipeline even though there is no dedicated file named for it. In practice, GitHub enrichment is a distinct external-data boundary that adds bio, company, followers, following, repository count, and account age to the profile.

8. Scoring and ranking: lead_scoring.py converts profile completeness into a quality score. The score is clearly additive and feature-based in the README example: email, LinkedIn, GitHub, and website each contribute to a lead value.

9. Persistence and dataset output: database.py suggests there is a storage abstraction for intermediate or future database-backed persistence, but the current canonical outputs are CSV and JSON. The cleaned dataset appears to be clean_developers.csv / clean_developers.json, with developers.csv as the primary exported table described in the README.

10. Metrics and inspection tools: enrichment_stats.py, quick_stats.py, and website_stats.py are support scripts for pipeline visibility. They likely measure coverage and quality across enrichment stages, which is useful because this repo is actively scaling and tuning the pipeline.

11. Validation and fixtures: test_enrichment.py and test_articles.py provide checks around parsing/enrichment logic, while sample_personal_sites.py looks like a fixture or sample-input generator. These files indicate the project is script-heavy but still has some automated verification around extraction behavior.

12. Important data flow boundary: The main architectural boundary is between raw discovered profiles and clean enriched developer records. Each stage adds more attributes and confidence: DEV discovery → profile normalization → website extraction → GitHub enrichment → scoring → export, producing a searchable developer intelligence dataset.