# disabandhu/developer-lead-pipeline architecture

> Architecture diagram of the GitHub repository disabandhu/developer-lead-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/disabandhu/developer-lead-pipeline
- Repository: https://github.com/disabandhu/developer-lead-pipeline

## Overview

1. **Repository shape:** This is a Python batch ETL pipeline, not a web app. The core job is to discover developers, enrich their public profiles from multiple sources, score them, and export a structured dataset.

2. **Primary orchestrator:** `main.py` is the likely top-level runner for the pipeline. `bulk_enrichment.py` suggests a batch mode for processing many developers, while `developers.csv`, `clean_developers.csv`, and `clean_developers.json` are the main dataset artifacts.

3. **Discovery layer:** `dev_api.py` connects to the DEV.to API, which is the starting source of developer identities. `username_collector.py` appears to normalize or derive usernames so downstream enrichment can join records across platforms.

4. **Profile collection layer:** `profile_collector.py` is the first transformation stage after discovery. Based on the README, it gathers identity fields such as name, summary, location, website, and social links into a structured internal record.

5. **Website enrichment boundary:** `website_enrichment.py` drives analysis of personal websites, and `website_parser.py` extracts signals from page content. This stage is responsible for finding emails, LinkedIn/GitHub URLs, and other social links from portfolio sites.

6. **Website classification:** `website_classifier.py` likely separates personal/portfolio sites from other kinds of websites so the parser only spends effort on relevant pages. This is important for precision because the pipeline is trying to enrich developer profiles, not crawl the whole web.

7. **External enrichment sources:** The README says GitHub API data is part of the pipeline even though there is no dedicated file named for it. In practice, GitHub enrichment is a distinct external-data boundary that adds bio, company, followers, following, repository count, and account age to the profile.

8. **Scoring and ranking:** `lead_scoring.py` converts profile completeness into a quality score. The score is clearly additive and feature-based in the README example: email, LinkedIn, GitHub, and website each contribute to a lead value.

9. **Persistence and dataset output:** `database.py` suggests there is a storage abstraction for intermediate or future database-backed persistence, but the current canonical outputs are CSV and JSON. The cleaned dataset appears to be `clean_developers.csv` / `clean_developers.json`, with `developers.csv` as the primary exported table described in the README.

10. **Metrics and inspection tools:** `enrichment_stats.py`, `quick_stats.py`, and `website_stats.py` are support scripts for pipeline visibility. They likely measure coverage and quality across enrichment stages, which is useful because this repo is actively scaling and tuning the pipeline.

11. **Validation and fixtures:** `test_enrichment.py` and `test_articles.py` provide checks around parsing/enrichment logic, while `sample_personal_sites.py` looks like a fixture or sample-input generator. These files indicate the project is script-heavy but still has some automated verification around extraction behavior.

12. **Important data flow boundary:** The main architectural boundary is between **raw discovered profiles** and **clean enriched developer records**. Each stage adds more attributes and confidence: DEV discovery → profile normalization → website extraction → GitHub enrichment → scoring → export, producing a searchable developer intelligence dataset.

Main entry points: [`website_parser.py`](https://github.com/disabandhu/developer-lead-pipeline/blob/main/website_parser.py) (Page parser), [`developers.csv`](https://github.com/disabandhu/developer-lead-pipeline/blob/main/developers.csv) (Developers CSV), [`profile_collector.py`](https://github.com/disabandhu/developer-lead-pipeline/blob/main/profile_collector.py) (Profiles), [`website_enrichment.py`](https://github.com/disabandhu/developer-lead-pipeline/blob/main/website_enrichment.py) (Web enrich)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_main["Main<br/>runner<br/>[main.py]"]
  node_bulk_enrichment["Bulk run<br/>batch runner<br/>[bulk_enrichment.py]"]
end

subgraph group_discovery["Discovery"]
  node_dev_api["DEV API<br/>source connector<br/>[dev_api.py]"]
  node_username_collector["Usernames<br/>normalizer"]
  node_profile_collector["Profiles<br/>profile builder"]
end

subgraph group_enrichment["Enrichment"]
  node_website_classifier["Site classifier"]
  node_website_enrichment["Web enrich"]
  node_website_parser["Page parser<br/>[website_parser.py]"]
  node_github_enrichment["GitHub enrich<br/>external enrichment"]
  node_lead_scoring{{"Lead score<br/>scorer<br/>[lead_scoring.py]"}}
end

subgraph group_storage["Storage"]
  node_database[("Database<br/>storage layer<br/>[database.py]")]
  node_developers_csv["Developers CSV<br/>dataset export<br/>[developers.csv]"]
  node_clean_developers_csv["Clean CSV<br/>clean dataset"]
  node_clean_developers_json["Clean JSON<br/>clean dataset"]
end

subgraph group_observability["Metrics"]
  node_enrichment_stats["Enrich stats<br/>metrics"]
  node_quick_stats["Quick stats<br/>metrics<br/>[quick_stats.py]"]
  node_website_stats["Web stats<br/>metrics<br/>[website_stats.py]"]
end

subgraph group_validation["Validation"]
  node_test_enrichment["Enrich tests<br/>[test_enrichment.py]"]
  node_test_articles["Article tests<br/>[test_articles.py]"]
  node_sample_personal_sites["Site samples<br/>fixture"]
end

node_main -->|"start discovery"| node_dev_api
node_main -->|"run enrichment"| node_website_enrichment
node_bulk_enrichment -->|"batch profiles"| node_profile_collector
node_dev_api -->|"raw identities"| node_username_collector
node_username_collector -->|"joined users"| node_profile_collector
node_profile_collector -->|"site candidates"| node_website_classifier
node_website_classifier -->|"personal sites"| node_website_enrichment
node_website_enrichment -->|"page content"| node_website_parser
node_profile_collector -->|"profile seeds"| node_github_enrichment
node_website_parser -->|"site signals"| node_lead_scoring
node_github_enrichment -->|"github signals"| node_lead_scoring
node_lead_scoring -->|"persist"| node_database
node_database -->|"export table"| node_developers_csv
node_developers_csv -->|"cleaned"| node_clean_developers_csv
node_developers_csv -->|"serialized"| node_clean_developers_json
node_developers_csv -->|"coverage"| node_enrichment_stats
node_developers_csv -->|"summary"| node_quick_stats
node_website_parser -->|"site metrics"| node_website_stats
node_website_parser -.->|"verify parser"| node_test_enrichment
node_website_parser -.->|"verify content"| node_test_articles
node_sample_personal_sites -.->|"fixture input"| node_website_enrichment

click node_main "https://github.com/disabandhu/developer-lead-pipeline/blob/main/main.py"
click node_bulk_enrichment "https://github.com/disabandhu/developer-lead-pipeline/blob/main/bulk_enrichment.py"
click node_dev_api "https://github.com/disabandhu/developer-lead-pipeline/blob/main/dev_api.py"
click node_username_collector "https://github.com/disabandhu/developer-lead-pipeline/blob/main/username_collector.py"
click node_profile_collector "https://github.com/disabandhu/developer-lead-pipeline/blob/main/profile_collector.py"
click node_website_classifier "https://github.com/disabandhu/developer-lead-pipeline/blob/main/website_classifier.py"
click node_website_enrichment "https://github.com/disabandhu/developer-lead-pipeline/blob/main/website_enrichment.py"
click node_website_parser "https://github.com/disabandhu/developer-lead-pipeline/blob/main/website_parser.py"
click node_lead_scoring "https://github.com/disabandhu/developer-lead-pipeline/blob/main/lead_scoring.py"
click node_database "https://github.com/disabandhu/developer-lead-pipeline/blob/main/database.py"
click node_developers_csv "https://github.com/disabandhu/developer-lead-pipeline/blob/main/developers.csv"
click node_clean_developers_csv "https://github.com/disabandhu/developer-lead-pipeline/blob/main/clean_developers.csv"
click node_clean_developers_json "https://github.com/disabandhu/developer-lead-pipeline/blob/main/clean_developers.json"
click node_enrichment_stats "https://github.com/disabandhu/developer-lead-pipeline/blob/main/enrichment_stats.py"
click node_quick_stats "https://github.com/disabandhu/developer-lead-pipeline/blob/main/quick_stats.py"
click node_website_stats "https://github.com/disabandhu/developer-lead-pipeline/blob/main/website_stats.py"
click node_test_enrichment "https://github.com/disabandhu/developer-lead-pipeline/blob/main/test_enrichment.py"
click node_test_articles "https://github.com/disabandhu/developer-lead-pipeline/blob/main/test_articles.py"
click node_sample_personal_sites "https://github.com/disabandhu/developer-lead-pipeline/blob/main/sample_personal_sites.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_bulk_enrichment toneBlue
class node_dev_api,node_username_collector,node_profile_collector toneAmber
class node_website_classifier,node_website_enrichment,node_website_parser,node_github_enrichment,node_lead_scoring toneMint
class node_database,node_developers_csv,node_clean_developers_csv,node_clean_developers_json toneRose
class node_enrichment_stats,node_quick_stats,node_website_stats toneIndigo
class node_test_enrichment,node_test_articles,node_sample_personal_sites toneTeal
```

## Components

### Orchestration

Batch entrypoints and run modes.

- [Main](https://github.com/disabandhu/developer-lead-pipeline/blob/main/main.py): runner · Primary pipeline entrypoint. · `main.py`
- [Bulk run](https://github.com/disabandhu/developer-lead-pipeline/blob/main/bulk_enrichment.py): batch runner · Batch mode for processing many developers. · `bulk_enrichment.py`

### Discovery

Identity collection and normalization from source APIs.

- [DEV API](https://github.com/disabandhu/developer-lead-pipeline/blob/main/dev_api.py): source connector · Discovers developer identities from DEV.to. · `dev_api.py`
- [Usernames](https://github.com/disabandhu/developer-lead-pipeline/blob/main/username_collector.py): normalizer · Derives or normalizes usernames for joins across platforms. · `username_collector.py`
- [Profiles](https://github.com/disabandhu/developer-lead-pipeline/blob/main/profile_collector.py): profile builder · Builds structured developer records from discovered identities. · `profile_collector.py`

### Enrichment

Website and external-profile enrichment plus scoring.

- [Site classifier](https://github.com/disabandhu/developer-lead-pipeline/blob/main/website_classifier.py): classifier · Separates personal sites from other web pages. · `website_classifier.py`
- [Web enrich](https://github.com/disabandhu/developer-lead-pipeline/blob/main/website_enrichment.py): web enrichment · Drives analysis of personal websites for contact and social signals. · `website_enrichment.py`
- [Page parser](https://github.com/disabandhu/developer-lead-pipeline/blob/main/website_parser.py): parser · Extracts emails, links, and profile signals from site content. · `website_parser.py`
- **GitHub enrich**: external enrichment · Adds GitHub bio, company, followers, repositories, and account-age signals.
- [Lead score](https://github.com/disabandhu/developer-lead-pipeline/blob/main/lead_scoring.py): scorer · Converts enrichment completeness into a lead-quality score. · `lead_scoring.py`

### Storage

Intermediate persistence and exported datasets.

- [Database](https://github.com/disabandhu/developer-lead-pipeline/blob/main/database.py): storage layer · Intermediate or future persistence abstraction. · `database.py`
- [Developers CSV](https://github.com/disabandhu/developer-lead-pipeline/blob/main/developers.csv): dataset export · Primary exported table of developer records. · `developers.csv`
- [Clean CSV](https://github.com/disabandhu/developer-lead-pipeline/blob/main/clean_developers.csv): clean dataset · Normalized downstream dataset artifact. · `clean_developers.csv`
- [Clean JSON](https://github.com/disabandhu/developer-lead-pipeline/blob/main/clean_developers.json): clean dataset · JSON version of the cleaned developer dataset. · `clean_developers.json`

### Metrics

Coverage and quality inspection scripts.

- [Enrich stats](https://github.com/disabandhu/developer-lead-pipeline/blob/main/enrichment_stats.py): metrics · Coverage and quality summary for enrichment stages. · `enrichment_stats.py`
- [Quick stats](https://github.com/disabandhu/developer-lead-pipeline/blob/main/quick_stats.py): metrics · Fast aggregate inspection of pipeline output. · `quick_stats.py`
- [Web stats](https://github.com/disabandhu/developer-lead-pipeline/blob/main/website_stats.py): metrics · Website-specific coverage and extraction reporting. · `website_stats.py`

### Validation

Tests and sample inputs used to verify extraction behavior.

- [Enrich tests](https://github.com/disabandhu/developer-lead-pipeline/blob/main/test_enrichment.py): test · Checks enrichment and parsing behavior. · `test_enrichment.py`
- [Article tests](https://github.com/disabandhu/developer-lead-pipeline/blob/main/test_articles.py): test · Validates article or content extraction behavior. · `test_articles.py`
- [Site samples](https://github.com/disabandhu/developer-lead-pipeline/blob/main/sample_personal_sites.py): fixture · Sample personal-site inputs for extraction workflows. · `sample_personal_sites.py`

## Connections

- Main → DEV API: start discovery · Main orchestrates the source collection flow.
- Main → Web enrich: run enrichment · Main triggers downstream enrichment stages.
- Bulk run → Profiles: batch profiles · Bulk mode processes many discovered developers.
- DEV API → Usernames: raw identities · DEV.to identities are normalized for joining.
- Usernames → Profiles: joined users · Normalized usernames feed profile assembly.
- Profiles → Site classifier: site candidates · Collected profiles expose website fields for classification.
- Site classifier → Web enrich: personal sites · Only relevant websites are sent to enrichment.
- Web enrich → Page parser: page content · Website analysis hands pages to the parser.
- Profiles → GitHub enrich: profile seeds · Profile records drive GitHub lookup and merge.
- Page parser → Lead score: site signals · Extracted contact and social links affect score.
- GitHub enrich → Lead score: github signals · GitHub completeness contributes to lead quality.
- Lead score → Database: persist · Scored developer records can be stored for later use.
- Database → Developers CSV: export table · Canonical structured dataset export.
- Developers CSV → Clean CSV: cleaned · Primary table is normalized into a cleaned CSV artifact.
- Developers CSV → Clean JSON: serialized · Primary table is also exported as JSON.
- Developers CSV → Enrich stats: coverage · Stats scripts inspect enrichment completeness.
- Developers CSV → Quick stats: summary · Fast reporting over pipeline output.
- Page parser → Web stats: site metrics · Website parsing coverage is measured separately.
- Page parser → Enrich tests: verify parser · Tests exercise enrichment and extraction behavior.
- Page parser → Article tests: verify content · Content extraction checks validate parsing logic.
- Site samples → Web enrich: fixture input · Sample sites support local enrichment experiments.