How elvicducou/bugcrowd-targets-fetcher works
Project type and purpose
• This is a small Python command-line tool (“Bugcrowd Targets Fetcher”) that scrapes Bugcrowd programs and extracts in-scope domains.
• It’s not a full-stack web app or microservices platform but a single-purpose ETL-style CLI application.
File structure
• There is a single script: Bugrowd_targets_fetcher.py (likely a typo; the README refers to bugcrowd_domains_fetcher.py).
• Supporting files: Readme.md, requirements.txt (implied by the README).
• No separate frontend/backend directories—everything runs in one process.
README insights
• Uses libraries: requests (HTTP client), beautifulsoup4 (HTML parser), tqdm (progress bar).
• Implements rate limiting (0.5s between requests).
• Outputs CSV files, one per program or one combined.
a. CLI Input Parser
• Reads command-line arguments (no args = fetch all, args = filter specific programs).
b. Controller / Orchestrator
• Decides whether to fetch all programs or only specified ones.
• Manages overall flow and invokes rate limiter and fetch routines.
c. Rate Limiter
• Enforces delay between HTTP requests to Bugcrowd.
d. HTTP Client
• Uses requests to GET program listing pages and in-scope domain pages.
e. HTML Parser
• Uses BeautifulSoup to extract program names and domain lists.
f. Data Aggregator / Formatter
• Transforms parsed data into rows (Program, Domain).
g. CSV Writer (Persistence)
• Writes rows to one or more CSV files on disk.
h. External Service
• Bugcrowd website or API endpoints (Programs listing, Program scope pages).
Relationships and interactions
• CLI → Controller → (Rate Limiter → HTTP Client → HTML Parser → Data Aggregator) → CSV Writer
• Controller also consults program-filter logic before requesting each program.
• All HTTP requests go to the external Bugcrowd service.
• Data flows from parsing into CSV files.
Architectural patterns and principles
• Pipeline pattern: sequential stages (fetch → parse → format → persist).
• Single responsibility: each module (rate limiting, HTTP, parsing, I/O) has one concern.
• Respectful resource usage: rate limiting to avoid server overload.
Technologies and libraries
• Python 3.6+
• requests (HTTP)
• beautifulsoup4 (parsing)
• tqdm (progress bar)
• csv (standard library, writing CSV)