How shine-six/asb works
Project type and purpose
• This is a Python-based backend service whose sole function is to crawl LinkedIn public profiles and company pages, extract structured profile and company data, and write that data to export files (CSV/JSON). It exposes a single HTTP endpoint (/crawl) to accept a list of LinkedIn URLs and return the extracted data.
File structure
• Top-level:
• .gitignore, README.md
• backend/ – contains the entire service
• backend/:
• .env.example – shows external configuration (LinkedIn credentials, API keys)
• pyproject.toml, requirements.txt – Python packaging and dependencies
• main.py – entry point that starts the HTTP server (likely FastAPI + Uvicorn)
• app/ – core application package:
– routes.py – defines the /crawl HTTP endpoint
– crawler.py – contains low-level HTTP requests, anti-bot headers, proxies, rate-limiting logic
– llm_processor.py – optional post-processing of raw scraped data (e.g., normalize or enrich)
– utils.py – shared helpers (parsing, retries, data cleaning)
– credentials.py – loads secrets from .env
– config.py – global configuration (timeouts, rate limits, output paths)
• exports/ – sample output files (CSV snapshots)
README for additional insights
• Features: lists the fields extracted, anti-bot strategies, output format
• Usage example: shows a POST /crawl with a payload of LinkedIn profile URLs
• Technology stack: Python + Uvicorn (via uv sync / uv run), environment variables for credentials
a. Main Components
• Client / Consumer – issues HTTP POST to /crawl
• HTTP Server – Uvicorn + FastAPI (or similar) listening on port 8000
• API Layer (routes.py) – request validation, error handling
• Crawler Module (crawler.py) – fetches pages, applies anti-bot headers, proxies, rate-limit
• LLM Processor (llm_processor.py) – optional NLP or transformation layer
• Utilities & Config – shared logic (utils.py), configuration (config.py), credentials loader (credentials.py)
• Exporter – writes structured JSON/CSV to exports/ folder
• External Service – LinkedIn public site, third-party proxy services
b. Relationships & Data Flow
1. Client sends URL list → HTTP Server (FastAPI)
2. Server forwards list → Crawler Module
3. Crawler fetches each page from LinkedIn (through proxies & respecting rate limits)
4. Raw HTML → LLM Processor (if enabled) to extract/normalize data
5. Processed data → Exporter → JSON/CSV files in exports/
6. Server returns final JSON response to Client
c. Architectural Patterns & Design Principles
• Layered architecture:
1. Presentation layer (API)
2. Business logic layer (Crawler + Processor)
3. Infrastructure layer (Utils, Config, Credentials)
4. Output/Storage layer (Exports folder)
• Single-responsibility: each module focuses on one task (routing, crawling, processing, exporting)
• Configuration-driven: use of .env and config.py for flexibility
• Anti-bot design: separate proxy and headers logic for maintainability
d. Technologies & Frameworks
• Python 3.x
• FastAPI (implied by Uvicorn usage) or Flask + Uvicorn
• Uvicorn server
• HTTP libraries: requests or httpx
• Proxy services (configurable in .env)
• CSV / JSON writing built-ins
Key Points
• Environment/configuration dependencies
• Where secrets (.env) are loaded and used
• File-based export folder as a simple data store