# shine-six/asb architecture

> Architecture diagram of the GitHub repository shine-six/asb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shine-six/asb
- Repository: https://github.com/shine-six/asb

## Overview

##### Project type and purpose
- This is a Python-based backend service whose sole function is to crawl LinkedIn public profiles and company pages, extract structured profile and company data, and write that data to export files (CSV/JSON). It exposes a single HTTP endpoint (`/crawl`) to accept a list of LinkedIn URLs and return the extracted data.

##### File structure
- Top-level:
- `.gitignore`, `README.md`
- `backend/` – contains the entire service
- `backend/`:
- `.env.example` – shows external configuration (LinkedIn credentials, API keys)
- `pyproject.toml`, `requirements.txt` – Python packaging and dependencies
- `main.py` – entry point that starts the HTTP server (likely FastAPI + Uvicorn)
- `app/` – core application package:
– `routes.py` – defines the `/crawl` HTTP endpoint
– `crawler.py` – contains low-level HTTP requests, anti-bot headers, proxies, rate-limiting logic
– `llm_processor.py` – optional post-processing of raw scraped data (e.g., normalize or enrich)
– `utils.py` – shared helpers (parsing, retries, data cleaning)
– `credentials.py` – loads secrets from `.env`
– `config.py` – global configuration (timeouts, rate limits, output paths)
- `exports/` – sample output files (CSV snapshots)

##### README for additional insights
- Features: lists the fields extracted, anti-bot strategies, output format
- Usage example: shows a POST `/crawl` with a payload of LinkedIn profile URLs
- Technology stack: Python + Uvicorn (via `uv sync` / `uv run`), environment variables for credentials

a. Main Components
- Client / Consumer – issues HTTP POST to `/crawl`
- HTTP Server – Uvicorn + FastAPI (or similar) listening on port 8000
- API Layer (`routes.py`) – request validation, error handling
- Crawler Module (`crawler.py`) – fetches pages, applies anti-bot headers, proxies, rate-limit
- LLM Processor (`llm_processor.py`) – optional NLP or transformation layer
- Utilities & Config – shared logic (`utils.py`), configuration (`config.py`), credentials loader (`credentials.py`)
- Exporter – writes structured JSON/CSV to `exports/` folder
- External Service – LinkedIn public site, third-party proxy services

b. Relationships & Data Flow
1. Client sends URL list → HTTP Server (FastAPI)
2. Server forwards list → Crawler Module
3. Crawler fetches each page from LinkedIn (through proxies & respecting rate limits)
4. Raw HTML → LLM Processor (if enabled) to extract/normalize data
5. Processed data → Exporter → JSON/CSV files in `exports/`
6. Server returns final JSON response to Client

c. Architectural Patterns & Design Principles
- Layered architecture:
1. Presentation layer (API)
2. Business logic layer (Crawler + Processor)
3. Infrastructure layer (Utils, Config, Credentials)
4. Output/Storage layer (Exports folder)
- Single-responsibility: each module focuses on one task (routing, crawling, processing, exporting)
- Configuration-driven: use of `.env` and `config.py` for flexibility
- Anti-bot design: separate proxy and headers logic for maintainability

d. Technologies & Frameworks
- Python 3.x
- FastAPI (implied by Uvicorn usage) or Flask + Uvicorn
- Uvicorn server
- HTTP libraries: `requests` or `httpx`
- Proxy services (configurable in `.env`)
- CSV / JSON writing built-ins

##### Key Points
- Environment/configuration dependencies
- Where secrets (.env) are loaded and used
- File-based export folder as a simple data store

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        Client["Client"]:::external
        HTTP["HTTP Server\n(main.py)"]:::api
        API["API Layer\n(routes.py)"]:::api
    end

    %% Business Logic Layer
    subgraph "Business Logic Layer"
        direction TB
        Crawler["Crawler Module\n(crawler.py)"]:::business
        LLM["LLM Processor\n(llm_processor.py)"]:::business
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        direction TB
        Utils["Utilities\n(utils.py)"]:::infra
        Config["Configuration\n(config.py)"]:::infra
        Creds["Credentials Loader\n(credentials.py)"]:::infra
        Env[".env.example"]:::infra
        Dep["pyproject.toml\nrequirements.txt"]:::infra
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        LI["LinkedIn Public Site"]:::external
        Proxy["Proxy Service"]:::external
    end

    %% Storage/Exports
    subgraph "Storage/Exports"
        direction TB
        Exports["Exports Folder\n(Exporter)"]:::storage
    end

    %% Data Flow
    Client -->|POST /crawl| HTTP
    HTTP -->|forwards request| API
    API -->|invoke crawl| Crawler
    Crawler -->|fetch pages| Proxy
    Crawler -->|fetch pages| LI
    LI -->|HTML response| Crawler
    Crawler -->|raw data| LLM
    LLM -->|processed data| Exports
    Exports -->|files| API
    API -->|JSON response| Client

    %% Infrastructure Dependencies
    Config --> API
    Config --> Crawler
    Config --> LLM
    Creds --> Crawler
    Utils --> Crawler
    Utils --> LLM
    Env --> Creds
    Dep --> HTTP

    %% Click Events
    click HTTP "https://github.com/shine-six/asb/blob/main/backend/main.py"
    click API "https://github.com/shine-six/asb/blob/main/backend/app/routes.py"
    click Crawler "https://github.com/shine-six/asb/blob/main/backend/app/crawler.py"
    click LLM "https://github.com/shine-six/asb/blob/main/backend/app/llm_processor.py"
    click Utils "https://github.com/shine-six/asb/blob/main/backend/app/utils.py"
    click Config "https://github.com/shine-six/asb/blob/main/backend/app/config.py"
    click Creds "https://github.com/shine-six/asb/blob/main/backend/app/credentials.py"
    click Exports "https://github.com/shine-six/asb/tree/main/backend/exports/"
    click Env "https://github.com/shine-six/asb/blob/main/backend/.env.example"
    click Dep "https://github.com/shine-six/asb/blob/main/backend/pyproject.toml"

    %% Styles
    classDef api fill:#D0E8FF,stroke:#0366d6,color:#0366d6;
    classDef business fill:#DFFFE0,stroke:#28a745,color:#28a745;
    classDef infra fill:#F0F0F0,stroke:#6a737d,color:#6a737d;
    classDef external fill:#FFF4E5,stroke:#d73a49,color:#d73a49;
    classDef storage fill:#F5E1FF,stroke:#6f42c1,color:#6f42c1;
```
