# cullenwatson/staffspy architecture

> Architecture diagram of the GitHub repository cullenwatson/staffspy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cullenwatson/staffspy
- Repository: https://github.com/cullenwatson/staffspy

## Overview

##### Project Type & Purpose
- This is a library (with example scripts for usage) that automates data fetching/scraping from LinkedIn.
- It is not a full-stack web application; rather, it’s a backend module integrated via Python with external browser automation support.
- Key functions include scraping staff, users, comments, companies, and connections by interfacing with LinkedIn’s web interface, with optional captcha solving.

##### File Structure
- Top-Level Directories:
- .github: Contains CI/CD workflows (e.g., publish-to-pypi), ensuring automated testing and deployment.
- examples: Contains sample scripts demonstrating how to use the library (e.g., auto connection, uploading staff information, etc.).
- staffspy: The main package which is further divided into:
⁃ linkedin: Contains modules for scraping various data types (certifications, comments, contact info, experiences, etc.).
⁃ solvers: Contains modules for captcha handling (e.g., CapSolver, two_captcha, and abstract solver definitions).
⁃ utils: Contains utility modules that manage driver types, exceptions, models, and other helper functionalities.
- Configuration Files:
- pyproject.toml and .pre-commit-config.yaml help with package configuration and code quality.
- LICENSE, README.md, and .gitignore contribute standard project management aspects.

##### Architectural Components
a. Main Library Interface:
- At the highest level, you have the LinkedInAccount class that acts as the entry point for all scraping operations. It takes parameters like session_file, driver configuration, log level, etc.
- This component is crucial as it orchestrates the scraping process.

b. LinkedIn Module Components:
- Several sub-modules (certifications.py, comments.py, contact_info.py, employee.py, etc.) handle extracting different segments of LinkedIn profiles.
- These components should be shown as part of the "LinkedIn Scraper" block interacting with LinkedIn endpoints.

c. Solvers Module:
- Contains modules to handle captchas: a generic solver interface (solver.py and solver_type.py) and specific implementations (capsolver.py and two_captcha.py).
- This component interacts with external captcha solving services.

d. Utility Module:
- Components for driver type configuration (driver_type.py), handling exceptions (exceptions.py), and managing models/data structures (models.py, utils.py).
- These support the main scraping functions and can be depicted as helper services.

e. External Services & Data Flow:
- Scraping process interacting externally with LinkedIn (via browser automation or direct HTTP requests).
- A connection to captcha services if required.

##### Relationships & Interactions
- That during scraping, if captcha detection occurs, a call is made to the “Captcha Solver” module, which in turn interacts with external services (CapSolver or 2Captcha).
- Also, indicate that utility functions support data processing, error handling, and model instantiation.

##### Architectural Patterns & Design Principles
- Separation of concerns: Different modules handle distinct responsibilities—scraping LinkedIn, solving captchas, and assisting with utility functions.
- Modularity: The structure in the file tree (linkedin, solvers, utils) reflects a modular design.
- Extensibility: The use of abstract classes or configurable parameters (like DriverType and SolverType) allows for easy integration with additional services or browser types.
- Reusability: Example files show usage patterns which indicate that components are reusable and can be extended or integrated as needed.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Entities
    A["User/Example Scripts"]:::external
    B["LinkedInAccount Manager"]:::interface
    A --> B

    %% LinkedIn Scraper Components
    subgraph "LinkedIn Scraper Components"
        C1["Certifications"]:::scraper
        C2["Comments"]:::scraper
        C3["Contact Info"]:::scraper
        C4["Employee Data"]:::scraper
        C5["Employee Biography"]:::scraper
        C6["Experiences"]:::scraper
        C7["Languages"]:::scraper
        C8["Schools"]:::scraper
        C9["Skills"]:::scraper
    end
    %% Connect Library Interface to Scraper Components vertically
    B --> C1
    C1 --> C2
    C2 --> C3
    C3 --> C4
    C4 --> C5
    C5 --> C6
    C6 --> C7
    C7 --> C8
    C8 --> C9

    %% External LinkedIn Service
    D["LinkedIn External Service"]:::external
    C9 --> D

    %% Captcha Handling Path
    E["Captcha Encounter?"]:::scraper
    B ---|"captcha detected"| E

    subgraph "Captcha Solver Service"
        F1["Abstract Solver"]:::captcha
        F2["Solver Types"]:::captcha
        F3["CapSolver"]:::captcha
        F4["2Captcha"]:::captcha
    end
    E --> F1
    F1 --> F2
    F2 --> F3
    F2 --> F4
    F3 --> G["Captcha External Service"]:::external
    F4 --> G

    %% Utility Modules
    subgraph Utilities
        U1["Driver Configuration"]:::utility
        U2["Exception Handling"]:::utility
        U3["Data Models"]:::utility
        U4["Helper Utils"]:::utility
    end
    B --> U1
    U1 --> U2
    U2 --> U3
    U3 --> U4

    %% Final Data Output
    H["Output Data (Pandas DataFrame/CSV)"]:::data
    D --> H
    G --> H
    U4 --> H

    %% Click Events for Library Interface and LinkedIn Scraper Components
    click B "https://github.com/cullenwatson/staffspy/blob/main/staffspy/linkedin/linkedin.py"
    click C1 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/linkedin/certifications.py"
    click C2 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/linkedin/comments.py"
    click C3 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/linkedin/contact_info.py"
    click C4 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/linkedin/employee.py"
    click C5 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/linkedin/employee_bio.py"
    click C6 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/linkedin/experiences.py"
    click C7 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/linkedin/languages.py"
    click C8 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/linkedin/schools.py"
    click C9 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/linkedin/skills.py"

    %% Click Events for Captcha Solver Modules
    click F1 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/solvers/solver.py"
    click F2 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/solvers/solver_type.py"
    click F3 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/solvers/capsolver.py"
    click F4 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/solvers/two_captcha.py"

    %% Click Events for Utility Modules
    click U1 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/utils/driver_type.py"
    click U2 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/utils/exceptions.py"
    click U3 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/utils/models.py"
    click U4 "https://github.com/cullenwatson/staffspy/blob/main/staffspy/utils/utils.py"

    %% Styles
    classDef interface fill:#ADD8E6,stroke:#00008B,stroke-width:2px;
    classDef scraper fill:#90EE90,stroke:#006400,stroke-width:2px;
    classDef captcha fill:#F4A460,stroke:#8B4513,stroke-width:2px;
    classDef utility fill:#FFD700,stroke:#B8860B,stroke-width:2px;
    classDef external fill:#D3D3D3,stroke:#696969,stroke-width:2px;
    classDef data fill:#FFB6C1,stroke:#C71585,stroke-width:2px;
```
