# jtrecenti/juscraper architecture

> Architecture diagram of the GitHub repository jtrecenti/juscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jtrecenti/juscraper
- Repository: https://github.com/jtrecenti/juscraper

## Overview

##### Project type and purpose
- juscraper is an open-source Python library for scraping judicial court systems (e.g., TJSP, TJRS, TJBR).
- It provides a uniform API to query case data (first and second degree processes) and jurisprudence, download raw pages, parse them into structured tables, and return pandas DataFrames.

##### File structure
- Top level:
- notebooks/ – example Jupyter notebooks demonstrating usage
- src/juscraper/ – core package
- tests/ – unit tests per tribunal and end-to-end
- pyproject.toml – dependencies and build config
- src/juscraper components:
- base_scraper.py – BaseScraper class with shared logic (template methods)
- tribunal_manager.py – factory/registry mapping tribunal codes to scraper classes
- {jusbr, tjrs, tjsp}_scraper.py – concrete Scraper subclasses implementing tribunal-specific URL construction, parsing logic, pagination
- utils.py – helper functions (HTTP requests, parsing HTML/JSON, progress bars)
- Tests mirror structure, validating each scraper’s behavior and parsing.

##### README for additional insights
- Public API: scraper('tjsp') returns a scraper instance with methods .cpopg(), .cposg(), .cjsg()
- Download/parse separation: methods with _download() download raw files, _parse() parse stored files
- The library discards raw files by default but allows local storage when desired
- No parallelization – sequential HTTP requests only
- Extensibility: new tribunals added by subclassing BaseScraper and registering in tribunal_manager

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Client Layer"
        Client["Jupyter Notebook"]:::client
    end
    click Client "https://github.com/jtrecenti/juscraper/tree/main/notebooks/"

    subgraph "Core Library"
        PublicAPI["scraper('tribunal') Facade"]:::core
        Factory["TribunalManager"]:::core
        Base["BaseScraper (Template)"]:::core
        subgraph "Tribunal Scrapers"
            JUSBR["JUSBRScraper"]:::core
            TJRS["TJRSScraper"]:::core
            TJSP["TJSPScraper"]:::core
        end
        Utils["Utils (HTTP, Parsing, Progress)"]:::core
    end
    click PublicAPI "https://github.com/jtrecenti/juscraper/blob/main/src/juscraper/__init__.py"
    click Factory "https://github.com/jtrecenti/juscraper/blob/main/src/juscraper/tribunal_manager.py"
    click Base "https://github.com/jtrecenti/juscraper/blob/main/src/juscraper/base_scraper.py"
    click JUSBR "https://github.com/jtrecenti/juscraper/blob/main/src/juscraper/jusbr_scraper.py"
    click TJRS "https://github.com/jtrecenti/juscraper/blob/main/src/juscraper/tjrs_scraper.py"
    click TJSP "https://github.com/jtrecenti/juscraper/blob/main/src/juscraper/tjsp_scraper.py"
    click Utils "https://github.com/jtrecenti/juscraper/blob/main/src/juscraper/utils.py"

    External["External Court Systems (HTTP/HTTPS)"]:::external
    FileSys["Local File Storage"]:::file
    DataFrame["Pandas DataFrame"]:::core

    Client -->|calls| PublicAPI
    PublicAPI -->|selects scraper| Factory
    Factory --> JUSBR
    Factory --> TJRS
    Factory --> TJSP
    JUSBR -->|requests| External
    TJRS -->|requests| External
    TJSP -->|requests| External
    JUSBR -->|uses| Utils
    TJRS -->|uses| Utils
    TJSP -->|uses| Utils
    Utils -->|parses to| DataFrame
    JUSBR -->|optional write| FileSys
    TJRS -->|optional write| FileSys
    TJSP -->|optional write| FileSys
    DataFrame -->|returns to| Client

    subgraph "Test Suite"
        GeneralTests["tests/test_juscraper.py"]:::test
        TJRSTests["tests/tjrs/"]:::test
        TJSPOTests["tests/tjsp/"]:::test
    end
    click GeneralTests "https://github.com/jtrecenti/juscraper/blob/main/tests/test_juscraper.py"
    click TJRSTests "https://github.com/jtrecenti/juscraper/tree/main/tests/tjrs/"
    click TJSPOTests "https://github.com/jtrecenti/juscraper/tree/main/tests/tjsp/"

    GeneralTests -->|validates| PublicAPI
    TJRSTests -->|validates| TJRS
    TJSPOTests -->|validates| TJSP

    Config["pyproject.toml"]:::config
    Config -->|build config| PublicAPI

    classDef core fill:#add8e6,stroke:#000,color:#000
    classDef external fill:#98fb98,stroke:#000,color:#000
    classDef file fill:#d3d3d3,stroke:#000,color:#000
    classDef client fill:#ffa500,stroke:#000,color:#000
    classDef test fill:#f08080,stroke:#000,color:#000
    classDef config fill:#e6e6fa,stroke:#000,color:#000
```
