How jtrecenti/juscraper works
Project type and purpose
• juscraper is an open-source Python library for scraping judicial court systems (e.g., TJSP, TJRS, TJBR).
• It provides a uniform API to query case data (first and second degree processes) and jurisprudence, download raw pages, parse them into structured tables, and return pandas DataFrames.
File structure
• Top level:
• notebooks/ – example Jupyter notebooks demonstrating usage
• src/juscraper/ – core package
• tests/ – unit tests per tribunal and end-to-end
• pyproject.toml – dependencies and build config
• src/juscraper components:
• base_scraper.py – BaseScraper class with shared logic (template methods)
• tribunal_manager.py – factory/registry mapping tribunal codes to scraper classes
• {jusbr, tjrs, tjsp}_scraper.py – concrete Scraper subclasses implementing tribunal-specific URL construction, parsing logic, pagination
• utils.py – helper functions (HTTP requests, parsing HTML/JSON, progress bars)
• Tests mirror structure, validating each scraper’s behavior and parsing.
README for additional insights
• Public API: scraper('tjsp') returns a scraper instance with methods .cpopg(), .cposg(), .cjsg()
• Download/parse separation: methods with _download() download raw files, _parse() parse stored files
• The library discards raw files by default but allows local storage when desired
• No parallelization – sequential HTTP requests only
• Extensibility: new tribunals added by subclassing BaseScraper and registering in tribunal_manager