How dgunning/edgartools works
Project type and purpose
• EdgarTools is a Python library (not a web app) for programmatic extraction and analysis of SEC EDGAR filings.
• Purpose: provide a clean, intuitive API that fetches filings (HTML, SGML, XBRL), parses them into Python objects and pandas DataFrames, caches results, and supports advanced XBRL analytics.
File structure
• Top-level package “edgar/”: contains sub-packages for core API, domain models, parsing, HTTP client, caching, storage, search, and XBRL logic.
• edgar/httpclient(.py, _cache.py, httprequests.py) – fetch & cache SEC data
• edgar/sgml – SGML download & parsing
• edgar/xbrl – XBRL parsing, standardization, stitching, rendering, analysis
• edgar/files – HTML extraction and conversion to markdown/tables
• edgar/entity, edgar/funds, edgar/ownership, edgar/offerings – domain modules for Company, Fund, Insider, Offerings
• edgar/core, _filings.py – high‐level API classes (Company, Filings, Filing objects)
• edgar/storage.py – local storage & cache management
• edgar/search – metadata & full‐text search functionality
• Support artifacts: “data/” (sample files, schemas), “docs/” (Markdown, notebooks), “tests/” (unit, performance, fixtures), CI config, mkdocs/pyproject for packaging.
README for additional insights
• Highlights automatic throttling, pandas integration, XBRL support, AI/LLM readyness, domain-specific objects (Form 4, 13F), high performance via lxml, optional PyArrow.
• Quick start shows 7 steps: identity→Company→get_filings()→filter→convert to data object.
a. Main components
1. Client Code / User Application
2. EdgarTools Library
– Public API Layer (Company, Filings, set_identity)
– Infrastructure Layer
• HTTP Client & Throttling
• Cache & Local Storage
– Parsing Layer
• SGML Parser
• HTML Extraction & Markdown/Tables
• XBRL Parser (instance, definition, calculation, presentation)
– Domain Layer
• Company/Entity Model
• Fund/13F Model
• Ownership/Form 4 Model
• Offerings/Form D/C Model
– Data Transformation Layer
• pandas DataFrame conversion
• Clean text / NLP chunking
3. External Services & Dependencies
– SEC EDGAR API (HTML, XML)
– Python libs: requests, lxml, pandas, pyarrow, hatch
b. Relationships & interactions
• Client calls Public API → HTTP Client fetches raw filings from SEC → Cache writes to Local Storage → Parsing Layer transforms raw (SGML, HTML, XBRL) → Domain Layer constructs structured objects → Data Transformation converts to DataFrames/JSON/text → returns to client.
• In parallel, search module indexes filings/metadata; XBRL components feed Analytics & Ratios modules.
c. Architectural patterns
• Layered architecture:
• Presentation/API
• Infrastructure
• Parsing & Processing
• Domain Model
• Data Output
• Modular plugin style for each form type.
• Cache-aside pattern for HTTP requests and local storage.
d. Key technologies / frameworks
• Python 3 package (pyproject, hatch)
• requests or builtin HTTP client
• lxml for XML/HTML/SGML parsing
• pandas for tabular data
• Optional PyArrow for performance
• MkDocs/Hatch for docs & CI workflows
Detail
• How HTTP throttling and caching intercept requests.
• Parsing stages of XBRL (instance parsing, standardization, stitching, rendering).
• How domain objects (Company, Fund, Ownership) rely on parsing outputs.