# dgunning/edgartools architecture

> Architecture diagram of the GitHub repository dgunning/edgartools, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dgunning/edgartools
- Repository: https://github.com/dgunning/edgartools

## Overview

##### Project type and purpose
- EdgarTools is a Python library (not a web app) for programmatic extraction and analysis of SEC EDGAR filings.
- Purpose: provide a clean, intuitive API that fetches filings (HTML, SGML, XBRL), parses them into Python objects and pandas DataFrames, caches results, and supports advanced XBRL analytics.

##### File structure
- Top-level package “edgar/”: contains sub-packages for core API, domain models, parsing, HTTP client, caching, storage, search, and XBRL logic.
- edgar/httpclient(.py, _cache.py, httprequests.py) – fetch & cache SEC data
- edgar/sgml – SGML download & parsing
- edgar/xbrl – XBRL parsing, standardization, stitching, rendering, analysis
- edgar/files – HTML extraction and conversion to markdown/tables
- edgar/entity, edgar/funds, edgar/ownership, edgar/offerings – domain modules for Company, Fund, Insider, Offerings
- edgar/core, _filings.py – high‐level API classes (Company, Filings, Filing objects)
- edgar/storage.py – local storage & cache management
- edgar/search – metadata & full‐text search functionality
- Support artifacts: “data/” (sample files, schemas), “docs/” (Markdown, notebooks), “tests/” (unit, performance, fixtures), CI config, mkdocs/pyproject for packaging.

##### README for additional insights
- Highlights automatic throttling, pandas integration, XBRL support, AI/LLM readyness, domain-specific objects (Form 4, 13F), high performance via lxml, optional PyArrow.
- Quick start shows 7 steps: identity→Company→get_filings()→filter→convert to data object.

a. Main components
1. Client Code / User Application
2. EdgarTools Library
– Public API Layer (Company, Filings, set_identity)
– Infrastructure Layer
- HTTP Client & Throttling
- Cache & Local Storage
– Parsing Layer
- SGML Parser
- HTML Extraction & Markdown/Tables
- XBRL Parser (instance, definition, calculation, presentation)
– Domain Layer
- Company/Entity Model
- Fund/13F Model
- Ownership/Form 4 Model
- Offerings/Form D/C Model
– Data Transformation Layer
- pandas DataFrame conversion
- Clean text / NLP chunking
3. External Services & Dependencies
– SEC EDGAR API (HTML, XML)
– Python libs: requests, lxml, pandas, pyarrow, hatch
b. Relationships & interactions
- Client calls Public API → HTTP Client fetches raw filings from SEC → Cache writes to Local Storage → Parsing Layer transforms raw (SGML, HTML, XBRL) → Domain Layer constructs structured objects → Data Transformation converts to DataFrames/JSON/text → returns to client.
- In parallel, search module indexes filings/metadata; XBRL components feed Analytics & Ratios modules.
c. Architectural patterns
- Layered architecture:
- Presentation/API
- Infrastructure
- Parsing & Processing
- Domain Model
- Data Output
- Modular plugin style for each form type.
- Cache-aside pattern for HTTP requests and local storage.
d. Key technologies / frameworks
- Python 3 package (pyproject, hatch)
- requests or builtin HTTP client
- lxml for XML/HTML/SGML parsing
- pandas for tabular data
- Optional PyArrow for performance
- MkDocs/Hatch for docs & CI workflows

##### Detail
- How HTTP throttling and caching intercept requests.
- Parsing stages of XBRL (instance parsing, standardization, stitching, rendering).
- How domain objects (Company, Fund, Ownership) rely on parsing outputs.

## Diagram (Mermaid)

```mermaid
graph TB
    %% External
    SEC["SEC EDGAR API"]:::external
    PyLibs["Python Libraries\n(requests, lxml, pandas, pyarrow)"]:::external

    %% Client
    ClientApp["Client Application"]:::client

    %% Public API Layer
    subgraph "Public API Layer"
        style PublicAPI fill:#D0E8FF
        CompanyAPI["Company API\n(edgar/core.py)"]:::api
        FilingsAPI["Filings API\n(edgar/_filings.py)"]:::api
        SetIdentity["set_identity\n(edgar/company_reports.py)"]:::api
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        style Infra fill:#DFF2BF
        HTTPClient["HTTP Client & Throttling\n(edgar/httpclient.py)"]:::infra
        HTTPRequests["HTTP Requests\n(edgar/httprequests.py)"]:::infra
        HTTPCache["HTTP Cache\n(edgar/httpclient_cache.py)"]:::infra
        CacheStorage["Cache & Local Storage\n(edgar/storage.py)"]:::infra
        LocalStorage["Local Storage\n(data/localstorage/)"]:::infra
    end

    %% Parsing Layer
    subgraph "Parsing Layer"
        style Parsing fill:#FEEFB3
        SGMLParser["SGML Parser\n(edgar/sgml/sgml_parser.py)"]:::parsing
        HTMLExtract["HTML Extraction\n(edgar/files/html.py)"]:::parsing
        Markdown["Markdown Conversion\n(edgar/files/markdown.py)"]:::parsing
        Tables["Table Extraction\n(edgar/files/tables.py)"]:::parsing
        TextParse["Text Extraction\n(edgar/files/text.py)"]:::parsing
        subgraph "XBRL Parsing & Analytics"
            XBRLParser["Instance Parser\n(edgar/xbrl/parser.py)"]:::parsing
            XBRLCore["XBRL Core\n(edgar/xbrl/core.py)"]:::parsing
            XBRLStd["Standardization\n(edgar/xbrl/standardization/core.py)"]:::parsing
            XBRLStitch["Stitching\n(edgar/xbrl/stitching/core.py)"]:::parsing
            XBRLUtils["Stitch Utils\n(edgar/xbrl/stitching/utils.py)"]:::parsing
            XBRLXbrls["XBRL Files\n(edgar/xbrl/stitching/xbrls.py)"]:::parsing
            XBRLRender["Rendering\n(edgar/xbrl/rendering.py)"]:::parsing
            XBRLAnalysis["Analysis & Ratios\n(edgar/xbrl/analysis/metrics.py,\nratios.py)"]:::parsing
        end
    end

    %% Domain Layer
    subgraph "Domain Layer"
        style Domain fill:#E3E4FA
        CompanyModel["Company Model\n(edgar/entity/core.py,\nentity/data.py,\nentity/filings.py)"]:::domain
        FundModel["Fund Model\n(edgar/funds/core.py,\nfunds/thirteenf.py,\nfunds/data.py)"]:::domain
        OwnershipModel["Ownership Model\n(edgar/ownership/core.py,\nownership/ownershipforms.py,\nownership/html_render.py)"]:::domain
        OfferingsModel["Offerings Model\n(edgar/offerings/formd.py,\nofferings/formc.py)"]:::domain
    end

    %% Data Output Layer
    subgraph "Data Output Layer"
        style Output fill:#FFF2CC
        PandasDF["pandas DataFrame\n(edgar/datatools.py,\nedgar/financials.py)"]:::output
        NLPText["Clean Text / NLP\n(edgar/_markdown.py,\nedgar/files/text.py)"]:::output
    end

    %% Search & Reference
    subgraph "Search & Reference"
        style SR fill:#F5E1FD
        DataSearch["Data Search\n(edgar/search/datasearch.py)"]:::search
        TextSearch["Text Search\n(edgar/search/textsearch.py)"]:::search
        Codes["Codes Reference\n(edgar/reference/_codes.py)"]:::reference
        RefData["Reference Data\n(edgar/reference/data/)"]:::reference
        Exhibits["Exhibits Reference\n(edgar/reference/exhibits.py)"]:::reference
        Tickers["Tickers Reference\n(edgar/reference/tickers.py)"]:::reference
    end

    %% Data flows
    ClientApp -->|calls| CompanyAPI
    ClientApp -->|calls| FilingsAPI
    ClientApp -->|sets| SetIdentity

    CompanyAPI -->|fetch| HTTPClient
    FilingsAPI -->|fetch| HTTPClient

    HTTPClient --> HTTPRequests
    HTTPClient --> HTTPCache
    HTTPCache --> LocalStorage
    LocalStorage --> CacheStorage
    HTTPClient -->|request| SEC
    SEC -->|response| HTTPClient

    HTTPClient --> SGMLParser
    HTTPClient --> HTMLExtract
    HTTPClient --> XBRLParser

    SGMLParser --> CompanyModel
    HTMLExtract --> DomainLayerParsing[Domain Input]
    Markdown --> DomainLayerParsing
    Tables --> DomainLayerParsing
    TextParse --> DomainLayerParsing
    DomainLayerParsing --> CompanyModel

    XBRLParser --> XBRLCore
    XBRLCore --> XBRLStd
    XBRLStd --> XBRLStitch
    XBRLStitch --> XBRLUtils
    XBRLUtils --> XBRLXbrls
    XBRLXbrls --> XBRLRender
    XBRLRender --> XBRLAnalysis
    XBRLAnalysis -->|data| CompanyModel

    CompanyModel --> PandasDF
    FundModel --> PandasDF
    OwnershipModel --> PandasDF
    OfferingsModel --> PandasDF

    CompanyModel --> NLPText
    FundModel --> NLPText
    OwnershipModel --> NLPText
    OfferingsModel --> NLPText

    PandasDF -->|return| ClientApp
    NLPText -->|return| ClientApp

    LocalStorage --> DataSearch
    LocalStorage --> TextSearch
    DataSearch -->|query| PublicAPI
    TextSearch -->|query| PublicAPI

    Codes -->|lookup| Parsing
    RefData -->|lookup| Parsing
    Exhibits -->|lookup| Parsing
    Tickers -->|lookup| Parsing

    PyLibs --> HTTPClient
    PyLibs --> SGMLParser
    PyLibs --> HTMLExtract
    PyLibs --> XBRLParser
    PyLibs --> PandasDF

    %% Click events
    click CompanyAPI "https://github.com/dgunning/edgartools/blob/main/edgar/core.py"
    click FilingsAPI "https://github.com/dgunning/edgartools/blob/main/edgar/_filings.py"
    click SetIdentity "https://github.com/dgunning/edgartools/blob/main/edgar/company_reports.py"
    click HTTPClient "https://github.com/dgunning/edgartools/blob/main/edgar/httpclient.py"
    click HTTPRequests "https://github.com/dgunning/edgartools/blob/main/edgar/httprequests.py"
    click HTTPCache "https://github.com/dgunning/edgartools/blob/main/edgar/httpclient_cache.py"
    click CacheStorage "https://github.com/dgunning/edgartools/blob/main/edgar/storage.py"
    click LocalStorage "https://github.com/dgunning/edgartools/tree/main/data/localstorage/"
    click SGMLParser "https://github.com/dgunning/edgartools/blob/main/edgar/sgml/sgml_parser.py"
    click HTMLExtract "https://github.com/dgunning/edgartools/blob/main/edgar/files/html.py"
    click Markdown "https://github.com/dgunning/edgartools/blob/main/edgar/files/markdown.py"
    click Tables "https://github.com/dgunning/edgartools/blob/main/edgar/files/tables.py"
    click TextParse "https://github.com/dgunning/edgartools/blob/main/edgar/files/text.py"
    click XBRLParser "https://github.com/dgunning/edgartools/blob/main/edgar/xbrl/parser.py"
    click XBRLCore "https://github.com/dgunning/edgartools/blob/main/edgar/xbrl/core.py"
    click XBRLStd "https://github.com/dgunning/edgartools/blob/main/edgar/xbrl/standardization/core.py"
    click XBRLStitch "https://github.com/dgunning/edgartools/blob/main/edgar/xbrl/stitching/core.py"
    click XBRLUtils "https://github.com/dgunning/edgartools/blob/main/edgar/xbrl/stitching/utils.py"
    click XBRLXbrls "https://github.com/dgunning/edgartools/blob/main/edgar/xbrl/stitching/xbrls.py"
    click XBRLRender "https://github.com/dgunning/edgartools/blob/main/edgar/xbrl/rendering.py"
    click XBRLAnalysis "https://github.com/dgunning/edgartools/blob/main/edgar/xbrl/analysis/metrics.py"
    click CompanyModel "https://github.com/dgunning/edgartools/blob/main/edgar/entity/core.py"
    click FundModel "https://github.com/dgunning/edgartools/blob/main/edgar/funds/core.py"
    click OwnershipModel "https://github.com/dgunning/edgartools/blob/main/edgar/ownership/core.py"
    click OfferingsModel "https://github.com/dgunning/edgartools/blob/main/edgar/offerings/formd.py"
    click PandasDF "https://github.com/dgunning/edgartools/blob/main/edgar/datatools.py"
    click NLPText "https://github.com/dgunning/edgartools/blob/main/edgar/_markdown.py"
    click DataSearch "https://github.com/dgunning/edgartools/blob/main/edgar/search/datasearch.py"
    click TextSearch "https://github.com/dgunning/edgartools/blob/main/edgar/search/textsearch.py"
    click Codes "https://github.com/dgunning/edgartools/blob/main/edgar/reference/_codes.py"
    click RefData "https://github.com/dgunning/edgartools/tree/main/edgar/reference/data/"
    click Exhibits "https://github.com/dgunning/edgartools/blob/main/edgar/reference/exhibits.py"
    click Tickers "https://github.com/dgunning/edgartools/blob/main/edgar/reference/tickers.py"

    %% Styles
    classDef external fill:#CCCCCC,stroke:#333,stroke-width:1px
    classDef client fill:#EFEFEF,stroke:#333,stroke-width:1px
    classDef api fill:#A3C4FF,stroke:#333,stroke-width:1px
    classDef infra fill:#BFFCC6,stroke:#333,stroke-width:1px
    classDef parsing fill:#FFD699,stroke:#333,stroke-width:1px
    classDef domain fill:#C8C8FF,stroke:#333,stroke-width:1px
    classDef output fill:#FFE699,stroke:#333,stroke-width:1px
    classDef search fill:#E9CFFF,stroke:#333,stroke-width:1px
    classDef reference fill:#F0D9FF,stroke:#333,stroke-width:1px
```
