# alions7000/sec-edgar-text architecture

> Architecture diagram of the GitHub repository alions7000/sec-edgar-text, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alions7000/sec-edgar-text
- Repository: https://github.com/alions7000/sec-edgar-text

## Overview

##### Project type and purpose
- This is a Python-based, command-line, batch-processing tool whose job is to fetch SEC EDGAR filings, pull out specified text sections, and store both the text and associated metadata on disk for downstream NLP or human inspection.

##### Main components (and how to represent them
a. External Service
- SEC EDGAR HTTP API
b. CLI / Orchestration Layer
- __main__.py & control.py: parse command‐line flags (via utils.py), kick off batches, coordinate multiprocessing.
- As a rectangle labeled “CLI Controller” or “Control Module”.
c. Configuration / Utilities
- utils.py: argument parsing, rate-limiting, storage path resolution, logging setup.
- metadata.py: builds/stores metadata JSON for each excerpt.
d. Download Manager
- download.py: orchestrates HTTP requests to EDGAR, respects rate limits and multiprocessing.
e. Document Parsers
- html_document.py & text_document.py: interpret raw HTML/Text, locate the configured sections from document_group_section_search.json, extract snippets.
- As a processing pipeline: HTTP response → HTML Parser → Text Extraction.
f. Storage
- File System: hierarchical directories (batch_XXXX/companyXXX/…); write text excerpts and metadata JSON files.
g. Logging / Error Handling
- Integrated via utils.py and control.py; logs failures, retries.

##### Relationships and Data Flow
- User runs “python SEC-EDGAR-text” → CLI Controller
- CLI Controller uses Utils for parsing args & setting up environment
- For each company/date/filing type:
- CLI Controller calls Download Manager → SEC EDGAR → HTTP response
- Pass response to Document Parser → extract text segments
- Pass extracted segments to Storage → writes excerpt.txt + metadata.json
- Any errors funnel back into Logger and optionally to retry logic
- Optionally use Python’s multiprocessing Pool to parallelize downloads/parsing

##### Architectural patterns and design principles
- Pipeline pattern: download → parse → store
- Separation of concerns:
- Orchestration (control.py)
- Downloading (download.py)
- Parsing (html/text_document.py)
- Metadata (metadata.py)
- CLI/config (utils.py)

##### Essential elements
- Clear component labels (e.g., Download Manager, HTML Parser, File System)
- Notes on optional multiprocessing and configurable JSON

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Service
    SEC["SEC EDGAR"]:::external

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        Main["__main__.py"]:::orchestration
        Control["Control Module"]:::orchestration
    end

    %% Configuration / Utilities
    subgraph "Config & Utils"
        Utils["Utils\n(argument parsing,\nrate-limiting, logging)"]:::support
        Metadata["Metadata Builder\n(metadata.py)"]:::support
        Config["Section Config\n(document_group_section_search.json)"]:::support
    end

    %% Processing Layers
    subgraph "Processing Pipeline"
        subgraph "Multiprocessing Pool" 
            DownloadMgr["Download Manager\n(download.py)"]:::processing
            subgraph "Parser Layer"
                BaseDoc["Document Base\n(document.py)"]:::processing
                HTMLParser["HTML Parser\n(html_document.py)"]:::processing
                TextParser["Text Parser\n(text_document.py)"]:::processing
            end
        end
    end

    %% Storage
    Storage["Local Storage\n(File System)"]:::storage

    %% Logger
    Logger["Logger"]:::support

    %% Data Flow
    Main -->|"parse args"| Utils
    Utils --> Control
    Control -->|"start batches"| DownloadMgr
    DownloadMgr -->|"HTTPS/REST GET"| SEC
    SEC -->|"HTTP Response"| DownloadMgr
    DownloadMgr -->|"raw response"| BaseDoc
    BaseDoc --> HTMLParser
    HTMLParser --> TextParser
    TextParser -->|"extracted text"| Storage
    TextParser -->|"pass metadata info"| Metadata
    Metadata -->|"write metadata JSON"| Storage

    %% Error Handling
    DownloadMgr --> Logger
    HTMLParser --> Logger
    TextParser --> Logger
    Control --> Logger
    Utils --> Logger

    %% Click Events
    click Main "https://github.com/alions7000/sec-edgar-text/blob/master/./__main__.py"
    click Control "https://github.com/alions7000/sec-edgar-text/blob/master/src/control.py"
    click Utils "https://github.com/alions7000/sec-edgar-text/blob/master/src/utils.py"
    click Metadata "https://github.com/alions7000/sec-edgar-text/blob/master/src/metadata.py"
    click Config "https://github.com/alions7000/sec-edgar-text/blob/master/document_group_section_search.json"
    click DownloadMgr "https://github.com/alions7000/sec-edgar-text/blob/master/src/download.py"
    click BaseDoc "https://github.com/alions7000/sec-edgar-text/blob/master/src/document.py"
    click HTMLParser "https://github.com/alions7000/sec-edgar-text/blob/master/src/html_document.py"
    click TextParser "https://github.com/alions7000/sec-edgar-text/blob/master/src/text_document.py"
    click Storage "https://github.com/alions7000/sec-edgar-text/tree/master/output_files_examples/"

    %% Styles
    classDef orchestration fill:#BBDEFB,stroke:#333,stroke-width:1px
    classDef processing fill:#90CAF9,stroke:#333,stroke-width:1px
    classDef support fill:#CFD8DC,stroke:#333,stroke-width:1px
    classDef external fill:#FFCC80,stroke:#333,stroke-width:1px
    classDef storage fill:#A5D6A7,stroke:#333,stroke-width:1px
```
