# lefterisloukas/edgar-crawler architecture

> Architecture diagram of the GitHub repository lefterisloukas/edgar-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lefterisloukas/edgar-crawler
- Repository: https://github.com/lefterisloukas/edgar-crawler

## Overview

EDGAR-CRAWLER supports two user workflows: download filtered SEC filings and parse downloaded 10-K, 10-Q, and 8-K filings into structured JSON. The download script orchestrates index retrieval, filtering, filing metadata enrichment, and raw downloads; the extraction script reads configured inputs and extracts report sections using filing-specific item lists. The excerpts establish these entry points and core responsibilities, but omit some implementation details, so uncertain internal call links are left out.

Main entry points: [`extract_items.py`](https://github.com/lefterisloukas/edgar-crawler/blob/main/extract_items.py) (Extraction workflow), [`download_filings.py`](https://github.com/lefterisloukas/edgar-crawler/blob/main/download_filings.py) (Download workflow), [`datasets`](https://github.com/lefterisloukas/edgar-crawler/tree/main/datasets) (Raw filings), [`config.json`](https://github.com/lefterisloukas/edgar-crawler/blob/main/config.json) (Workflow configuration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_download["Filing download"]
  node_download_cli["Download workflow"]
  node_index_filter["Index retrieval and filtering"]
  node_filing_crawl["Filing crawl and enrichment"]
  node_raw_filings["Raw filings"]
  node_filing_metadata["Filing metadata"]
end

subgraph group_extract["Section extraction"]
  node_extract_cli["Extraction workflow<br/>[extract_items.py]"]
  node_section_parser["Section extraction<br/>[extract_items.py]"]
  node_item_catalog["Filing item lists<br/>[item_lists.py]"]
  node_json_output["Structured JSON"]
end

subgraph group_runtime["Shared runtime"]
  node_configuration["Workflow configuration<br/>[config.json]"]
  node_logging["Logging<br/>[logger.py]"]
end

node_researcher(("Researcher"))
node_sec{{"SEC EDGAR"}}

node_researcher -->|"runs download"| node_download_cli
node_researcher -->|"runs extraction"| node_extract_cli
node_download_cli -->|"reads settings"| node_configuration
node_download_cli -->|"orchestrates"| node_index_filter
node_index_filter -->|"requests indices"| node_sec
node_index_filter -->|"selects filings"| node_filing_crawl
node_filing_crawl -->|"requests filings"| node_sec
node_filing_crawl -->|"writes filings"| node_raw_filings
node_filing_crawl -->|"writes metadata"| node_filing_metadata
node_extract_cli -->|"reads settings"| node_configuration
node_extract_cli -->|"reads metadata"| node_filing_metadata
node_extract_cli -->|"reads filings"| node_raw_filings
node_extract_cli -->|"invokes parser"| node_section_parser
node_section_parser -->|"uses item lists"| node_item_catalog
node_section_parser -->|"writes JSON"| node_json_output
node_download_cli -->|"logs activity"| node_logging
node_extract_cli -->|"logs activity"| node_logging

click node_download_cli "https://github.com/lefterisloukas/edgar-crawler/blob/main/download_filings.py"
click node_index_filter "https://github.com/lefterisloukas/edgar-crawler/blob/main/download_filings.py"
click node_filing_crawl "https://github.com/lefterisloukas/edgar-crawler/blob/main/download_filings.py"
click node_raw_filings "https://github.com/lefterisloukas/edgar-crawler/tree/main/datasets"
click node_filing_metadata "https://github.com/lefterisloukas/edgar-crawler/tree/main/datasets"
click node_extract_cli "https://github.com/lefterisloukas/edgar-crawler/blob/main/extract_items.py"
click node_section_parser "https://github.com/lefterisloukas/edgar-crawler/blob/main/extract_items.py"
click node_item_catalog "https://github.com/lefterisloukas/edgar-crawler/blob/main/item_lists.py"
click node_json_output "https://github.com/lefterisloukas/edgar-crawler/tree/main/datasets"
click node_configuration "https://github.com/lefterisloukas/edgar-crawler/blob/main/config.json"
click node_logging "https://github.com/lefterisloukas/edgar-crawler/blob/main/logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_download_cli,node_index_filter,node_filing_crawl,node_raw_filings,node_filing_metadata toneBlue
class node_extract_cli,node_section_parser,node_item_catalog,node_json_output toneAmber
class node_configuration,node_logging toneMint
class node_researcher,node_sec toneIndigo
```

## Components

### Filing download

- [Download workflow](https://github.com/lefterisloukas/edgar-crawler/blob/main/download_filings.py): `download_filings.py`
- [Index retrieval and filtering](https://github.com/lefterisloukas/edgar-crawler/blob/main/download_filings.py): `download_filings.py`
- [Filing crawl and enrichment](https://github.com/lefterisloukas/edgar-crawler/blob/main/download_filings.py): `download_filings.py`
- [Raw filings](https://github.com/lefterisloukas/edgar-crawler/tree/main/datasets): `datasets`
- [Filing metadata](https://github.com/lefterisloukas/edgar-crawler/tree/main/datasets): `datasets`

### Section extraction

- [Extraction workflow](https://github.com/lefterisloukas/edgar-crawler/blob/main/extract_items.py): `extract_items.py`
- [Section extraction](https://github.com/lefterisloukas/edgar-crawler/blob/main/extract_items.py): `extract_items.py`
- [Filing item lists](https://github.com/lefterisloukas/edgar-crawler/blob/main/item_lists.py): `item_lists.py`
- [Structured JSON](https://github.com/lefterisloukas/edgar-crawler/tree/main/datasets): `datasets`

### Shared runtime

- [Workflow configuration](https://github.com/lefterisloukas/edgar-crawler/blob/main/config.json): `config.json`
- [Logging](https://github.com/lefterisloukas/edgar-crawler/blob/main/logger.py): `logger.py`

### Other components

- **Researcher**
- **SEC EDGAR**

## Connections

- Researcher → Download workflow: runs download
- Researcher → Extraction workflow: runs extraction
- Download workflow → Workflow configuration: reads settings
- Download workflow → Index retrieval and filtering: orchestrates
- Index retrieval and filtering → SEC EDGAR: requests indices
- Index retrieval and filtering → Filing crawl and enrichment: selects filings
- Filing crawl and enrichment → SEC EDGAR: requests filings
- Filing crawl and enrichment → Raw filings: writes filings
- Filing crawl and enrichment → Filing metadata: writes metadata
- Extraction workflow → Workflow configuration: reads settings
- Extraction workflow → Filing metadata: reads metadata
- Extraction workflow → Raw filings: reads filings
- Extraction workflow → Section extraction: invokes parser
- Section extraction → Filing item lists: uses item lists
- Section extraction → Structured JSON: writes JSON
- Download workflow → Logging: logs activity
- Extraction workflow → Logging: logs activity