# unstructured-io/pipeline-sec-filings architecture

> Architecture diagram of the GitHub repository unstructured-io/pipeline-sec-filings, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/unstructured-io/pipeline-sec-filings
- Repository: https://github.com/unstructured-io/pipeline-sec-filings

## Overview

##### Project type and purpose
- This is a Python-based document pre-processing pipeline for SEC filings, packaged as a library (`prepline_sec_filings`) and exposed via a FastAPI web service. It also provides Jupyter notebook integration for exploration and a set of CLI scripts for development, testing, and Docker-based deployment.

##### File structure
- Top-level directories & files:
- `.github/` – CI (GitHub Actions) and Dependabot config
- `Dockerfile` / `Makefile` / `scripts/` – build, test, format, and Docker commands
- `pipeline-notebooks/` & `exploration-notebooks/` – Jupyter notebooks driving or exploring the pipeline
- `prepline_sec_filings/` – core Python package with two subpackages:
– `api/` – FastAPI app (`app.py`, `section.py`)
– pipeline modules (`fetch.py`, `sec_document.py`, `sections.py`)
- `requirements/` – base, dev, and test dependencies
- `test_*` directories – unit and integration tests for each module and notebook consistency
- `sample-docs/` – sample SEC documents for local testing
- `logger_config.yaml` – logging configuration

##### README for additional insights
- The README confirms the stack: Python 3.8, FastAPI for the `/section` endpoint, optional notebook interface, Docker support, and helper functions for interacting with the SEC EDGAR API.
- It describes how the pipeline extracts narrative text from 10-K, 10-Q, and S-1 filings and returns JSON, CSV, or LabelStudio-compatible output.

##### Main components
a. Client layer
- Users / scripts / notebooks / `curl` / SDK
b. API Gateway
- FastAPI service (`prepline_sec_filings.api.app`)
c. Pipeline Orchestrator
- Endpoint handler (`section.py`) that:
- Accepts multipart/form-data requests
- Invokes core pipeline modules
d. Core Pipeline Modules
- fetch.py – retrieves filings from the SEC EDGAR API or local files
- sec_document.py – parses raw iXBRL/HTML into a document object model
- sections.py – applies section definitions or regex to extract narrative text
- serializer – formats output as JSON, CSV, or LabelStudio schema
e. External Services
- SEC EDGAR API (cloud)
f. Data Repositories (optional)
- Local sample files (`sample-docs/`), fixtures for tests
g. Development & CI/CD
- Jupyter Notebooks (exploration & pipeline)
- GitHub Actions workflows (`.github/workflows/`)
- Docker (Build / Start API / Start Jupyter)
h. Tests
- Unit tests (`test_sec_filings`, `test_utils`)
- Integration tests (`test_sec_filings_integration`, notebook consistency)

##### Relationships and interactions
- Client → FastAPI endpoint (HTTP POST `/sec-filings/.../section`)
- API → Pipeline Orchestrator → Core Modules in sequence:
1. fetch.py pulls document (from local or EDGAR API)
2. sec_document.py parses the raw filing
3. sections.py extracts requested sections or regex matches
4. serializer formats and returns response
- Core Modules → External SEC EDGAR API (when fetch.py uses `get_filing` or `get_form_by_ticker`)
- CI/CD pipelines automatically run tests, format checks, and security scans on push or PR

##### Architectural patterns and design principles
- Modular separation of concerns: fetch, parse, extract, serialize
- Pipeline pattern: linear flow of data through discrete transformation stages
- Layered architecture: Client → API layer → Business logic (pipeline modules) → External systems
- Single codebase with clear directory boundaries for API, core logic, notebooks, and tests
- DevOps integration: Makefile, Dockerfile, GitHub Actions for reproducible builds and deployments

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Client Layer"
      E1[/ "Jupyter Notebooks" /]:::client
      click E1 "https://github.com/unstructured-io/pipeline-sec-filings/tree/main/exploration-notebooks/"
      E2[/ "Pipeline Notebooks" /]:::client
      click E2 "https://github.com/unstructured-io/pipeline-sec-filings/tree/main/pipeline-notebooks/"
      E3[/ "CLI Scripts" /]:::client
      click E3 "https://github.com/unstructured-io/pipeline-sec-filings/tree/main/scripts/"
    end

    subgraph "API Gateway"
      A1["FastAPI App\nprepline_sec_filings/api/app.py"]:::api
      click A1 "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/prepline_sec_filings/api/app.py"
      A2["Section Endpoint\nprepline_sec_filings/api/section.py"]:::api
      click A2 "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/prepline_sec_filings/api/section.py"
    end

    subgraph "Pipeline"
      P1["Fetch Module\nprepline_sec_filings/fetch.py"]:::pipeline
      click P1 "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/prepline_sec_filings/fetch.py"
      P2["Document Parser\nprepline_sec_filings/sec_document.py"]:::pipeline
      click P2 "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/prepline_sec_filings/sec_document.py"
      P3["Sections Extractor\nprepline_sec_filings/sections.py"]:::pipeline
      click P3 "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/prepline_sec_filings/sections.py"
      Serializer["Serializer\nformats JSON/CSV"]:::pipeline
    end

    EC["SEC EDGAR API"]:::external
    DS["Local Sample Docs"]:::datastore
    click DS "https://github.com/unstructured-io/pipeline-sec-filings/tree/main/sample-docs/"

    subgraph "DevOps & CI"
      CI1["ci.yml"]:::devops
      click CI1 "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/.github/workflows/ci.yml"
      CI2["codeql-analysis.yml"]:::devops
      click CI2 "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/.github/workflows/codeql-analysis.yml"
      Dockerfile["Dockerfile"]:::devops
      click Dockerfile "https://github.com/unstructured-io/pipeline-sec-filings/tree/main/Dockerfile"
      Makefile["Makefile"]:::devops
      click Makefile "https://github.com/unstructured-io/pipeline-sec-filings/tree/main/Makefile"
      DCsh["docker-build.sh"]:::devops
      click DCsh "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/scripts/docker-build.sh"
      ReqBase["requirements/base.*"]:::devops
      click ReqBase "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/requirements/base.*"
      ReqDev["requirements/dev.*"]:::devops
      click ReqDev "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/requirements/dev.*"
      ReqTest["requirements/test.*"]:::devops
      click ReqTest "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/requirements/test.*"
      subgraph "Tests"
        UT["Unit Tests"]:::tests
        click UT "https://github.com/unstructured-io/pipeline-sec-filings/tree/main/test_sec_filings/"
        click UT "https://github.com/unstructured-io/pipeline-sec-filings/tree/main/test_utils/"
        IT["Integration Tests"]:::tests
        click IT "https://github.com/unstructured-io/pipeline-sec-filings/tree/main/test_sec_filings_integration/"
        click IT "https://github.com/unstructured-io/pipeline-sec-filings/tree/main/test_real_docs/"
        NT["Notebook Tests"]:::tests
        click NT "https://github.com/unstructured-io/pipeline-sec-filings/blob/main/test_sec_filings_integration/test_notebooks.py"
      end
    end

    E1 -->|HTTP POST /section| A2
    E2 -->|HTTP POST /section| A2
    E3 -->|HTTP POST /section| A2
    A1 --> A2
    A2 -->|"Invokes"| P1
    P1 -->|"Raw Data"| P2
    P2 -->|"Parsed Document"| P3
    P3 -->|"Sections Data"| Serializer
    Serializer -->|Response JSON/CSV| A2
    P1 -->|Fetch API| EC
    A2 -->|HTTP Response| E1

    classDef client fill:#f9e79f,stroke:#d4ac0d,color:#6e2c00,stroke-width:2px;
    classDef api fill:#aed6f1,stroke:#2e86c1,color:#154360,stroke-width:2px;
    classDef pipeline fill:#abebc6,stroke:#28b463,color:#1e4620,stroke-width:2px;
    classDef external fill:#fadbd8,stroke:#c0392b,color:#641e16,stroke-width:2px,stroke-dasharray: 5 5;
    classDef datastore fill:#e8daef,stroke:#8e44ad,color:#4a235a,stroke-width:2px;
    classDef devops fill:#d5dbdb,stroke:#566573,color:#1c2833,stroke-width:2px;
    classDef tests fill:#f5b7b1,stroke:#922b21,color:#641e16,stroke-width:2px;
```
