How unstructured-io/pipeline-sec-filings works
Project type and purpose
• This is a Python-based document pre-processing pipeline for SEC filings, packaged as a library (prepline_sec_filings) and exposed via a FastAPI web service. It also provides Jupyter notebook integration for exploration and a set of CLI scripts for development, testing, and Docker-based deployment.
File structure
• Top-level directories & files:
• .github/ – CI (GitHub Actions) and Dependabot config
• Dockerfile / Makefile / scripts/ – build, test, format, and Docker commands
• pipeline-notebooks/ & exploration-notebooks/ – Jupyter notebooks driving or exploring the pipeline
• prepline_sec_filings/ – core Python package with two subpackages:
– api/ – FastAPI app (app.py, section.py)
– pipeline modules (fetch.py, sec_document.py, sections.py)
• requirements/ – base, dev, and test dependencies
• test_* directories – unit and integration tests for each module and notebook consistency
• sample-docs/ – sample SEC documents for local testing
• logger_config.yaml – logging configuration
README for additional insights
• The README confirms the stack: Python 3.8, FastAPI for the /section endpoint, optional notebook interface, Docker support, and helper functions for interacting with the SEC EDGAR API.
• It describes how the pipeline extracts narrative text from 10-K, 10-Q, and S-1 filings and returns JSON, CSV, or LabelStudio-compatible output.
Main components
a. Client layer
• Users / scripts / notebooks / curl / SDK
b. API Gateway
• FastAPI service (prepline_sec_filings.api.app)
c. Pipeline Orchestrator
• Endpoint handler (section.py) that:
• Accepts multipart/form-data requests
• Invokes core pipeline modules
d. Core Pipeline Modules
• fetch.py – retrieves filings from the SEC EDGAR API or local files
• sec_document.py – parses raw iXBRL/HTML into a document object model
• sections.py – applies section definitions or regex to extract narrative text
• serializer – formats output as JSON, CSV, or LabelStudio schema
e. External Services
• SEC EDGAR API (cloud)
f. Data Repositories (optional)
• Local sample files (sample-docs/), fixtures for tests
g. Development & CI/CD
• Jupyter Notebooks (exploration & pipeline)
• GitHub Actions workflows (.github/workflows/)
• Docker (Build / Start API / Start Jupyter)
h. Tests
• Unit tests (test_sec_filings, test_utils)
• Integration tests (test_sec_filings_integration, notebook consistency)
Relationships and interactions
• Client → FastAPI endpoint (HTTP POST /sec-filings/.../section)
• API → Pipeline Orchestrator → Core Modules in sequence:
1. fetch.py pulls document (from local or EDGAR API)
2. sec_document.py parses the raw filing
3. sections.py extracts requested sections or regex matches
4. serializer formats and returns response
• Core Modules → External SEC EDGAR API (when fetch.py uses get_filing or get_form_by_ticker)
• CI/CD pipelines automatically run tests, format checks, and security scans on push or PR
Architectural patterns and design principles
• Modular separation of concerns: fetch, parse, extract, serialize
• Pipeline pattern: linear flow of data through discrete transformation stages
• Layered architecture: Client → API layer → Business logic (pipeline modules) → External systems
• Single codebase with clear directory boundaries for API, core logic, notebooks, and tests
• DevOps integration: Makefile, Dockerfile, GitHub Actions for reproducible builds and deployments