How alphanome-ai/sec-parser works
Core Purpose and Project Type
• Purpose: sec-parser is a library that parses SEC EDGAR HTML filings and converts them into a semantic tree of elements (such as titles, tables, and paragraphs). It is used for financial data analysis—in AI, machine learning, and regulatory data processing.
• Project Type: Although it is predominantly a processing library, the project also provides:
• A dashboard application in the dev_utils/dashboard_app folder (leveraging Streamlit for visualization).
• Extensive tests (unit, integration, snapshot, etc.) and CI/CD pipelines in the .github/workflows folder.
• Developer and user documentation (docs and readthedocs configs).
Thus, you have core processing components, a UI demonstration, and supporting infrastructure for CI, documentation, and testing.
Main Components
a. Core Parser Library (sec_parser package – main engine)
• Internal modules:
• processing_engine: Handles the core parsing routines (e.g., parsing HTML tags, logging, type definitions).
• processing_steps: Contains various classification and element extraction steps (like text classifiers, image classifiers, and table classifiers).
• semantic_elements: Defines the different kinds of parsed elements (titles, highlighted texts, table elements, etc.) along with mixins.
• semantic_tree: Builds the logical tree structure from parsed semantic elements.
• utils: Contains helper functions such as those based on BeautifulSoup (bs4_) and environment variable helpers.
b. Dashboard / Demo Application (dev_utils/dashboard_app)
• Provides a web-based interface (using Streamlit) that uses the sec-parser library to render semantic trees for interactive visualization.
• Contains modules to download metadata, select reports, and display parsed content.
c. External Integration
• SEC Downloader (mentioned in the README and used in examples) is used to fetch HTML filings from SEC EDGAR. Represent it as an external input that feeds HTML documents to the parser.
d. Infrastructure & Testing
• CI/CD Pipelines: Represent the GitHub workflows (ci.yml and cd.yml) that run tests, code quality checks (mypy, black, pre-commit, ruff) and deployment processes.
• Tests: Unit tests, integration tests, and exploratory tests (found in tests/) ensure validity across all processing steps and core functionalities.
• Containerization & Dev Environment: Note the .devcontainer configuration for an isolated development environment.
e. Documentation & Community Components
• Documentation: Generated via readthedocs (using .readthedocs.yaml and docs/).
• Supporting files like LICENSE, CONTRIBUTING.md, and badges indicate open-source quality and collaboration aspects.
Depicting Relationships and Data Flow
• Processing pipeline: HTML input → Edgar10QParser (or similar) → processing_engine & processing_steps → formation of semantic_elements → tree building by TreeBuilder (in the semantic_tree module).
• That the final parsed semantic tree is used by the dashboard application (displayed via Streamlit) and can be exported or rendered.
• Testing infrastructure to both the core parser and the dashboard to indicate automated regression and integration tests.
Architectural Patterns
• Utilize a pipeline pattern: Data flows through multiple processing "stages" (initial parsing, step-wise classification, and tree construction).
• Modularity and separation of concerns. Each major submodule in the sec_parser package is responsible for a well-defined part of the parsing task.
• Usage of common frameworks (like BeautifulSoup for HTML parsing, streamlit for UI, and integration with GitHub workflows for automation).