# alphanome-ai/sec-parser architecture

> Architecture diagram of the GitHub repository alphanome-ai/sec-parser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alphanome-ai/sec-parser
- Repository: https://github.com/alphanome-ai/sec-parser

## Overview

##### Core Purpose and Project Type
- Purpose: sec-parser is a library that parses SEC EDGAR HTML filings and converts them into a semantic tree of elements (such as titles, tables, and paragraphs). It is used for financial data analysis—in AI, machine learning, and regulatory data processing.
- Project Type: Although it is predominantly a processing library, the project also provides:
- A dashboard application in the dev_utils/dashboard_app folder (leveraging Streamlit for visualization).
- Extensive tests (unit, integration, snapshot, etc.) and CI/CD pipelines in the .github/workflows folder.
- Developer and user documentation (docs and readthedocs configs).
Thus, you have core processing components, a UI demonstration, and supporting infrastructure for CI, documentation, and testing.

##### Main Components
a. Core Parser Library (sec_parser package – main engine)
- Internal modules:
- processing_engine: Handles the core parsing routines (e.g., parsing HTML tags, logging, type definitions).
- processing_steps: Contains various classification and element extraction steps (like text classifiers, image classifiers, and table classifiers).
- semantic_elements: Defines the different kinds of parsed elements (titles, highlighted texts, table elements, etc.) along with mixins.
- semantic_tree: Builds the logical tree structure from parsed semantic elements.
- utils: Contains helper functions such as those based on BeautifulSoup (bs4_) and environment variable helpers.

b. Dashboard / Demo Application (dev_utils/dashboard_app)
- Provides a web-based interface (using Streamlit) that uses the sec-parser library to render semantic trees for interactive visualization.
- Contains modules to download metadata, select reports, and display parsed content.

c. External Integration
- SEC Downloader (mentioned in the README and used in examples) is used to fetch HTML filings from SEC EDGAR. Represent it as an external input that feeds HTML documents to the parser.

d. Infrastructure & Testing
- CI/CD Pipelines: Represent the GitHub workflows (ci.yml and cd.yml) that run tests, code quality checks (mypy, black, pre-commit, ruff) and deployment processes.
- Tests: Unit tests, integration tests, and exploratory tests (found in tests/) ensure validity across all processing steps and core functionalities.
- Containerization & Dev Environment: Note the .devcontainer configuration for an isolated development environment.

e. Documentation & Community Components
- Documentation: Generated via readthedocs (using .readthedocs.yaml and docs/).
- Supporting files like LICENSE, CONTRIBUTING.md, and badges indicate open-source quality and collaboration aspects.

##### Depicting Relationships and Data Flow
- Processing pipeline: HTML input → Edgar10QParser (or similar) → processing_engine & processing_steps → formation of semantic_elements → tree building by TreeBuilder (in the semantic_tree module).
- That the final parsed semantic tree is used by the dashboard application (displayed via Streamlit) and can be exported or rendered.
- Testing infrastructure to both the core parser and the dashboard to indicate automated regression and integration tests.

##### Architectural Patterns
- Utilize a pipeline pattern: Data flows through multiple processing "stages" (initial parsing, step-wise classification, and tree construction).
- Modularity and separation of concerns. Each major submodule in the sec_parser package is responsible for a well-defined part of the parsing task.
- Usage of common frameworks (like BeautifulSoup for HTML parsing, streamlit for UI, and integration with GitHub workflows for automation).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Integration
    SECDownloader["SEC Downloader - Fetches SEC EDGAR HTML filings"]:::external
    click SECDownloader "https://github.com/alphanome-ai/sec-parser/blob/main/dev_utils/core/sec_edgar_reports_getter.py"
    
    %% Core Library Node (represents sec-parser package)
    SecParserLib["sec-parser Library"]:::core
    click SecParserLib "https://github.com/alphanome-ai/sec-parser/tree/main/sec_parser/"
    
    %% Data Flow from SEC Downloader to Core Library
    SECDownloader -->|"providesHTMLfilings"| SecParserLib

    %% Core Modules Subgraph
    subgraph "Core Modules"
        PE["Processing Engine - Core Parsing Logic"]:::core
        PS["Processing Steps - Element Extraction & Classification"]:::core
        SE["Semantic Elements - Defines Parsed Elements"]:::core
        ST["Semantic Tree Builder - Constructs Logical Tree"]:::core
        UT["Utility Functions - Helper Methods with BeautifulSoup"]:::core
    end

    %% Link Core Library to its internal processing modules
    SecParserLib -->|"initiates"| PE
    PE -->|"parsesHTML"| PS
    PS -->|"extractsElements"| SE
    SE -->|"buildsSemanticTree"| ST

    %% Dashboard Application consumes the parsed output
    Dashboard["Dashboard Application - UI Demo via Streamlit"]:::ui
    click Dashboard "https://github.com/alphanome-ai/sec-parser/tree/main/dev_utils/dashboard_app/"
    ST -->|"outputsParsedTree"| Dashboard

    %% Infrastructure & Testing Subgraph
    subgraph "Infrastructure & Testing"
        CICD["CI/CD Pipelines - Automated Tests & Deployment"]:::infra
        Tests["Testing Suite - Unit/Integration/Snapshot Tests"]:::infra
        DevEnv["Dev Environment Containerization (.devcontainer)"]:::infra
        Docs["Documentation & ReadTheDocs Configurations"]:::infra
    end
    click CICD "https://github.com/alphanome-ai/sec-parser/tree/main/.github/workflows/"
    click Tests "https://github.com/alphanome-ai/sec-parser/tree/main/tests/"
    click DevEnv "https://github.com/alphanome-ai/sec-parser/tree/main/.devcontainer/"
    click Docs "https://github.com/alphanome-ai/sec-parser/tree/main/docs/"

    %% Feedback and Relationships between Components
    Dashboard -->|"leveragesParserLibrary"| SecParserLib
    Tests -->|"validates"| SecParserLib
    CICD -->|"executesTests"| Tests
    DevEnv -->|"supports"| SecParserLib
    Docs -->|"documents"| SecParserLib

    %% Click Events for Core Modules
    click PE "https://github.com/alphanome-ai/sec-parser/tree/main/sec_parser/processing_engine/"
    click PS "https://github.com/alphanome-ai/sec-parser/tree/main/sec_parser/processing_steps/"
    click SE "https://github.com/alphanome-ai/sec-parser/tree/main/sec_parser/semantic_elements/"
    click ST "https://github.com/alphanome-ai/sec-parser/tree/main/sec_parser/semantic_tree/"
    click UT "https://github.com/alphanome-ai/sec-parser/tree/main/sec_parser/utils/"

    %% Styles
    classDef core fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef ui fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef infra fill:#D5F5E3,stroke:#196F3D,stroke-width:2px;
    classDef external fill:#F5B7B1,stroke:#943126,stroke-width:2px;
```
