# dmmiller612/bert-extractive-summarizer architecture

> Architecture diagram of the GitHub repository dmmiller612/bert-extractive-summarizer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dmmiller612/bert-extractive-summarizer
- Repository: https://github.com/dmmiller612/bert-extractive-summarizer

## Overview

##### Project type and purpose
- This is a Python library plus optional REST service for extractive text summarization using BERT (and SBERT) embeddings, clustering, and coreference resolution.
- It provides both an importable “Summarizer” API (summarizer/…) and a Flask‐based HTTP service (server.py + Dockerfile.service).

##### File structure
- Top‐level:
– setup.py, setup.cfg, requirements.txt: packaging and installable library
– server.py, Dockerfile.service, Makefile: dockerized Flask service for HTTP summarization
– examples/, colab/: sample usages and Jupyter notebook demo
– .github/workflows: CI for tests and publishing
- summarizer/ package:
– bert.py, sbert.py: wrappers around HuggingFace model loading and embedding extraction
– summary_processor.py: clustering and sentence selection logic
– text_processors/: sentence handlers and coreference resolution (neuralcoref + spaCy)
– transformer_embeddings/: encapsulates embedding extraction per model type
– util.py, cluster_features.py: helper functions for clustering and feature calculations
- tests/: pytest modules covering core pipeline components

##### README for additional insights
- Describes algorithm: sentence embedding → clustering → choose nearest‐to‐centroid sentences
- Mentions external dependencies: HuggingFace Transformers, Sentence Transformers, neuralcoref, spaCy
- Outlines the HTTP API: POST /summarize with query params for ratio, min_length, max_length

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Layer
    subgraph "Client Layer"
        LIB["Python Summarizer API"]:::internal
        EX1["Example Scripts"]:::io
        EX2["Colab Notebook"]:::io
        HTTPCLIENT["HTTP Client"]:::io
    end

    %% API Service Layer
    subgraph "API Service Layer"
        SERVER["Flask Service (server.py)"]:::io
    end

    %% Summarization Engine
    subgraph "Summarization Engine"
        subgraph "Text Processing"
            CORE["Coreference Resolution Handler"]:::internal
            SENTABC["Sentence ABC"]:::internal
            SENTSPLIT["Sentence Handler"]:::internal
        end
        subgraph "Embedding Extraction"
            BERTAPI["BertEmbedding API"]:::internal
            BERTIMP["BERT Model Wrapper"]:::internal
            SBERTAPI["SBertEmbedding API"]:::internal
            SBERTIMP["SBERT Model Wrapper"]:::internal
        end
        FEATURE["Clustering Feature Computation"]:::internal
        PROCESS["Summary Processor"]:::internal
        UTIL["Utility Functions"]:::internal
    end

    %% External Dependencies
    subgraph "External Dependencies"
        HUB["HuggingFace Model Hub"]:::external
        SPACY["spaCy + neuralcoref"]:::external
    end

    %% Packaging & CI/CD
    subgraph "Packaging & CI/CD"
        subgraph "Packaging & Dependencies"
            SETUP["setup.py"]:::internal
            SETUPCFG["setup.cfg"]:::internal
            REQ["requirements.txt"]:::internal
            REQSVC["requirements-service.txt"]:::internal
            MAKE["Makefile"]:::internal
            DOCKER["Dockerfile.service"]:::io
        end
        subgraph "CI/CD Workflows"
            TESTYML["test.yml"]:::internal
            PUBLISHYML["publish.yml"]:::internal
        end
    end

    %% Connections
    EX1 -->|imports| LIB
    EX2 -->|imports| LIB
    HTTPCLIENT -->|HTTP POST| SERVER
    LIB -->|function call| SERVER

    SERVER -->|raw text| CORE
    CORE -->|resolved text| SENTABC
    SENTABC -->|sentences| SENTSPLIT
    SENTSPLIT -->|sentences| BERTAPI
    SENTSPLIT -->|sentences| SBERTAPI
    BERTAPI -->|tokenized input| BERTIMP
    SBERTAPI -->|tokenized input| SBERTIMP
    UTIL -->|pooling| FEATURE
    BERTIMP -->|embeddings| FEATURE
    SBERTIMP -->|embeddings| FEATURE
    FEATURE -->|features| PROCESS
    PROCESS -->|summary| SERVER
    HUB -->|model files| BERTIMP
    HUB -->|model files| SBERTIMP
    SPACY -->|pipeline| CORE

    SETUP -->|defines| LIB
    SETUPCFG -->|configures| LIB
    REQ -->|deps for API| LIB
    REQSVC -->|deps for service| SERVER
    MAKE -->|build/run| SERVER
    DOCKER -->|container| SERVER

    TESTYML -->|CI tests| LIB
    TESTYML -->|CI tests| SERVER
    PUBLISHYML -->|publish package| LIB

    %% Click Events
    click LIB "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/__init__.py"
    click SERVER "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/server.py"
    click EX1 "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/examples/summarize.py"
    click EX2 "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/colab/summary_gpu.ipynb"
    click SETUP "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/setup.py"
    click SETUPCFG "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/setup.cfg"
    click REQ "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/requirements.txt"
    click REQSVC "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/requirements-service.txt"
    click MAKE "https://github.com/dmmiller612/bert-extractive-summarizer/tree/master/Makefile"
    click DOCKER "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/Dockerfile.service"
    click TESTYML "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/.github/workflows/test.yml"
    click PUBLISHYML "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/.github/workflows/publish.yml"
    click CORE "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/text_processors/coreference_handler.py"
    click SENTABC "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/text_processors/sentence_abc.py"
    click SENTSPLIT "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/text_processors/sentence_handler.py"
    click BERTAPI "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/bert.py"
    click BERTIMP "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/transformer_embeddings/bert_embedding.py"
    click SBERTAPI "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/sbert.py"
    click SBERTIMP "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/transformer_embeddings/sbert_embedding.py"
    click FEATURE "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/cluster_features.py"
    click PROCESS "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/summary_processor.py"
    click UTIL "https://github.com/dmmiller612/bert-extractive-summarizer/blob/master/summarizer/util.py"

    %% Styles
    classDef internal fill:#D6EAF8,stroke:#21618C
    classDef external fill:#D5F5E3,stroke:#196F3D
    classDef io fill:#FDEDEC,stroke:#CD6155
```
