# giangchicken/firm_segmentation architecture

> Architecture diagram of the GitHub repository giangchicken/firm_segmentation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/giangchicken/firm_segmentation
- Repository: https://github.com/giangchicken/firm_segmentation

## Overview

##### Project type and purpose
- This is a data‐processing & machine‐learning pipeline, not a typical web or full‐stack app.
- Purpose: segment (cluster) enterprises by crawling their official website text, encoding it with SBERT, building a similarity graph, and applying graph‐based clustering (e.g., Louvain) to output enterprise groups.

##### File structure
Top‐level directories and scripts:
- config/ – environment variables (config.env)
- data/ – raw HTML files and a mapping.json for source data
- src/ – modular pipeline code, split into embedding and similarity_calculation packages
- model/ – graph‐based processing: edge‐creation rules, clustering, top‐k neighbor selection
- embeddings/ – precomputed SBERT embeddings and metadata (numpy arrays, parquet)
- similarity_matrix/ – cached similarity matrices in parquet
- output/ – final JSON and parquet results
- main.py & init.ipynb – orchestration entry points
- docs/README.md & requirements.txt

##### README for additional insights
- Confirms 7‐step workflow: crawl → preprocess → embedding → similarity → graph → clustering → export
- Highlights core technologies: HTML parsing, SBERT, cosine similarity, Louvain/community detection.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration and Orchestration
    Config["Config\nconfig/config.env"]:::config
    Main["Orchestration\nmain.py"]:::orchestration
    Notebook["Orchestration Notebook\ninit.ipynb"]:::orchestration

    %% Data Ingestion
    subgraph "Data Ingestion"
        direction TB
        DataRaw["Raw HTML Data\ndata/"]:::storage
        Mapping["Metadata\nmapping.json"]:::storage
    end

    %% Preprocessing
    Preprocessing["Preprocessing Module\nsrc/embedding/preprocessing.py"]:::compute

    %% Embedding Generation
    subgraph "Embedding Generation"
        direction TB
        EmbeddingMod["Embedding Module\nsrc/embedding/embedding.py"]:::compute
        EmbNpy["X_dev.npy\nembeddings/SBERT"]:::storage
        EmbSplit["split_info_dev.json\nembeddings/SBERT"]:::storage
        EmbParquet["dev_vectors.parquet\nembeddings"]:::storage
    end

    %% Similarity Calculation
    subgraph "Similarity Calculation"
        direction TB
        SimilarityMod["Similarity Module\nsrc/similarity_calculation/similarity_matrix.py"]:::compute
        SimParquet["part.0.parquet\nsimilarity_matrix"]:::storage
    end

    %% Graph Construction
    subgraph "Graph Construction"
        direction TB
        EdgeRule["Edge Creation\nmodel/create_edges_by_rule.py"]:::compute
        TopK["Top K Neighbors\nmodel/top_k_nearest_neighbors.py"]:::compute
    end

    %% Graph Clustering
    Clustering["Graph Clustering\nmodel/graph_clustering.py"]:::compute

    %% Output Generation
    subgraph "Output Artifacts"
        direction TB
        Out1["final_clusters.json\noutput"]:::storage
        Out2["inter_edges.json\noutput"]:::storage
        Out3["top_10_nearest_domain.parquet\noutput"]:::storage
    end

    %% Documentation and Dependencies
    Docs["Documentation\ndocs/README.md"]:::doc
    Req["Dependencies\nrequirements.txt"]:::doc

    %% Data Flow
    Config --> Main
    Config --> Notebook
    Main --> DataRaw
    Main --> Mapping
    Notebook --> DataRaw
    Notebook --> Mapping
    DataRaw --> Preprocessing
    Mapping --> Preprocessing
    Preprocessing --> EmbeddingMod
    EmbeddingMod --> EmbNpy
    EmbeddingMod --> EmbSplit
    EmbeddingMod --> EmbParquet
    EmbeddingMod --> SimilarityMod
    EmbParquet --> SimilarityMod
    SimilarityMod --> SimParquet
    SimilarityMod --> EdgeRule
    SimilarityMod --> TopK
    EdgeRule --> Clustering
    TopK --> Clustering
    Clustering --> Out1
    Clustering --> Out2
    Clustering --> Out3
    Docs --- Main
    Req --- Main

    %% Click Events
    click Config "https://github.com/giangchicken/firm_segmentation/blob/main/config/config.env"
    click Main "https://github.com/giangchicken/firm_segmentation/blob/main/main.py"
    click Notebook "https://github.com/giangchicken/firm_segmentation/blob/main/init.ipynb"
    click DataRaw "https://github.com/giangchicken/firm_segmentation/tree/main/data/"
    click Mapping "https://github.com/giangchicken/firm_segmentation/blob/main/data/mapping.json"
    click Preprocessing "https://github.com/giangchicken/firm_segmentation/blob/main/src/embedding/preprocessing.py"
    click EmbeddingMod "https://github.com/giangchicken/firm_segmentation/blob/main/src/embedding/embedding.py"
    click EmbNpy "https://github.com/giangchicken/firm_segmentation/blob/main/embeddings/SBERT/X_dev.npy"
    click EmbSplit "https://github.com/giangchicken/firm_segmentation/blob/main/embeddings/SBERT/split_info_dev.json"
    click EmbParquet "https://github.com/giangchicken/firm_segmentation/blob/main/embeddings/dev_vectors.parquet"
    click SimilarityMod "https://github.com/giangchicken/firm_segmentation/blob/main/src/similarity_calculation/similarity_matrix.py"
    click SimParquet "https://github.com/giangchicken/firm_segmentation/blob/main/similarity_matrix/part.0.parquet"
    click EdgeRule "https://github.com/giangchicken/firm_segmentation/blob/main/model/create_edges_by_rule.py"
    click TopK "https://github.com/giangchicken/firm_segmentation/blob/main/model/top_k_nearest_neighbors.py"
    click Clustering "https://github.com/giangchicken/firm_segmentation/blob/main/model/graph_clustering.py"
    click Out1 "https://github.com/giangchicken/firm_segmentation/blob/main/output/final_clusters.json"
    click Out2 "https://github.com/giangchicken/firm_segmentation/blob/main/output/inter_edges.json"
    click Out3 "https://github.com/giangchicken/firm_segmentation/blob/main/output/top_10_nearest_domain.parquet"
    click Docs "https://github.com/giangchicken/firm_segmentation/blob/main/docs/README.md"
    click Req "https://github.com/giangchicken/firm_segmentation/blob/main/requirements.txt"

    %% Styles
    classDef config fill:#FFA500,stroke:#333,stroke-width:1px;
    classDef orchestration fill:#FF8C00,stroke:#333,stroke-width:1px;
    classDef compute fill:#87CEFA,stroke:#333,stroke-width:1px;
    classDef storage fill:#98FB98,stroke:#333,stroke-width:1px;
    classDef doc fill:#D3D3D3,stroke:#333,stroke-width:1px;
```
