How giangchicken/firm_segmentation works
Project type and purpose
• This is a data‐processing & machine‐learning pipeline, not a typical web or full‐stack app.
• Purpose: segment (cluster) enterprises by crawling their official website text, encoding it with SBERT, building a similarity graph, and applying graph‐based clustering (e.g., Louvain) to output enterprise groups.
File structure
Top‐level directories and scripts:
• config/ – environment variables (config.env)
• data/ – raw HTML files and a mapping.json for source data
• src/ – modular pipeline code, split into embedding and similarity_calculation packages
• model/ – graph‐based processing: edge‐creation rules, clustering, top‐k neighbor selection
• embeddings/ – precomputed SBERT embeddings and metadata (numpy arrays, parquet)
• similarity_matrix/ – cached similarity matrices in parquet
• output/ – final JSON and parquet results
• main.py & init.ipynb – orchestration entry points
• docs/README.md & requirements.txt
README for additional insights
• Confirms 7‐step workflow: crawl → preprocess → embedding → similarity → graph → clustering → export
• Highlights core technologies: HTML parsing, SBERT, cosine similarity, Louvain/community detection.