# hanyaaaaaa/dataanalysis architecture

> Architecture diagram of the GitHub repository hanyaaaaaa/dataanalysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hanyaaaaaa/dataanalysis
- Repository: https://github.com/hanyaaaaaa/dataanalysis

## Overview

##### Main Components
- Data Source/Input:
– The many case_XX_detail.html files (ranging from case_1_detail.html to case_404_detail.html and beyond) serve as primary input data.
– The materials folder and any additional CSV files (like importantcsv/import.csv) provide supplementary data.
– The chrome-headless-shell-win64 folder indicates that a headless browser may be used for web scraping or rendering dynamic pages. In addition, spiderr.py suggests a web crawling or scraping component.

- Data Preprocessing:
– Scripts such as tidy.py, tree.py, and possibly extract_sentence.log and date_parsing.log indicate data cleaning, parsing, and organization.

- Feature Extraction / Vectorization:
– The Tovector1NowUse.py, Tovecter2.py (and materials/Tovector.py) scripts point to converting the cleaned text into vector representations.
– The output is eventually stored in CSV files (e.g., outputarea/Tovector.csv, 材料/Tovector.csv).

- Indexing and Similarity Search:
– FAISS.py and FAISS_legal目前用這個.py are clearly using Facebook’s FAISS library to build vector-based indexes.
– There are several pre-built FAISS index files (e.g., court_index_ivf.faiss, court_index_ivf_legal.faiss, court_index_ivf_optimized.faiss) that indicate persistent storage of these indexes.

- Visualization/Additional Analytics:
– Script like paint.py could be responsible for visualizing outputs.
– linearreturn.py might be part of a calculation or result-returning utility.

- Logs and Supporting Artifacts:
– Log files (date_parsing.log, extract_sentence.log) help track processing steps.
– The outputarea directory organizes generated CSV and other outputs.

##### Relationships and Data Flow
– Next, indicate a Preprocessing Block that cleans, extracts, and organizes the data (using tidy.py etc.). This component should capture logging elements.
– Then, capture the Feature Extraction / Vectorization Phase where the Tovector scripts transform the cleaned text into numerical vectors.
– Follow this with an Indexing Block where the FAISS-based scripts process these vectors to generate searchable indexes (stored as court_index_ivf*.faiss).
– Finally, show an Output/Visualization component that produces CSV outputs and possibly visual analytics through paint.py and similar utilities.

##### Architectural Patterns and Design Principles
– The system is organized as a pipeline with clear stages (data ingestion, preprocessing, vectorization, indexing, and output). This separation of concerns is key—each stage is modular.
– There’s a blend of locally stored HTML data and dynamic web data extraction (with chrome-headless-shell and spiderr.py); this indicates the system is built for flexibility in sourcing data.
– Using FAISS suggests that high-performance vector similarity search is a core requirement.
– Logging and auxiliary files are used to monitor process flows and are integral to debugging and performance tuning.

##### Relevant Technologies and Frameworks
– Python is used throughout as the scripting language.
– FAISS (the library from Facebook) is central to the indexing component.
– The chrome-headless-shell-win64 folder indicates the use of a headless browser for rendering pages (implying some web automation/scraping technology).
– CSV file output (used in both the outputarea and importantcsv directories) is a simple but critical data exchange format.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Input
    subgraph "Data Input"
        DS1["HTML Case Files"]:::data
        DS2["Supplementary CSV Data"]:::data
        DS3["Supplementary Materials"]:::data
        DS4a["chrome-headless-shell-win64"]:::external
        DS4b["spiderr.py"]:::external
    end

    %% Data Preprocessing
    subgraph "Data Preprocessing"
        DP1["tidy.py"]:::process
        DP2["tree.py"]:::process
        DP3["date_parsing.log"]:::process
        DP4["extract_sentence.log"]:::process
    end

    %% Vectorization
    subgraph "Vectorization"
        V1["Tovector1NowUse.py"]:::vector
        V2["Tovecter2.py"]:::vector
    end

    %% Indexing & Search
    subgraph "Indexing & Search"
        I1["FAISS.py"]:::index
        I2["FAISS_legal目前用這個.py"]:::index
        I3["court_index_ivf.faiss"]:::index
        I4["court_index_ivf_legal.faiss"]:::index
        I5["court_index_ivf_optimized.faiss"]:::index
    end

    %% Output/Visualization
    subgraph "Output/Visualization"
        OV1["CSV Outputs"]:::output
        OV2["paint.py"]:::visual
        OV3["linearreturn.py"]:::visual
    end

    %% Data Flow
    DS1 --> DP1
    DS2 --> DP1
    DS3 --> DP1
    DS4a --> DP1
    DS4b --> DP1

    DP1 --> V1
    DP2 --> V1
    DP3 --> V1
    DP4 --> V1

    V1 --> I1
    V2 --> I1
    V1 --> I2
    V2 --> I2

    I1 --> OV1
    I2 --> OV1
    I3 --> OV1
    I4 --> OV1
    I5 --> OV1

    I1 --> OV2
    I1 --> OV3

    %% Click Events for Data Input
    click DS1 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/case_1_detail.html"
    click DS2 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/importantcsv/import.csv"
    click DS3 "https://github.com/hanyaaaaaa/dataanalysis/tree/main/材料"
    click DS4a "https://github.com/hanyaaaaaa/dataanalysis/tree/main/chrome-headless-shell-win64"
    click DS4b "https://github.com/hanyaaaaaa/dataanalysis/blob/main/spiderr.py"

    %% Click Events for Data Preprocessing
    click DP1 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/tidy.py"
    click DP2 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/tree.py"
    click DP3 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/date_parsing.log"
    click DP4 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/extract_sentence.log"

    %% Click Events for Vectorization
    click V1 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/Tovector1NowUse.py"
    click V2 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/Tovecter2.py"

    %% Click Events for Indexing & Search
    click I1 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/FAISS.py"
    click I2 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/FAISS_legal目前用這個.py"
    click I3 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/court_index_ivf.faiss"
    click I4 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/court_index_ivf_legal.faiss"
    click I5 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/court_index_ivf_optimized.faiss"

    %% Click Events for Output/Visualization
    click OV2 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/paint.py"
    click OV3 "https://github.com/hanyaaaaaa/dataanalysis/blob/main/linearreturn.py"

    %% Styles
    classDef data fill:#F0E68C,stroke:#B8860B,stroke-width:2px;
    classDef external fill:#FFD700,stroke:#DAA520,stroke-width:2px;
    classDef process fill:#98FB98,stroke:#32CD32,stroke-width:2px;
    classDef vector fill:#87CEFA,stroke:#4682B4,stroke-width:2px;
    classDef index fill:#FFB6C1,stroke:#FF69B4,stroke-width:2px;
    classDef visual fill:#D8BFD8,stroke:#DA70D6,stroke-width:2px;
    classDef output fill:#E6E6FA,stroke:#9370DB,stroke-width:2px;
```
