How hanyaaaaaa/dataanalysis works
Main Components
• Data Source/Input:
– The many case_XX_detail.html files (ranging from case_1_detail.html to case_404_detail.html and beyond) serve as primary input data.
– The materials folder and any additional CSV files (like importantcsv/import.csv) provide supplementary data.
– The chrome-headless-shell-win64 folder indicates that a headless browser may be used for web scraping or rendering dynamic pages. In addition, spiderr.py suggests a web crawling or scraping component.
• Data Preprocessing:
– Scripts such as tidy.py, tree.py, and possibly extract_sentence.log and date_parsing.log indicate data cleaning, parsing, and organization.
• Feature Extraction / Vectorization:
– The Tovector1NowUse.py, Tovecter2.py (and materials/Tovector.py) scripts point to converting the cleaned text into vector representations.
– The output is eventually stored in CSV files (e.g., outputarea/Tovector.csv, 材料/Tovector.csv).
• Indexing and Similarity Search:
– FAISS.py and FAISS_legal目前用這個.py are clearly using Facebook’s FAISS library to build vector-based indexes.
– There are several pre-built FAISS index files (e.g., court_index_ivf.faiss, court_index_ivf_legal.faiss, court_index_ivf_optimized.faiss) that indicate persistent storage of these indexes.
• Visualization/Additional Analytics:
– Script like paint.py could be responsible for visualizing outputs.
– linearreturn.py might be part of a calculation or result-returning utility.
• Logs and Supporting Artifacts:
– Log files (date_parsing.log, extract_sentence.log) help track processing steps.
– The outputarea directory organizes generated CSV and other outputs.
Relationships and Data Flow
– Next, indicate a Preprocessing Block that cleans, extracts, and organizes the data (using tidy.py etc.). This component should capture logging elements.
– Then, capture the Feature Extraction / Vectorization Phase where the Tovector scripts transform the cleaned text into numerical vectors.
– Follow this with an Indexing Block where the FAISS-based scripts process these vectors to generate searchable indexes (stored as court_index_ivf*.faiss).
– Finally, show an Output/Visualization component that produces CSV outputs and possibly visual analytics through paint.py and similar utilities.
Architectural Patterns and Design Principles
– The system is organized as a pipeline with clear stages (data ingestion, preprocessing, vectorization, indexing, and output). This separation of concerns is key—each stage is modular.
– There’s a blend of locally stored HTML data and dynamic web data extraction (with chrome-headless-shell and spiderr.py); this indicates the system is built for flexibility in sourcing data.
– Using FAISS suggests that high-performance vector similarity search is a core requirement.
– Logging and auxiliary files are used to monitor process flows and are integral to debugging and performance tuning.
Relevant Technologies and Frameworks
– Python is used throughout as the scripting language.
– FAISS (the library from Facebook) is central to the indexing component.
– The chrome-headless-shell-win64 folder indicates the use of a headless browser for rendering pages (implying some web automation/scraping technology).
– CSV file output (used in both the outputarea and importantcsv directories) is a simple but critical data exchange format.