# woygwl/---- architecture

> Architecture diagram of the GitHub repository woygwl/----, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/woygwl/----
- Repository: https://github.com/woygwl/----

## Overview

##### Project type and purpose
- This is a command-line data-pipeline tool (an “open-source” style Python project) for mining, validating, and submitting quantitative “alpha” factors to the WorldQuantBrain platform.
- Key use cases:
- Stepwise factor generation (“digging_1step.py”, “digging_2step.py”, “digging_3step.py”)
- Automatic checking of which factors are submittable (“check.py”)
- Automatic submission of approved factors (“submit_alpha.py”)
- It integrates with an external REST API (WorldQuantBrain), writes intermediate results to local files under “records/”, and optionally pushes notifications via WeChat.

##### File structure
- Top-level scripts:
- digging_1step.py, digging_2step.py, digging_3step.py – successive mining stages
- check.py – queries the API for candidate factors and writes “submitable_alpha.csv”
- submit_alpha.py – reads the CSV to perform submission
- cal_correlation_locally.py – local analytics support
- machine_lib.py – HTTP client (+ aiohttp connector, SSL bypass)
- fields.py – data-field definitions (e.g. 豆包推荐)
- config.py + user_info.txt – credentials & pipeline parameters
- records/
- Raw & processed outputs (alpha id lists, PNL pickles, simulation results, submitable CSVs)
- requirements.txt – lists Python dependencies (aiohttp, pandas, etc.)

##### README for additional context
- Version history confirms the pipeline stages and the introduction of WeChat notifications in check.py
- Usage steps outline the sequential execution order, how to set step tags and dataset IDs
- Emphasizes configuring user_info.txt with API tokens and dataset identifiers

##### Main components & interactions
a. Configuration Layer
– config.py / user_info.txt: contains API URLs, secrets, step tags, dataset IDs
b. Data Mining Pipeline
– Stage 1 (digging_1step.py): fetch raw factor candidates via API → write records/*.txt
– Stage 2 (digging_2step.py): input from stage1, apply filters/transformations → records/*.txt
– Stage 3 (digging_3step.py): final screening → records/*.txt
c. Analytics & Validation
– cal_correlation_locally.py: compute correlations/stats on locally saved data
– machine_lib.py: provides the async HTTP client for all API calls
d. Submission Workflow
– check.py: calls the API to see which factor IDs are eligible → outputs submitable_alpha.csv
– submit_alpha.py: reads CSV, posts submission calls back to the API
– Optional WeChat notifications via Server酱 in check.py
e. Data Store
– Local filesystem under records/ for all intermediate and final data
f. External Service
– WorldQuantBrain REST API for data sets, factor validation, factor submission

##### Architectural patterns & design principles
- Pipeline (stepwise ETL-style): each script picks up output from the prior stage
- Separation of concerns:
- machine_lib.py abstracts HTTP interaction
- config.py centralizes all environment/secret configuration
- each “digging_Xstep” and the “check/submit” scripts have single responsibilities
- File-based persistence: simple, stateless execution of each stage

##### Essential details to capture
- The sequential nature of the pipeline
- The abstraction of HTTP logic in machine_lib.py
- The role of “records/” as the single source of truth for intermediate outputs
- External dependencies (WorldQuantBrain API, Server酱)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    subgraph "Configuration Layer"
        CFG["Config (config.py, user_info.txt)"]:::config
    end
    click CFG "https://github.com/woygwl/----/blob/main/config.py"

    %% Field Definitions & Documentation
    subgraph "Field Definitions & Documentation"
        FD["Fields Module (fields.py)"]:::scripts
        REQ["Requirements (requirements.txt)"]:::docs
        DOC["Documentation (README.md)"]:::docs
    end
    click FD "https://github.com/woygwl/----/blob/main/fields.py"
    click REQ "https://github.com/woygwl/----/blob/main/requirements.txt"
    click DOC "https://github.com/woygwl/----/blob/main/README.md"

    %% HTTP Client Abstraction
    subgraph "HTTP Client"
        HC["Machine Lib (machine_lib.py)"]:::scripts
    end
    click HC "https://github.com/woygwl/----/blob/main/machine_lib.py"

    %% Data Mining Pipeline
    subgraph "Data Mining Pipeline"
        S1["Digging Stage 1 (digging_1step.py)"]:::scripts
        S2["Digging Stage 2 (digging_2step.py)"]:::scripts
        S3["Digging Stage 3 (digging_3step.py)"]:::scripts
    end
    click S1 "https://github.com/woygwl/----/blob/main/digging_1step.py"
    click S2 "https://github.com/woygwl/----/blob/main/digging_2step.py"
    click S3 "https://github.com/woygwl/----/blob/main/digging_3step.py"

    %% Analytics Module
    AN["Local Analytics (cal_correlation_locally.py)"]:::scripts
    click AN "https://github.com/woygwl/----/blob/main/cal_correlation_locally.py"

    %% Validation & Submission
    subgraph "Validation & Submission"
        CH["Eligibility Check (check.py)"]:::scripts
        SB["Submission Module (submit_alpha.py)"]:::scripts
    end
    click CH "https://github.com/woygwl/----/blob/main/check.py"
    click SB "https://github.com/woygwl/----/blob/main/submit_alpha.py"

    %% Data Store
    DS["Local Data Store (records/)"]:::datastore
    click DS "https://github.com/woygwl/----/tree/main/records/"

    %% External Services
    EXTAPI["WorldQuantBrain API"]:::external
    NOTI["WeChat Notification (Server酱)"]:::external

    %% Connections
    CFG --> S1
    CFG --> S2
    CFG --> S3
    CFG --> CH
    CFG --> SB

    FD --> S1
    FD --> S2
    FD --> S3

    S1 -->|"writes TXT"| DS
    DS -->|"reads TXT"| S2
    S2 -->|"writes TXT"| DS
    DS -->|"reads TXT"| S3
    S3 -->|"writes TXT"| DS

    DS -->|"provides data"| AN
    AN -->|"analysis output"| DS

    DS -->|"provides IDs"| CH
    CH -->|"writes CSV"| DS
    CH -->|"notify"| NOTI

    DS -->|"reads CSV"| SB
    SB -->|"writes final CSV"| DS

    S1 --> HC
    S2 --> HC
    S3 --> HC
    CH --> HC
    SB --> HC

    HC -->|"HTTP calls"| EXTAPI
    SB -->|"submit factors"| EXTAPI

    %% Styles
    classDef config fill:#DDDDDD,stroke:#333,stroke-width:1px;
    classDef scripts fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef datastore fill:#FFFF66,stroke:#333,stroke-width:1px;
    classDef external fill:#90EE90,stroke:#333,stroke-width:1px;
    classDef docs fill:#F5DEB3,stroke:#333,stroke-width:1px;
```
