How woygwl/---- works
Project type and purpose
• This is a command-line data-pipeline tool (an “open-source” style Python project) for mining, validating, and submitting quantitative “alpha” factors to the WorldQuantBrain platform.
• Key use cases:
• Stepwise factor generation (“digging_1step.py”, “digging_2step.py”, “digging_3step.py”)
• Automatic checking of which factors are submittable (“check.py”)
• Automatic submission of approved factors (“submit_alpha.py”)
• It integrates with an external REST API (WorldQuantBrain), writes intermediate results to local files under “records/”, and optionally pushes notifications via WeChat.
File structure
• Top-level scripts:
• digging_1step.py, digging_2step.py, digging_3step.py – successive mining stages
• check.py – queries the API for candidate factors and writes “submitable_alpha.csv”
• submit_alpha.py – reads the CSV to perform submission
• cal_correlation_locally.py – local analytics support
• machine_lib.py – HTTP client (+ aiohttp connector, SSL bypass)
• fields.py – data-field definitions (e.g. 豆包推荐)
• config.py + user_info.txt – credentials & pipeline parameters
• records/
• Raw & processed outputs (alpha id lists, PNL pickles, simulation results, submitable CSVs)
• requirements.txt – lists Python dependencies (aiohttp, pandas, etc.)
README for additional context
• Version history confirms the pipeline stages and the introduction of WeChat notifications in check.py
• Usage steps outline the sequential execution order, how to set step tags and dataset IDs
• Emphasizes configuring user_info.txt with API tokens and dataset identifiers
Main components & interactions
a. Configuration Layer
– config.py / user_info.txt: contains API URLs, secrets, step tags, dataset IDs
b. Data Mining Pipeline
– Stage 1 (digging_1step.py): fetch raw factor candidates via API → write records/*.txt
– Stage 2 (digging_2step.py): input from stage1, apply filters/transformations → records/*.txt
– Stage 3 (digging_3step.py): final screening → records/*.txt
c. Analytics & Validation
– cal_correlation_locally.py: compute correlations/stats on locally saved data
– machine_lib.py: provides the async HTTP client for all API calls
d. Submission Workflow
– check.py: calls the API to see which factor IDs are eligible → outputs submitable_alpha.csv
– submit_alpha.py: reads CSV, posts submission calls back to the API
– Optional WeChat notifications via Server酱 in check.py
e. Data Store
– Local filesystem under records/ for all intermediate and final data
f. External Service
– WorldQuantBrain REST API for data sets, factor validation, factor submission
Architectural patterns & design principles
• Pipeline (stepwise ETL-style): each script picks up output from the prior stage
• Separation of concerns:
• machine_lib.py abstracts HTTP interaction
• config.py centralizes all environment/secret configuration
• each “digging_Xstep” and the “check/submit” scripts have single responsibilities
• File-based persistence: simple, stateless execution of each stage
Essential details to capture
• The sequential nature of the pipeline
• The abstraction of HTTP logic in machine_lib.py
• The role of “records/” as the single source of truth for intermediate outputs
• External dependencies (WorldQuantBrain API, Server酱)