How firmai/financial-machine-learning works
Project Type and Purpose
• This is a documentation‐generation tool (wiki and README builder) for a curated list of financial‐machine‐learning GitHub repositories.
• It’s not a full-stack web application; rather, it’s a CLI-based ETL-style pipeline written in Python, orchestrated by GitHub Actions.
File-Structure Analysis
• Top level:
• Python scripts: git_search.py, git_status.py, wiki_gen.py, git_util.py
• Configuration: conf.py, requirements.txt, .gitignore
• Data input: raw_data/url_list.csv, raw_data/process.drawio
• Output directory: generated_wiki/*.md
• CI workflows: .github/workflows/{repo_search.yml, repo_status.yml, wiki_gen.yml}
• Patterns:
• git_search/gistatus modules fetch and normalize metadata from target repos
• wiki_gen glues everything, applies templates, and writes markdown pages
• generated_wiki contains section files to be pushed into the project’s wiki
README Insights
• README.md contains placeholder tables and badges pointing at three GitHub Actions workflows:
• Repo-Search: updates the list of repositories
• Repo-Status: checks each repo’s last commit, star count, etc.
• Wiki-Generator: regenerates full README and wiki pages in realtime
Main Architectural Components
a. External Services
• GitHub API – to query repository metadata (stars, commits)
• Git (local clone or shallow clone) – for detailed status or fallback
b. Data Input
• url_list.csv – the master list of GitHub repo URLs and metadata
c. Processing Modules
• git_search.py – refreshes/expands the URL list from GitHub (e.g., adds new repos)
• git_status.py – loops over url_list.csv, fetches current status (last commit date, star count, status badges)
• git_util.py – shared helper functions (e.g., API wrappers, CSV readers/writers)
• wiki_gen.py – reads status CSV + templates, injects the “PLACEHOLDER” sections into README.md and emits per-section markdown under generated_wiki/
d. Outputs
• README.md – auto-populated tables of repos by category
• generated_wiki/*.md – individual wiki pages per section (deep_learning_and_reinforcement_learning.md, data.md, etc.)
e. CI/CD Orchestration
• GitHub Actions workflows trigger these scripts on schedule or on push, commit updated markdown back to the repo (or wiki branch).
Relationships & Data Flow
• GitHub Actions workflow “repo_search.yml” → runs git_search.py → reads/writes raw_data/url_list.csv
• Workflow “repo_status.yml” → runs git_status.py → reads raw_data/url_list.csv and writes an updated status CSV or in-memory table
• Workflow “wiki_gen.yml” → runs wiki_gen.py → reads status CSV + templates (README.md with placeholders) → writes README.md and populates generated_wiki/*.md
• On success, the workflow commits the modified files back to the repository (or pushes to the wiki).
Architectural Patterns & Design Principles
• Pipeline/Stages Pattern: distinct stages for search, status, and generation
• Single Responsibility: each script has one main responsibility
• Configuration via CSV & conf.py: minimal code changes if data source or template locations change
• Idempotency: rerunning the generator produces the same markdown if inputs haven’t changed
• CI-driven updates: changes occur automatically on a schedule or code push, no manual intervention
Key Technologies & Frameworks
• Python 3.x (requests or PyGitHub, pandas or csv, Jinja2 or simple string interpolation)
• GitHub Actions for CI/CD
• Git CLI or direct API calls for repository metadata
• Markdown as the output format
Essential Elements
• Clear, descriptive labels for each script, workflow, and data store
• Major interactions annotated succinctly (inputs, outputs, triggers)