# firmai/financial-machine-learning architecture

> Architecture diagram of the GitHub repository firmai/financial-machine-learning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/firmai/financial-machine-learning
- Repository: https://github.com/firmai/financial-machine-learning

## Overview

##### Project Type and Purpose
- This is a documentation‐generation tool (wiki and README builder) for a curated list of financial‐machine‐learning GitHub repositories.
- It’s not a full-stack web application; rather, it’s a CLI-based ETL-style pipeline written in Python, orchestrated by GitHub Actions.

##### File-Structure Analysis
- Top level:
- Python scripts: git_search.py, git_status.py, wiki_gen.py, git_util.py
- Configuration: conf.py, requirements.txt, .gitignore
- Data input: raw_data/url_list.csv, raw_data/process.drawio
- Output directory: generated_wiki/*.md
- CI workflows: .github/workflows/{repo_search.yml, repo_status.yml, wiki_gen.yml}
- Patterns:
- git_search/gistatus modules fetch and normalize metadata from target repos
- wiki_gen glues everything, applies templates, and writes markdown pages
- generated_wiki contains section files to be pushed into the project’s wiki

##### README Insights
- README.md contains placeholder tables and badges pointing at three GitHub Actions workflows:
- Repo-Search: updates the list of repositories
- Repo-Status: checks each repo’s last commit, star count, etc.
- Wiki-Generator: regenerates full README and wiki pages in realtime

##### Main Architectural Components
a. External Services
- GitHub API – to query repository metadata (stars, commits)
- Git (local clone or shallow clone) – for detailed status or fallback
b. Data Input
- url_list.csv – the master list of GitHub repo URLs and metadata
c. Processing Modules
- git_search.py – refreshes/expands the URL list from GitHub (e.g., adds new repos)
- git_status.py – loops over url_list.csv, fetches current status (last commit date, star count, status badges)
- git_util.py – shared helper functions (e.g., API wrappers, CSV readers/writers)
- wiki_gen.py – reads status CSV + templates, injects the “PLACEHOLDER” sections into README.md and emits per-section markdown under generated_wiki/
d. Outputs
- README.md – auto-populated tables of repos by category
- generated_wiki/*.md – individual wiki pages per section (deep_learning_and_reinforcement_learning.md, data.md, etc.)
e. CI/CD Orchestration
- GitHub Actions workflows trigger these scripts on schedule or on push, commit updated markdown back to the repo (or wiki branch).

##### Relationships & Data Flow
- GitHub Actions workflow “repo_search.yml” → runs git_search.py → reads/writes raw_data/url_list.csv
- Workflow “repo_status.yml” → runs git_status.py → reads raw_data/url_list.csv and writes an updated status CSV or in-memory table
- Workflow “wiki_gen.yml” → runs wiki_gen.py → reads status CSV + templates (README.md with placeholders) → writes README.md and populates generated_wiki/*.md
- On success, the workflow commits the modified files back to the repository (or pushes to the wiki).

##### Architectural Patterns & Design Principles
- Pipeline/Stages Pattern: distinct stages for search, status, and generation
- Single Responsibility: each script has one main responsibility
- Configuration via CSV & conf.py: minimal code changes if data source or template locations change
- Idempotency: rerunning the generator produces the same markdown if inputs haven’t changed
- CI-driven updates: changes occur automatically on a schedule or code push, no manual intervention

##### Key Technologies & Frameworks
- Python 3.x (requests or PyGitHub, pandas or csv, Jinja2 or simple string interpolation)
- GitHub Actions for CI/CD
- Git CLI or direct API calls for repository metadata
- Markdown as the output format

##### Essential Elements
- Clear, descriptive labels for each script, workflow, and data store
- Major interactions annotated succinctly (inputs, outputs, triggers)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "CI Workflows"
        RS["repo_search.yml"]:::ci
        ST["repo_status.yml"]:::ci
        WGWF["wiki_gen.yml"]:::ci
    end

    subgraph "Data-Processing"
        GS["git_search.py"]:::process
        GST["git_status.py"]:::process
        WGG["wiki_gen.py"]:::process
        GU["git_util.py"]:::process
    end

    subgraph "Configuration"
        CP["conf.py"]:::config
        REQ["requirements.txt"]:::config
    end

    subgraph "Data Input"
        URL["raw_data/url_list.csv"]:::data
        DIAG["raw_data/process.drawio"]:::data
    end

    subgraph "Outputs"
        RD["README.md"]:::output
        subgraph "Generated Wiki Pages"
            GW_DIR["generated_wiki/"]:::output
            AF["alternative_finance.md"]:::output
            CC["colleges_centers_and_departments.md"]:::output
            C["courses.md"]:::output
            D["data.md"]:::output
            DPT["data_processing_techniques_and_transformations.md"]:::output
            DL["deep_learning_and_reinforcement_learning.md"]:::output
            DER["derivatives_and_hedging.md"]:::output
            ER["extended_research.md"]:::output
            FRA["factor_and_risk_analysis.md"]:::output
            FI["fixed_income.md"]:::output
            OM["other_models.md"]:::output
            PP["personal_papers.md"]:::output
            PO["portfolio_selection_and_optimisation.md"]:::output
            RLF["reinforcement_learning.md"]:::output
            T["textual.md"]:::output
            U["unsupervised.md"]:::output
        end
    end

    GHAPI["GitHub API"]:::external
    GITCLI["Git CLI"]:::external
    COMMIT["Commit Updated Files"]:::action

    RS -->|"runs"| GS
    RS -->|"uses"| CP
    RS -->|"uses"| REQ

    GS -->|"read/write"| URL
    GS -->|"call API"| GHAPI
    GS -->|"helper"| GU

    ST -->|"runs"| GST
    ST -->|"uses"| GU
    GST -->|"read"| URL
    GST -->|"call API/Git"| GHAPI
    GST -->|"call API/Git"| GITCLI

    WGWF -->|"runs"| WGG
    WGG -->|"reads"| URL
    WGG -->|"uses"| GU
    WGG -->|"reads"| CP
    WGG -->|"writes"| RD
    WGG -->|"emit .md"| GW_DIR

    GW_DIR --> AF
    GW_DIR --> CC
    GW_DIR --> C
    GW_DIR --> D
    GW_DIR --> DPT
    GW_DIR --> DL
    GW_DIR --> DER
    GW_DIR --> ER
    GW_DIR --> FRA
    GW_DIR --> FI
    GW_DIR --> OM
    GW_DIR --> PP
    GW_DIR --> PO
    GW_DIR --> RLF
    GW_DIR --> T
    GW_DIR --> U

    RD -->|"staged for commit"| COMMIT
    GW_DIR -->|"staged files"| COMMIT

    class RS,ST,WGWF ci
    class GS,GST,WGG,GU process
    class CP,REQ config
    class URL,DIAG data
    class RD,AF,CC,C,D,DPT,DL,DER,ER,FRA,FI,OM,PP,PO,RLF,T,U output
    class GHAPI,GITCLI external
    class COMMIT action

    click GS "https://github.com/firmai/financial-machine-learning/blob/master/git_search.py"
    click GST "https://github.com/firmai/financial-machine-learning/blob/master/git_status.py"
    click GU "https://github.com/firmai/financial-machine-learning/blob/master/git_util.py"
    click WGG "https://github.com/firmai/financial-machine-learning/blob/master/wiki_gen.py"
    click CP "https://github.com/firmai/financial-machine-learning/blob/master/conf.py"
    click REQ "https://github.com/firmai/financial-machine-learning/blob/master/requirements.txt"
    click URL "https://github.com/firmai/financial-machine-learning/blob/master/raw_data/url_list.csv"
    click DIAG "https://github.com/firmai/financial-machine-learning/blob/master/raw_data/process.drawio"
    click AF "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/alternative_finance.md"
    click CC "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/colleges_centers_and_departments.md"
    click C "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/courses.md"
    click D "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/data.md"
    click DPT "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/data_processing_techniques_and_transformations.md"
    click DL "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/deep_learning_and_reinforcement_learning.md"
    click DER "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/derivatives_and_hedging.md"
    click ER "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/extended_research.md"
    click FRA "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/factor_and_risk_analysis.md"
    click FI "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/fixed_income.md"
    click OM "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/other_models.md"
    click PP "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/personal_papers.md"
    click PO "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/portfolio_selection_and_optimisation.md"
    click RLF "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/reinforcement_learning.md"
    click T "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/textual.md"
    click U "https://github.com/firmai/financial-machine-learning/blob/master/generated_wiki/unsupervised.md"
    click RD "https://github.com/firmai/financial-machine-learning/blob/master/README.md"
    click GW_DIR "https://github.com/firmai/financial-machine-learning/tree/master/generated_wiki/"
    click RS "https://github.com/firmai/financial-machine-learning/blob/master/.github/workflows/repo_search.yml"
    click ST "https://github.com/firmai/financial-machine-learning/blob/master/.github/workflows/repo_status.yml"
    click WGWF "https://github.com/firmai/financial-machine-learning/blob/master/.github/workflows/wiki_gen.yml"

    classDef ci fill:#FFA500,stroke:#333,stroke-width:1px
    classDef process fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef data fill:#90EE90,stroke:#333,stroke-width:1px
    classDef output fill:#FFFFE0,stroke:#333,stroke-width:1px
    classDef external fill:#CCCCCC,stroke:#333,stroke-width:1px
    classDef action fill:#FFD700,stroke:#333,stroke-width:1px
```
