# jdhruv1503/brainiac architecture

> Architecture diagram of the GitHub repository jdhruv1503/brainiac, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jdhruv1503/brainiac
- Repository: https://github.com/jdhruv1503/brainiac

## Overview

1. **Overall shape**
   - Brainiac is a two-stage pipeline: a local **alpha generation/orchestration layer** produces candidate alphas from research and strategy inputs, then a separate **simulation service** submits those alphas for backtesting and stores results.
   - The repo is organized around this boundary: `AlphaGenerator/` creates and prepares alphas, while `Simulation/` is the execution/backtest side.

2. **Primary input sources**
   - The main raw inputs are financial research PDFs in `AlphaGenerator/ResearchPDFs/` and CSV datasets in `Datasets/`.
   - The README also describes text strategies as an input source; the code tree does not show a dedicated strategies folder, so that appears to be an expected runtime/source directory rather than a committed one.
   - The CSVs are a domain knowledge base covering prices, fundamentals, sentiment, news, analyst estimates, risk metrics, and China-specific data.

3. **Alpha generation subsystem**
   - `AlphaGenerator/generate_from_research_pdfs.py` and `AlphaGenerator/generate_from_strategies.py` are the main entry points for turning source material into alpha candidates.
   - `utils/program_alphas.py`, `utils/distill_alphas.py`, and `utils/split_alphas.py` suggest a staged transformation pipeline: extract candidate ideas, refine or distill them, and split them into usable units for downstream processing.
   - This layer is described in the README as using agentic AI / LangChain / RAG-style retrieval, so it is the “intelligence” layer that converts unstructured research into executable alpha expressions.

4. **Knowledge-base and syntax support**
   - `AlphaGenerator/utils/BrainKnowledgebaseSyntax/` contains `syntax.md` and `datasets_chn.md`, which act as prompt/reference material for the generation layer.
   - These files likely define the expression grammar and dataset naming conventions needed to map natural-language ideas into WorldQuant Fast Expression-compatible alphas.
   - This is an important boundary: generated ideas must conform to a fixed syntax and approved dataset vocabulary before they can be simulated.

5. **Dataset selection / retrieval boundary**
   - The `Datasets/*.csv` files function as the local lookup corpus for dataset retrieval and matching.
   - Architecturally, this is the bridge between “alpha idea” and “what data fields can actually be used.”
   - The README explicitly frames this as RAG-assisted dataset retrieval: the generator chooses datasets that fit the alpha before handing the expression to simulation.

6. **Orchestration and file-watching**
   - `AlphaGenerator/watchdog.py` is the automation trigger.
   - It watches for changes/new files in the research and strategy source folders, then kicks off generation and forwarding automatically.
   - This makes the generator event-driven rather than requiring manual batch runs for every new input artifact.

7. **Submission to simulation**
   - `AlphaGenerator/send_to_simulation.py` is the handoff point from generated alphas to the simulation backend.
   - Conceptually, this is the integration seam where local alpha artifacts become API requests to the backtesting service.
   - This script is the key integration boundary between “content generation” and “execution/backtest.”

8. **Simulation service**
   - `Simulation/main.py` is the runtime entrypoint for the backtesting API.
   - `Simulation/schemas.py` holds request/response contracts, and `Simulation/simulate.py` contains the simulation/backtest logic.
   - The service is intended to run continuously in the background so that generated alphas can be posted to it asynchronously.

9. **Credentials and external platform dependency**
   - `Simulation/creds/example.json` indicates the simulation service expects per-account credential files.
   - The README states that the service integrates with WorldQuant BRAIN for automated backtesting, so that external platform is a core architectural dependency.
   - This means the repo is not a standalone simulator; it is an automation wrapper around a third-party backtesting environment.

10. **Persistence and results**
   - Results are stored in a SQLite database at `Simulation/alphas/alpha_results.db` according to the README.
   - That makes the simulation layer stateful: it records submitted alphas and outcomes for later inspection or iteration.
   - The database is the main durable output artifact of the pipeline.

11. **External AI/API services**
   - The README calls out `LLAMA_PARSE_API_KEYS` and `DEEPINFRA_API_KEYS`, which indicates the generation side depends on external AI/document parsing services.
   - These services are used to process PDFs and support the agentic generation workflow.
   - So the generation path depends on both domain data files and external model/parsing APIs.

12. **End-to-end data flow**
   - In practice, the flow is: research PDFs/strategy text + local dataset corpus → generation/refinement/splitting → alpha expression formatting and dataset matching → watchdog-triggered submission → simulation API → WorldQuant BRAIN backtest → SQLite result storage.
   - The architecture is best drawn as two major blocks with a clear API boundary between them, plus shared domain assets (`Datasets/` and syntax docs) feeding the generation side.

Main entry points: [`Simulation/main.py`](https://github.com/jdhruv1503/brainiac/blob/main/Simulation/main.py) (Sim API), [`AlphaGenerator/generate_from_research_pdfs.py`](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/generate_from_research_pdfs.py) (PDF generator), [`AlphaGenerator/utils/program_alphas.py`](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/utils/program_alphas.py) (Program alphas), [`AlphaGenerator/watchdog.py`](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/watchdog.py) (Watchdog)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_alpha["Alpha Generation"]
  node_n_pdf_gen["PDF generator<br/>entrypoint"]
  node_n_strategy_gen["Strategy generator<br/>entrypoint"]
  node_n_program["Program alphas<br/>transform<br/>[program_alphas.py]"]
  node_n_distill["Distill alphas<br/>refine<br/>[distill_alphas.py]"]
  node_n_split["Split alphas<br/>partition<br/>[split_alphas.py]"]
  node_n_watchdog["Watchdog<br/>automation<br/>[watchdog.py]"]
  node_n_send["Send to sim<br/>handoff"]
end

subgraph group_g_sim["Simulation Service"]
  node_n_sim_main["Sim API<br/>service entrypoint<br/>[main.py]"]
  node_n_schemas["Schemas<br/>contracts<br/>[schemas.py]"]
  node_n_simulate["Simulate<br/>backtest engine<br/>[simulate.py]"]
  node_n_creds["Credentials<br/>config secret<br/>[example.json]"]
  node_n_db[("Results DB<br/>sqlite store")]
end

subgraph group_g_inputs["Shared Inputs"]
  node_n_research_pdfs["Research PDFs<br/>source corpus<br/>[ssrn-2472571.pdf]"]
  node_n_datasets[("Datasets<br/>market data corpus")]
  node_n_syntax["Syntax refs<br/>prompt/spec<br/>[syntax.md]"]
end

node_n_readme["README<br/>overview<br/>[README.md]"]
node_n_worldquant(("WorldQuant BRAIN<br/>external platform"))
node_n_llm_api(("LLM / parse APIs<br/>external service"))

node_n_readme -->|"describes"| node_n_pdf_gen
node_n_readme -->|"describes"| node_n_sim_main
node_n_research_pdfs -->|"input"| node_n_pdf_gen
node_n_datasets -->|"retrieval corpus"| node_n_pdf_gen
node_n_syntax -->|"constraints"| node_n_program
node_n_pdf_gen -->|"candidates"| node_n_program
node_n_strategy_gen -->|"candidates"| node_n_program
node_n_program -->|"refine"| node_n_distill
node_n_distill -->|"partition"| node_n_split
node_n_watchdog -->|"triggers"| node_n_pdf_gen
node_n_watchdog -->|"triggers"| node_n_strategy_gen
node_n_split -->|"handoff"| node_n_send
node_n_send -->|"API request"| node_n_sim_main
node_n_sim_main -->|"uses"| node_n_schemas
node_n_sim_main -->|"dispatches"| node_n_simulate
node_n_simulate -->|"executes on"| node_n_worldquant
node_n_creds -->|"auth"| node_n_sim_main
node_n_sim_main -->|"stores"| node_n_db
node_n_llm_api -->|"supports"| node_n_pdf_gen

click node_n_readme "https://github.com/jdhruv1503/brainiac/blob/main/README.md"
click node_n_research_pdfs "https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/ResearchPDFs/ssrn-2472571.pdf"
click node_n_datasets "https://github.com/jdhruv1503/brainiac/blob/main/Datasets/PriceVolumeDataForEquity.csv"
click node_n_syntax "https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/utils/BrainKnowledgebaseSyntax/syntax.md"
click node_n_pdf_gen "https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/generate_from_research_pdfs.py"
click node_n_strategy_gen "https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/generate_from_strategies.py"
click node_n_program "https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/utils/program_alphas.py"
click node_n_distill "https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/utils/distill_alphas.py"
click node_n_split "https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/utils/split_alphas.py"
click node_n_watchdog "https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/watchdog.py"
click node_n_send "https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/send_to_simulation.py"
click node_n_sim_main "https://github.com/jdhruv1503/brainiac/blob/main/Simulation/main.py"
click node_n_schemas "https://github.com/jdhruv1503/brainiac/blob/main/Simulation/schemas.py"
click node_n_simulate "https://github.com/jdhruv1503/brainiac/blob/main/Simulation/simulate.py"
click node_n_creds "https://github.com/jdhruv1503/brainiac/blob/main/Simulation/creds/example.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_pdf_gen,node_n_strategy_gen,node_n_program,node_n_distill,node_n_split,node_n_watchdog,node_n_send toneBlue
class node_n_sim_main,node_n_schemas,node_n_simulate,node_n_creds,node_n_db toneAmber
class node_n_research_pdfs,node_n_datasets,node_n_syntax toneMint
class node_n_readme,node_n_worldquant,node_n_llm_api toneNeutral
```

## Components

### Alpha Generation

Research ingestion, alpha drafting, refinement, and handoff.

- [PDF generator](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/generate_from_research_pdfs.py): entrypoint · Generates alpha candidates from research PDFs. · `AlphaGenerator/generate_from_research_pdfs.py`
- [Strategy generator](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/generate_from_strategies.py): entrypoint · Generates alpha candidates from strategy text inputs. · `AlphaGenerator/generate_from_strategies.py`
- [Program alphas](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/utils/program_alphas.py): transform · Stages candidate ideas into executable alpha expressions. · `AlphaGenerator/utils/program_alphas.py`
- [Distill alphas](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/utils/distill_alphas.py): refine · Refines and compresses generated alpha candidates. · `AlphaGenerator/utils/distill_alphas.py`
- [Split alphas](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/utils/split_alphas.py): partition · Breaks alpha output into downstream-friendly units. · `AlphaGenerator/utils/split_alphas.py`
- [Watchdog](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/watchdog.py): automation · Watches source folders and triggers generation flow. · `AlphaGenerator/watchdog.py`
- [Send to sim](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/send_to_simulation.py): handoff · Submits generated alphas to the simulation backend. · `AlphaGenerator/send_to_simulation.py`

### Simulation Service

Backtest API, execution logic, and result persistence.

- [Sim API](https://github.com/jdhruv1503/brainiac/blob/main/Simulation/main.py): service entrypoint · Runs the backtesting service. · `Simulation/main.py`
- [Schemas](https://github.com/jdhruv1503/brainiac/blob/main/Simulation/schemas.py): contracts · Request and response shapes for simulation requests. · `Simulation/schemas.py`
- [Simulate](https://github.com/jdhruv1503/brainiac/blob/main/Simulation/simulate.py): backtest engine · Executes backtest logic against external platform. · `Simulation/simulate.py`
- [Credentials](https://github.com/jdhruv1503/brainiac/blob/main/Simulation/creds/example.json): config secret · Per-account credentials used by the simulation service. · `Simulation/creds/example.json`
- **Results DB**: sqlite store · Persistent storage for submitted alphas and outcomes.

### Shared Inputs

Local knowledge sources and external dependencies used by generation.

- [Research PDFs](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/ResearchPDFs/ssrn-2472571.pdf): source corpus · Financial research documents used to generate alpha ideas. · `AlphaGenerator/ResearchPDFs/ssrn-2472571.pdf`
- [Datasets](https://github.com/jdhruv1503/brainiac/blob/main/Datasets/PriceVolumeDataForEquity.csv): market data corpus · CSV knowledge base for dataset retrieval and alpha grounding. · `Datasets/PriceVolumeDataForEquity.csv`
- [Syntax refs](https://github.com/jdhruv1503/brainiac/blob/main/AlphaGenerator/utils/BrainKnowledgebaseSyntax/syntax.md): prompt/spec · Brain knowledgebase syntax and dataset naming references. · `AlphaGenerator/utils/BrainKnowledgebaseSyntax/syntax.md`

### Other components

- [README](https://github.com/jdhruv1503/brainiac/blob/main/README.md): overview · Project overview and operating model. · `README.md`
- **WorldQuant BRAIN**: external platform · Third-party backtesting platform integrated by the simulation service.
- **LLM / parse APIs**: external service · External parsing and model APIs used during generation.

## Connections

- README → PDF generator: describes · Documents the generation pipeline.
- README → Sim API: describes · Documents the simulation service.
- Research PDFs → PDF generator: input · Research documents are converted into alpha ideas.
- Datasets → PDF generator: retrieval corpus · Local data fields ground dataset selection.
- Syntax refs → Program alphas: constraints · Syntax references guide executable alpha formatting.
- PDF generator → Program alphas: candidates · Generated ideas are staged for execution.
- Strategy generator → Program alphas: candidates · Strategy inputs are converted into programmatic alphas.
- Program alphas → Distill alphas: refine · Programmed alphas are distilled further.
- Distill alphas → Split alphas: partition · Refined alphas are split for downstream use.
- Watchdog → PDF generator: triggers · File changes start generation runs.
- Watchdog → Strategy generator: triggers · New strategy inputs start generation runs.
- Split alphas → Send to sim: handoff · Prepared alpha units are forwarded for simulation.
- Send to sim → Sim API: API request · Generated alphas are posted to the simulation service.
- Sim API → Schemas: uses · API endpoint validates request/response contracts.
- Sim API → Simulate: dispatches · Service entrypoint invokes backtest logic.
- Simulate → WorldQuant BRAIN: executes on · Backtests are run through the external platform.
- Credentials → Sim API: auth · Credential files configure platform access.
- Sim API → Results DB: stores · Simulation outputs are persisted for later review.
- LLM / parse APIs → PDF generator: supports · External AI and parsing services assist generation.
