How jdhruv1503/brainiac works
1. Overall shape
- Brainiac is a two-stage pipeline: a local alpha generation/orchestration layer produces candidate alphas from research and strategy inputs, then a separate simulation service submits those alphas for backtesting and stores results.
- The repo is organized around this boundary: AlphaGenerator/ creates and prepares alphas, while Simulation/ is the execution/backtest side.
2. Primary input sources
- The main raw inputs are financial research PDFs in AlphaGenerator/ResearchPDFs/ and CSV datasets in Datasets/.
- The README also describes text strategies as an input source; the code tree does not show a dedicated strategies folder, so that appears to be an expected runtime/source directory rather than a committed one.
- The CSVs are a domain knowledge base covering prices, fundamentals, sentiment, news, analyst estimates, risk metrics, and China-specific data.
3. Alpha generation subsystem
- AlphaGenerator/generate_from_research_pdfs.py and AlphaGenerator/generate_from_strategies.py are the main entry points for turning source material into alpha candidates.
- utils/program_alphas.py, utils/distill_alphas.py, and utils/split_alphas.py suggest a staged transformation pipeline: extract candidate ideas, refine or distill them, and split them into usable units for downstream processing.
- This layer is described in the README as using agentic AI / LangChain / RAG-style retrieval, so it is the “intelligence” layer that converts unstructured research into executable alpha expressions.
4. Knowledge-base and syntax support
- AlphaGenerator/utils/BrainKnowledgebaseSyntax/ contains syntax.md and datasets_chn.md, which act as prompt/reference material for the generation layer.
- These files likely define the expression grammar and dataset naming conventions needed to map natural-language ideas into WorldQuant Fast Expression-compatible alphas.
- This is an important boundary: generated ideas must conform to a fixed syntax and approved dataset vocabulary before they can be simulated.
5. Dataset selection / retrieval boundary
- The Datasets/*.csv files function as the local lookup corpus for dataset retrieval and matching.
- Architecturally, this is the bridge between “alpha idea” and “what data fields can actually be used.”
- The README explicitly frames this as RAG-assisted dataset retrieval: the generator chooses datasets that fit the alpha before handing the expression to simulation.
6. Orchestration and file-watching
- AlphaGenerator/watchdog.py is the automation trigger.
- It watches for changes/new files in the research and strategy source folders, then kicks off generation and forwarding automatically.
- This makes the generator event-driven rather than requiring manual batch runs for every new input artifact.
7. Submission to simulation
- AlphaGenerator/send_to_simulation.py is the handoff point from generated alphas to the simulation backend.
- Conceptually, this is the integration seam where local alpha artifacts become API requests to the backtesting service.
- This script is the key integration boundary between “content generation” and “execution/backtest.”
8. Simulation service
- Simulation/main.py is the runtime entrypoint for the backtesting API.
- Simulation/schemas.py holds request/response contracts, and Simulation/simulate.py contains the simulation/backtest logic.
- The service is intended to run continuously in the background so that generated alphas can be posted to it asynchronously.
9. Credentials and external platform dependency
- Simulation/creds/example.json indicates the simulation service expects per-account credential files.
- The README states that the service integrates with WorldQuant BRAIN for automated backtesting, so that external platform is a core architectural dependency.
- This means the repo is not a standalone simulator; it is an automation wrapper around a third-party backtesting environment.
10. Persistence and results
- Results are stored in a SQLite database at Simulation/alphas/alpha_results.db according to the README.
- That makes the simulation layer stateful: it records submitted alphas and outcomes for later inspection or iteration.
- The database is the main durable output artifact of the pipeline.
11. External AI/API services
- The README calls out LLAMA_PARSE_API_KEYS and DEEPINFRA_API_KEYS, which indicates the generation side depends on external AI/document parsing services.
- These services are used to process PDFs and support the agentic generation workflow.
- So the generation path depends on both domain data files and external model/parsing APIs.
12. End-to-end data flow
- In practice, the flow is: research PDFs/strategy text + local dataset corpus → generation/refinement/splitting → alpha expression formatting and dataset matching → watchdog-triggered submission → simulation API → WorldQuant BRAIN backtest → SQLite result storage.
- The architecture is best drawn as two major blocks with a clear API boundary between them, plus shared domain assets (Datasets/ and syntax docs) feeding the generation side.
Main entry points: main.py Sim API · generate_from_research_pdfs.py PDF generator · program_alphas.py Program alphas · watchdog.py Watchdog
Components 18
Alpha Generation
Research ingestion, alpha drafting, refinement, and handoff.
- PDF generator · Generates alpha candidates from research PDFs.
AlphaGenerator/generate_from_research_pdfs.py - Strategy generator · Generates alpha candidates from strategy text inputs.
AlphaGenerator/generate_from_strategies.py - Program alphas · Stages candidate ideas into executable alpha expressions.
AlphaGenerator/utils/program_alphas.py - Distill alphas · Refines and compresses generated alpha candidates.
AlphaGenerator/utils/distill_alphas.py - Split alphas · Breaks alpha output into downstream-friendly units.
AlphaGenerator/utils/split_alphas.py - Watchdog · Watches source folders and triggers generation flow.
AlphaGenerator/watchdog.py - Send to sim · Submits generated alphas to the simulation backend.
AlphaGenerator/send_to_simulation.py
Simulation Service
Backtest API, execution logic, and result persistence.
- Sim API · Runs the backtesting service.
Simulation/main.py - Schemas · Request and response shapes for simulation requests.
Simulation/schemas.py - Simulate · Executes backtest logic against external platform.
Simulation/simulate.py - Credentials · Per-account credentials used by the simulation service.
Simulation/creds/example.json - Results DB · Persistent storage for submitted alphas and outcomes.
Shared Inputs
Local knowledge sources and external dependencies used by generation.
- Research PDFs · Financial research documents used to generate alpha ideas.
AlphaGenerator/ResearchPDFs/ssrn-2472571.pdf - Datasets · CSV knowledge base for dataset retrieval and alpha grounding.
Datasets/PriceVolumeDataForEquity.csv - Syntax refs · Brain knowledgebase syntax and dataset naming references.
AlphaGenerator/utils/BrainKnowledgebaseSyntax/syntax.md
Other components
- README · Project overview and operating model.
README.md - WorldQuant BRAIN · Third-party backtesting platform integrated by the simulation service.
- LLM / parse APIs · External parsing and model APIs used during generation.
Connections 19
- README → PDF generator: describes · Documents the generation pipeline.
- README → Sim API: describes · Documents the simulation service.
- Research PDFs → PDF generator: input · Research documents are converted into alpha ideas.
- Datasets → PDF generator: retrieval corpus · Local data fields ground dataset selection.
- Syntax refs → Program alphas: constraints · Syntax references guide executable alpha formatting.
- PDF generator → Program alphas: candidates · Generated ideas are staged for execution.
- Strategy generator → Program alphas: candidates · Strategy inputs are converted into programmatic alphas.
- Program alphas → Distill alphas: refine · Programmed alphas are distilled further.
- Distill alphas → Split alphas: partition · Refined alphas are split for downstream use.
- Watchdog → PDF generator: triggers · File changes start generation runs.
- Watchdog → Strategy generator: triggers · New strategy inputs start generation runs.
- Split alphas → Send to sim: handoff · Prepared alpha units are forwarded for simulation.
- Send to sim → Sim API: API request · Generated alphas are posted to the simulation service.
- Sim API → Schemas: uses · API endpoint validates request/response contracts.
- Sim API → Simulate: dispatches · Service entrypoint invokes backtest logic.
- Simulate → WorldQuant BRAIN: executes on · Backtests are run through the external platform.
- Credentials → Sim API: auth · Credential files configure platform access.
- Sim API → Results DB: stores · Simulation outputs are persisted for later review.
- LLM / parse APIs → PDF generator: supports · External AI and parsing services assist generation.