# evdubs/zacks-estimates-financial-statements architecture

> Architecture diagram of the GitHub repository evdubs/zacks-estimates-financial-statements, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/evdubs/zacks-estimates-financial-statements
- Repository: https://github.com/evdubs/zacks-estimates-financial-statements

## Overview

##### Project type and purpose
- This is a command-line ETL (Extract-Transform-Load) pipeline project written in Racket.
- It automates the download of HTML from Zacks (“Detailed Estimates,” “Financials,” “Earnings Calendar”) and loads parsed data into a PostgreSQL database.

##### File structure
- Top-level Racket scripts are grouped by function:
- `*-extract.rkt` files fetch raw HTML and write to a local folder (`/var/tmp/zacks/<dataset>`).
- `*-transform-load.rkt` files parse the downloaded HTML and insert records into Postgres.
- Shell wrappers (`*.sh`) orchestrate chained steps for date handling or multi-step jobs.
- `schema.sql` defines the target DB schema; the project assumes a preloaded NASDAQ symbol table.
- No frontend or web server—this is purely a back-end data pipeline bound to CLI and file system.

##### README for additional insights
- Usage examples show a two-phase workflow:
a. Extraction: call `racket X-extract.rkt` with symbol/date options.
b. Transformation/Load: call `racket X-transform-load.rkt` pointing at extracted HTML.
- Dependencies: Racket libraries (`gregor`, `html-parsing`, `sxml`, `tasks`, `threading`), PostgreSQL.
- Operational notes: large data volume (≈2 GB), recommended frequency (weekly, monthly), optional compression.

- Blue for compute components (Racket scripts, scheduler)
- Green for external services (Zacks)
- Orange for persistence (file store, database)

##### Detail
- Configurable parameters (symbol range, folder date, DB credentials) as call-outs or notes.
- Schema dependency: annotate that all load scripts conform to the tables defined in `schema.sql` and rely on the NASDAQ symbols table.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Scheduler
    subgraph "Scheduler (cron)"
        Scheduler["Cron Scheduler"]:::compute
    end

    %% External Source
    ExtSrc["Zacks HTML Endpoints"]:::external

    %% Orchestration Scripts
    subgraph "Orchestration Scripts"
        EstOrch["estimate-extract-transform-load.sh"]:::compute
        EarnOrch["earnings-calendar-extract-transform-load.sh"]:::compute
        FinStmtOrch["financial-statement-transform-load-yesterday.sh"]:::compute
    end

    %% Extraction Phase
    subgraph "Extraction Phase"
        EstExt["estimate-extract.rkt"]:::compute
        EarnExt["earnings-calendar-extract.rkt"]:::compute
        FinStmtExt["financial-statement-extract.rkt"]:::compute
    end

    %% Staging Area
    Staging["/var/tmp/zacks/..."]:::persistence

    %% Transform-Load Phase
    subgraph "Transform-Load Phase"
        EstTL["estimate-transform-load.rkt"]:::compute
        EarnTL["earnings-calendar-transform-load.rkt"]:::compute
        BS_TL["balance-sheet-transform-load.rkt"]:::compute
        CF_TL["cash-flow-statement-transform-load.rkt"]:::compute
        CF_TL2["cash-flow-statement-transform-load.2024-02-01.rkt"]:::compute
        IS_TL["income-statement-transform-load.rkt"]:::compute
    end

    %% Database & Schema
    subgraph "Database & Schema"
        Schema["schema.sql"]:::persistence
        Postgres["PostgreSQL DB"]:::persistence
    end

    %% Utilities
    subgraph "Utilities"
        DumpDAT["dump-dat.rkt"]:::compute
        DumpCal["dump-dolt-calendar.rkt"]:::compute
        DumpEst["dump-dolt-estimates.rkt"]:::compute
        DumpStmt["dump-dolt-statements.rkt"]:::compute
    end

    %% Documentation
    subgraph "Documentation"
        Readme["README.md"]:::compute
        License["LICENSE"]:::compute
    end

    %% Connections
    Scheduler --> EstOrch
    Scheduler --> EarnOrch
    Scheduler --> FinStmtOrch

    EstOrch --> EstExt
    EstOrch --> EstTL

    EarnOrch --> EarnExt
    EarnOrch --> EarnTL

    FinStmtOrch --> FinStmtExt
    FinStmtOrch --> BS_TL
    FinStmtOrch --> CF_TL
    FinStmtOrch --> CF_TL2
    FinStmtOrch --> IS_TL

    ExtSrc --> EstExt
    ExtSrc --> EarnExt
    ExtSrc --> FinStmtExt

    EstExt --> Staging
    EarnExt --> Staging
    FinStmtExt --> Staging

    Staging --> EstTL
    Staging --> EarnTL
    Staging --> BS_TL
    Staging --> CF_TL
    Staging --> CF_TL2
    Staging --> IS_TL

    EstTL --> Postgres
    EarnTL --> Postgres
    BS_TL --> Postgres
    CF_TL --> Postgres
    CF_TL2 --> Postgres
    IS_TL --> Postgres

    Schema --> Postgres

    %% Click Events
    click EstExt "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/estimate-extract.rkt"
    click EstTL "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/estimate-transform-load.rkt"
    click EstOrch "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/estimate-extract-transform-load.sh"
    click EarnExt "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/earnings-calendar-extract.rkt"
    click EarnTL "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/earnings-calendar-transform-load.rkt"
    click EarnOrch "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/earnings-calendar-extract-transform-load.sh"
    click FinStmtExt "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/financial-statement-extract.rkt"
    click BS_TL "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/balance-sheet-transform-load.rkt"
    click CF_TL "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/cash-flow-statement-transform-load.rkt"
    click CF_TL2 "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/cash-flow-statement-transform-load.2024-02-01.rkt"
    click IS_TL "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/income-statement-transform-load.rkt"
    click FinStmtOrch "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/financial-statement-transform-load-yesterday.sh"
    click Schema "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/schema.sql"
    click DumpDAT "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/dump-dat.rkt"
    click DumpCal "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/dump-dolt-calendar.rkt"
    click DumpEst "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/dump-dolt-estimates.rkt"
    click DumpStmt "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/dump-dolt-statements.rkt"
    click Readme "https://github.com/evdubs/zacks-estimates-financial-statements/blob/master/README.md"
    click License "https://github.com/evdubs/zacks-estimates-financial-statements/tree/master/LICENSE"

    %% Styles
    classDef compute fill:#add8e6,stroke:#000;
    classDef external fill:#90ee90,stroke:#000;
    classDef persistence fill:#ffa500,stroke:#000;
```
