How evdubs/zacks-estimates-financial-statements works
Project type and purpose
• This is a command-line ETL (Extract-Transform-Load) pipeline project written in Racket.
• It automates the download of HTML from Zacks (“Detailed Estimates,” “Financials,” “Earnings Calendar”) and loads parsed data into a PostgreSQL database.
File structure
• Top-level Racket scripts are grouped by function:
• *-extract.rkt files fetch raw HTML and write to a local folder (/var/tmp/zacks/<dataset>).
• *-transform-load.rkt files parse the downloaded HTML and insert records into Postgres.
• Shell wrappers (*.sh) orchestrate chained steps for date handling or multi-step jobs.
• schema.sql defines the target DB schema; the project assumes a preloaded NASDAQ symbol table.
• No frontend or web server—this is purely a back-end data pipeline bound to CLI and file system.
README for additional insights
• Usage examples show a two-phase workflow:
a. Extraction: call racket X-extract.rkt with symbol/date options.
b. Transformation/Load: call racket X-transform-load.rkt pointing at extracted HTML.
• Dependencies: Racket libraries (gregor, html-parsing, sxml, tasks, threading), PostgreSQL.
• Operational notes: large data volume (≈2 GB), recommended frequency (weekly, monthly), optional compression.
• Blue for compute components (Racket scripts, scheduler)
• Green for external services (Zacks)
• Orange for persistence (file store, database)
Detail
• Configurable parameters (symbol range, folder date, DB credentials) as call-outs or notes.
• Schema dependency: annotate that all load scripts conform to the tables defined in schema.sql and rely on the NASDAQ symbols table.