How jerbouma/financedatabase works
Project Type and Purpose
• This project is a Python package providing a free, community‐driven financial database. It organizes financial product categorizations (e.g., Equities, ETFs, Funds, Indices, Currencies, Cryptocurrencies, Money Markets) into CSV files and compressed data formats. It supplies an API (via modules like Equities.py, ETFs.py, etc.) that makes it easy for users to select, search, and aggregate ticker data. Users can also seamlessly integrate with related projects such as the Finance Toolkit.
• The README emphasizes that while it does not provide live stock data itself (which can be obtained from other sources), it gives organized financial product information to aid deeper financial analytics.
Detailed File Structure Analysis
• The top-level directories show good separation of concerns:
• .github/: Contains funding info, issue templates, and CI/CD workflows (e.g., linting, database update). This suggests an emphasis on quality checks and automated maintenance.
• compression/: Holds compressed data files, JSON category files, and even a Jupyter Notebook (compression.ipynb) that likely demonstrates how to inspect or use the compressed datasets.
• database/: Contains CSV files for various asset classes (cryptos.csv, equities.csv, etc.), which are the raw/canonical data sources.
• financedatabase/: Contains the core Python modules. Each asset class has a separate module (Equities.py, ETFs.py, etc.) plus helper routines and an __init__.py to manage package exposure. This modular internal structure is indicative of a design that encapsulates business logic per product category.
• examples/: Provides several Jupyter Notebooks showcasing different usage patterns (from basic queries to advanced analyses such as Dupont Analysis or technical analysis charts).
• The presence of pyproject.toml, requirements.txt, pre-commit, and CONTRIBUTING.md further emphasize good practices and ease for developers to build, contribute, and run tests.
Main Components
• Data Source / Storage Layer:
• Database CSV files (in the database/ folder).
• Compressed data files along with category JSON and Excel files (in compression/).
• Business Logic / API Layer:
• The “financedatabase” package (modules such as Equities.py, ETFs.py, Cryptos.py, etc.) that reads the raw CSV/compressed files.
• Shared helper functions found in helpers.py.
• This layer provides functions for “select”, “search” and “options” to query the database.
• Integration Layer:
• Interfaces provided (e.g., to_toolkit functionality) that allow users to pass DataFrames or ticker lists to external systems like FinanceToolkit.
• User/Application Layer:
• Examples and notebooks from the examples/ directory.
• Scripts that utilize the package for financial analysis (including interactions like plotting recommendations, each showing how selections and transformations lead to analytics).
• DevOps/Quality Layer:
• CI/CD workflows (in .github/workflows) that automate linting, testing, and database updates.
• Contribution guidelines and license information.
Relationships & Interactions
• Data flows upward from the static CSV/compressed files (data sources) to the “financedatabase” API modules that parse the data.
• The API modules expose methods (such as select, options, search) and convert raw data into Pandas DataFrames.
• These DataFrames can then be passed to other utilities – for example, wrapped with FinanceToolkit functionality (via the to_toolkit method) for further financial analyses.
• External systems (like FinanceToolkit or user-facing notebooks) consume the processed data.
• CI/CD workflows trigger on pushes/PRs and are connected to code quality and automated database update scripts.
Architectural Patterns & Design Principles
• Modular Design: Different asset classes are encapsulated in separate modules (e.g., Equities.py, ETFs.py) which simplifies adding new financial products.
• Separation of Concerns: Data storage (CSV/JSON/Compressed files) is clearly separated from the logic that performs queries and data transformations.
• Interoperability: The API is designed so that the output (Pandas DataFrames) can be easily consumed by other tools and libraries.
• Community-Driven Extensibility: The presence of plain CSV files and clear contribution guidelines promote ease of modification and community contributions.
• Automated Maintenance: GitHub workflows ensure that quality checks, linting, and even database refresh/update tasks occur regularly, automating maintenance operations.
b. Above that, group related business logic components (the modules under financedatabase/). Inside this group, partition your modules by functional grouping (e.g., “Asset Classes”).
c. Next, illustrate the Integration Layer: show that outputs (Pandas DataFrames) are consumed by user code (in the examples/) and further processed by external tools like FinanceToolkit.
d. On the side or as an overlay, illustrate the DevOps/CI CD layer from your .github/workflows, linking it to the code repository and automation of tests.
f. Label every component clearly and add annotations where necessary.