# gt-lit-lab/dbword architecture

> Architecture diagram of the GitHub repository gt-lit-lab/dbword, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gt-lit-lab/dbword
- Repository: https://github.com/gt-lit-lab/dbword

## Overview

- Core Module (dbword package):
- Database handling: Represent the Database class in database.py (the primary interface for querying word data from the pre-loaded databases).
- Data Extraction & Processing: Include datasets.py which might handle dataset-specific functions and extraction logic.
- Preprocessing Module: Emphasize preprocess.py, along with its internal functions (listify(), rm_punct(), rm_invalid(), consolidate()) that are responsible for turning raw text into a processed list of words.
- Utilities Module: Include utils.py and the download function for managing online retrieval and installation of new databases.
- Data Storage:
- Data files: Represent the data directory which contains .pkl files (e.g., subtlex-us.pkl and kuperman.pkl) that store the word statistics as dictionaries.
- External Dependencies:
- Pandas: Used for converting the extracted dictionary into a pandas.DataFrame.
- Pickle and OS: Used for loading the .pkl files and navigating paths via PACKAGE_DIR.

##### Relationships and Interactions
- Within Database.extract(), there is interaction with the preprocessed list of words and lookup operations using the data stored in the .pkl files.
- Utilities (download from utils.py) are connected to data storage; they download and convert raw files into .pkl format, ensuring the data directory is populated.
- The conversion to a pandas DataFrame via to_pandas() should be highlighted as the final transformation before presenting the data to the user.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Each file/module has a specific responsibility (data extraction, preprocessing, utilities) making it easier to identify layers.
- Extensibility: The presence of utilities for downloading data and configurable extraction implies a design that can be later expanded with more databases or additional processing methods.
- Modularity: The project structure emphasizes a modular separation between data handling (database.py), preprocessing (preprocess.py), and core utilities (utils.py).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Subgraph for Client
    subgraph "Client"
        UC["User Code"]:::client
    end

    %% Subgraph for Core Library
    subgraph "Core Library"
        DB["database.py - Database Class"]:::core
        DS["datasets.py - Extraction & Processing"]:::core
        PP["preprocess.py - Preprocessing Module"]:::core
        UT["utils.py - Utilities Module"]:::core
    end

    %% Subgraph for Data Storage
    subgraph "Data Storage"
        DT["Word-Statistic Databases (.pkl files)"]:::data
    end

    %% Subgraph for External Dependencies
    subgraph "External Dependencies"
        PD["Pandas (DataFrame Conversion)"]:::external
        PK["Pickle & OS (File Loader/Navigator)"]:::external
    end

    %% Connections
    UC -->|"calls_API"| DB
    UC -->|"provides_raw_text"| PP
    PP -->|"processed_words"| DB
    DB -->|"uses_extraction"| DS
    DB -->|"queries_PKL"| DT
    UT -->|"downloads_databases"| DT
    DB -->|"to_dataframe"| PD

    %% Click Events
    click DB "https://github.com/gt-lit-lab/dbword/blob/main/dbword/database.py"
    click DS "https://github.com/gt-lit-lab/dbword/blob/main/dbword/datasets.py"
    click PP "https://github.com/gt-lit-lab/dbword/blob/main/dbword/preprocess.py"
    click UT "https://github.com/gt-lit-lab/dbword/blob/main/dbword/utils.py"
    click DT "https://github.com/gt-lit-lab/dbword/tree/main/dbword/data/"

    %% Styles
    classDef core fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef data fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef external fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef client fill:#fff3cd,stroke:#856404,stroke-width:2px;
```
