How gt-lit-lab/dbword works
• Core Module (dbword package):
• Database handling: Represent the Database class in database.py (the primary interface for querying word data from the pre-loaded databases).
• Data Extraction & Processing: Include datasets.py which might handle dataset-specific functions and extraction logic.
• Preprocessing Module: Emphasize preprocess.py, along with its internal functions (listify(), rm_punct(), rm_invalid(), consolidate()) that are responsible for turning raw text into a processed list of words.
• Utilities Module: Include utils.py and the download function for managing online retrieval and installation of new databases.
• Data Storage:
• Data files: Represent the data directory which contains .pkl files (e.g., subtlex-us.pkl and kuperman.pkl) that store the word statistics as dictionaries.
• External Dependencies:
• Pandas: Used for converting the extracted dictionary into a pandas.DataFrame.
• Pickle and OS: Used for loading the .pkl files and navigating paths via PACKAGE_DIR.
Relationships and Interactions
• Within Database.extract(), there is interaction with the preprocessed list of words and lookup operations using the data stored in the .pkl files.
• Utilities (download from utils.py) are connected to data storage; they download and convert raw files into .pkl format, ensuring the data directory is populated.
• The conversion to a pandas DataFrame via to_pandas() should be highlighted as the final transformation before presenting the data to the user.
Architectural Patterns and Design Principles
• Separation of Concerns: Each file/module has a specific responsibility (data extraction, preprocessing, utilities) making it easier to identify layers.
• Extensibility: The presence of utilities for downloading data and configurable extraction implies a design that can be later expanded with more databases or additional processing methods.
• Modularity: The project structure emphasizes a modular separation between data handling (database.py), preprocessing (preprocess.py), and core utilities (utils.py).